Previous: Directive Reference, Up: Configurations



2.2 Configuration Example

The following configuration executes a multi-threaded Postmark benchmark, and is included in the distribution as postmark.ap:

1  #!/usr/bin/perl /usr/local/bin/auto-pilot
2  #
3  # Package: auto-pilot
4  # Erez Zadok <ezk AT cs.sunysb.edu>
5  # Charles P. Wright <cwright AT cs.sunysb.edu>
6  # Copyright (c) 2001-2005 Stony Brook University
7  
8  # What is the name of this test?  Auto-pilot doesn't internally
9  # use it for Postmark, but we may want it for setting paths, etc.
10 VAR BENCH=postmark
11 
12 # How many times do we run it?
13 #  We could for example, run it ten times
14 #   VAR TERMINATE=10
15 #  However, it is more interesting to run it five times, then run
16 #  getstats to check if the half-width of our confidence intervals
17 #  are less than 5% of the mean for elapsed, user, and system time.
18 #  We also don't want it to run away and run more than 30 times on us.
19 VAR TERMINATE=10 1 getstats --predicate \
20 '("$name" ne "User" && "$name" ne "System" && "$name" ne "Elapsed") || \
21 ("$delta" < 0.05 * $mean) || ($count > 30)' --
22 
23 # How many threads should we go up to?
24 VAR NTHREADS=32
25 
26 # What file systems do we want to test?
27 VAR TESTFS=ext2 ext3 reiserfs
28 
29 # Include common.inc.  You can override variables and include your own
30 # commands by creating local.inc, which is automatically included by
31 # common.inc.
32 INCLUDE common.inc
33 
34 # Do the actual tests
35 FOREACH FS %TESTFS%
36 	FOR THREADCOUNT=1 TO %NTHREADS% FACTOR 2
37 		THREADS=%THREADCOUNT%
38 
39 		TEST %FS%-%THREADS%	%TERMINATE%
40 			SETUP fs-setup.sh %FS%
41 			EXEC postmark.sh
42 			CLEANUP fs-cleanup.sh %FS%
43 		DONE
44 	DONE
45 DONE
46 
47 # All done
48 INCLUDE ok.inc

Line 1 informs the OS to use Perl as the interpreter for this Auto-pilot configuration. Only one level of interpreter is allowed, so you must execute Perl with Auto-pilot's path as the argument, because Auto-pilot itself uses an interpreter. Auto-pilot uses the pound sign (#) as a comment symbol, and any text after a pound sign is ignored. Lines 3–6 are comments containing copyright notice. Blank lines are ignored.

Line 10 assigns the value postmark to the variable bench. This variable is not directly used by this configuration file, but later on we will use it to chose where to store the Auto-pilot results.

Lines 12–19 set a variable named TERMINATE, which is used later in the configuration to determine at what point a TEST should end. The simplest method is to specify a fixed number of tests (e.g., VAR TERMINATE=10).

A more powerful method is to run a predicate command. In this example, we execute the test at least ten times. For each iteration (denoted by the "1" in the variable), we execute a predicate. To execute the predicate after every other iteration, we would replace the "1" with a "2". It can be useful to run the predicate only after several runs, so that computing the predicate does not use more time than the benchmark itself. In this case we execute the command getstats --predicate '("$name" ne "User" && "$name" ne "System" && "$name" ne "Elapsed") || ("$delta" < 0.05 * $mean) || ($count > 30)' --. Getstats is our generic results processing engine; for a complete description See Getstats. Getstats can run arbitrary predicates using various summary statistics (e.g., the mean, median, or standard deviation). The predicate is checked for each quantity that Auto-pilot measures. The first part of the predicate, ("$name" ne "User" && "$name" ne "System" && "$name" ne "Elapsed"), skips all but User, System, and Elapsed time, because these are the measured quantities we are generally interested in. Wait time and CPU utilization are excluded because they are derived from these quantities. The next portion of the predicate, ("$delta" < 0.05 * $mean), returns true if the half-width of a confidence interval (the confidence level is 95% by default) is less than 5% of the mean. The final portion, ($count > 30) prevents a poorly behaving test from executing for more than 30 iterations.

The TERMINATE variable is rather long, and spans multiple lines using a backslash. To continue a line, the backslash must be the very last character before the newline (no spaces are allowed).

We then set another two variables NTHREADS (line 24), which is later used to determine the maximum number of threads, and TESTFS (line 27) which is used to determine the file systems to execute the benchmark on.

The next directive (on line 32), is INCLUDE common.inc (this file is included with the Auto-pilot distribution). The common.inc file performs several tasks:

  1. Adds your libexec directory (by default /usr/local/libexec) to the shell's PATH, so that programs like aptime can be found.
  2. Adds your share directory (by default /usr/local/share/auto-pilot) to the shell's PATH, so that shared shell script components and hooks can be found.
  3. The FASTFAIL directive is used so that benchmarks stop after one fails.
  4. The FORMAT and NOSERVICES variables are set to "0".
  5. The file local.inc is included.
  6. If NOSERVICES is set to one, then background services (e.g., cron) are turned off to prevent them from interfering with the benchmark.

To override any previous settings, and define new ones without editing the installed configuration files you should create a local.inc in the directory where you run the benchmarks. For example, if you wanted to use another predicate, then you would override TERMINATE here. You also need to set the TESTDEV, TESTROOT, RESULTS, LOGS, OKADDR, and FAILADDR.

An example local.inc could have the following:

     RESULTS=$HOME$/results/%BENCH%
     LOGS=$HOME$/logs/%BENCH%
     
     VAR TESTFS=ext2
     VAR NOSERVICES=1
     
     ENV FORMAT=1
     ENV TESTDEV=/dev/sda1
     ENV TESTROOT=/mnt/test
     
     ENV OKADDR=you@example.com
     ENV FAILADDR=you@example.com

In this example, the results files are stored under the current user's home directory in results/%BENCH%, where %BENCH% is the name of the benchmark as defined in the configuration file. The stdout/stderr logs are similarly stored in logs/%BENCH% under the current user's home directory.

The third directive, VAR TESTFS=ext2, overrides the default value of TESTFS, so that only ext2 is tested. The fourth causes common.inc to shutdown excess services.

The remaining lines set environment variables that are used by the default Auto-pilot shell scripts. Setting FORMAT to 1 causes the scripts to recreate the file system. TESTDEV determines what device should be used, and TESTROOT determines what directory is used.

The two remaining lines set addresses that should be used to send email. OKADDR, FAILADDR are the address that mail is sent to on successful completion and failure, respectively. I have found it useful to add .procmailrc recipes like the following to forward these messages to a mobile phone:

     # Send a copy of all messages with a subject starting with Benchmarks to
     # another email address for Auto-pilot paging.
     :0 c
     * ^Subject: Benchmarks
     ! yourphone@telco.com

The remaining configuration files are similar to postmark.ap, and using these as templates you should be able to create your own.