All QMTest functionality is available using the qmtest command.
These options can be used with any QMTest command, and must precede the command name on the command line.
All options are available in a "long form" prefixed with "--" (two hyphens). Some options also may be specified in a "short form" consisting of a single hyphen and a one-letter abbreviation. Short-form options may be combined; for example, -abc is equivalent to -a -b -c.
Use the test database located in the directory given by path. This flag overrides the value of the environment variable QMTEST_DB_PATH. If neither this flag nor the environment variable is specified, QMTest assumes that the current directory should be used as the database. See Section 3.1.6.
Display help information, listing commands and general options for the qmtest command.
Generate progress and status messages while executing. This option may be specified more than once; the more times it is specified, the higher the verbose level, and the more messages are printed.
Describe the version of QMTest in use.
Additional options are available for specific commands; these are presented with each command. Options specific to a command must follow the command on the command line. Specify the --help (-h) option after the command for a description of the command and a list of of available options for that command.
The qmtest create-target command creates a new target. A target is an entity that runs tests; normally, a target corresponds to a particular machine.
The target's name and class must be specified. An optional group may also be specified. When QMTest decides which target to use to run a particular tests, it will select a target that matches the test's requested target group.
The create-target command accepts these options:
Set the target class argument name to value. The set of arguments names and valid values is dependent on the target class in use.
Write the target description to the indicated file. If there are already targets listed in file, they will be preserved, except that any target with the same name as the new target will be removed. If this option is not present, the file used will be the QMTest/targets file in the test database directory.
The qmtest create-tdb command creates a new, empty test database. A test database is a directory in which QMTest stores configuration files, tests, and other data. Certain test database classes may also store data elsewhere, such as in an external relational database.
The test database is created in the directory specified by --tdb (-D) option or by setting the QMTEST_DB_PATH environment variable. If no database path is specified, QMTest assumes that the current directory is the test database.
By default, QMTest creates a new test database that uses the standard XML-based implementation. (See Section 4.2 for information about writing a test database class.)
The create-tdb command accepts these options:
Set the database attribute name to value. The set of attribute names and valid values is dependent on the database class in use. The default database class accepts no attributes.
Use the test database class given by class. Once you create a test database, you cannot change the test database implementation it uses. If you do not use this option, QMTest will use the default test database implementation, which uses an XML file format to store tests.
The qmtest gui starts the graphical user interface. The graphical user interface is accessed through a web browser. You must have a web browser that supports JavaScript to use the graphical interface. QMTest has been tested with recent versions of Internet Explorer and Netscape Navigator. Other web browsers may or may nor work with QMTest.
The gui command accepts these options:
Bind the server to the indicated internet address, which should be a dotted quad. By default, the server binds itself to the address 127.0.0.1, which is the address of the local machine. If you specify another address, the server will be accessible to users on other machines. QMTest does not perform any authentication of remote users, so you should not use this option unless you have a firewall in place that blocks all untrusted users.
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
Run the QMTest GUI as a daemon. In this mode, QMTest will detach from the controlling terminal and run in the background until explicitly shutdown.
For details about this option, see the description of the qmtest run command.
Do not attempt to start a web browser when starting the GUI. QMTest will still print out the URL at which the server can be accessed. You can then connect to this URL manually using the browser of your choice.
Specify the path to which the QMTest GUI will write its process ID. This option is useful if you want to run QMTest as a daemon. If this option is not provided, no PID file is written. If you specify this option, but path is the empty string, QMTest will check the .qmrc configuration file for a pid-file entry. If there is no such entry, QMTest will use an appropriate platform-specific default value.
Specify the port on which the QMTest GUI will listen for connections. If this option is not provided, QMTest will select an available port automatically.
For details about this option, see the description of the qmtest run command.
The qmtest extensions lists available extension classes and provides a brief description of each class. You can use this command to list all of the available extension classes, or to list all of the available extension classes of a particular type. For example, you can use this command to list all of the available test classes.
The extensions command accepts these options:
List the available extension classes of the indicated kind. The kind must be one of test, resource, target, or database.
The qmtest register registers an extension class with QMTest. As part of this process, QMTest will load your extension class. If the extension class cannot be loaded, QMTest will tell you what went wrong.
QMTest will search for your extension class in the directories it would search when running tests, including those given by the environment variable QMTEST_CLASS_PATH.
The kind argument tells QMTest what kind of extension class you are registering. The kind must be one of test, resource, target, or database.
The class-name argument gives the name of the class in the form module.Class. QMTest will look for a file whose basename is the module name and whose extension is either py, pyc, or pyo.
The qmtest run command runs tests and displays the results. If no test or suite names are specified, QMTest runs all of the tests in the test database. If test or suite names are specified, only those tests or suites are run. Tests listed more than once (directly or by inclusion in a test suite) are run only once.
The run command accepts these options:
Add a property to the test execution context. The name of the property is name, and its value is set to the string value.
This option may be specified multiple times.
Read properties for the test execution context from the file file.
The file should be a text file with one context property on each line, in the format name=value. Leading and trailing whitespace on each line are ignored. Also, blank lines and lines that begin with "#" (a hash mark) are ignored as comments.
This option may be specified more than once, and used in conjunction with the --context option. All of the context properties specified are added to the eventual context. If a property is set more than once, the last value provided is the one used.
If this option is not specified, but a file named context exists in the current directory, that file is read. The properties specified in this file are processed first; the values in this file can be overridden by subsequent uses of the --context option on the command line.
Control the format used when displaying results. The format specified must be one of full, brief, stats, or none. The brief format is the default. In the full format, QMTest displays any annotations provided in test results. In the brief mode only the causes of failures are shown; detailed annotations are not shown. In the stats format, no details about failing tests are displayed; only statistics showing the number of passing and failing tests are displayed. In the none mode, no results are displayed, but a results file is still created, unless the --no-output option is also provided.
Run tests in multiple count concurrent processes on the local computer. On multiprocessor machines, the processes may be scheduled to run in parallel on different processors. QMTest automatically collects results from the processes and presents combines test results and summary. By default, one process is used.
This option may not be combined with the --targets (-T) option.
Do not produce a test results file.
Write full test results to file. Specify "-" (a hyphen) to write results to standard output. If neither this option nor --no-output is specified, the results are written to the file named results.qmr in the current directory.
Treat file as a set of expected outcomes. The file must have be a results file created either by qmtest run, or by saving results in the graphical user interface. QMTest will expect the results of the current test run to match those specified in the file and will highlight differences from those results.
Run the tests in a random order.
This option can be used to find hidden dependencies between tests in the testsuite. (You may not notice the dependencies if you always run the tests in the same order.)
Rerun only those tests that had unexpected outcomes.
The tests run are determined as follows. QMTest starts with all of the tests specified on the command line, or, if no tests are explicitly specified, all of the tests in the database. If no expectations file is specified (see the description of the --outcomes option), then all tests that passed in the results file indicated by the --rerun option are removed form the set of eligible tests. If an expectations file is specified, then the tests removed are tests whose outcome in the results file indicated by the --rerun option is the same as in the expectations file.
The --rerun provides a simple way of rerunning failing tests. If you run your tests and notice failures, you might try to fix those failing tests. Then, you can rerun the failing tests to see if you succeeded by using the --rerun option.
Write a summary of the test run to file. Specify "-" (a hyphen) to write results to standard output (the default).
QMTest prints a summary of test results, including statistics and the names of tests that did not pass. If expected outcomes were specified, the the names of tests that had unexpected results are printed, instead of the names of tests that did not pass.
Do not produce a summary of the test run.
If the --random is used, QMTest randomizes the order in which tests are run, subject to the constraints described in Section 3.2.1. By default, the random number generator is seeded using the system time.
For debugging purposes, it is sometimes necessary to obtain a reproducible sequence of tests. Use the --seed option to specify the seed for the random number generator.
Note that even with the same random number seed, if tests are run in parallel, scheduling uncertainty may still produce variation in the order in which tests are run.
Use targets specified in target specification file file. If this option is not present, the QMTest/targets in the test database directory will be used. If that file is not present, the tests will be run in serial on the local machine.
The qmtest summarize extracts information stored in a results file and displays this information on the console. The information is formatted just as if the tests had just been run, but QMTest does not actually run the tests.
The summarize command accepts the following options:
For details about this option, see the description of the qmtest run command.
For details about this option, see the description of the qmtest run command.
QMTest recognizes the following environment variables:
If this environment variable is set, QMTest uses it as as the path to the Python interpreter. If this environment variable is not set, QMTest looks for a file named python in the bin directory where QM is installed. If this file does not exist, but /usr/bin/python2 exists, QMTest will use that path. Otherwise, QMTest searches for python in the directories listed in the PATH environment variable.
If this environment variable is set, it should contain a list of directories in the same format as used for the system's PATH environment variable. These directories are searched (before the directories that QMTest searches by default) when looking for extension classes such as test classes and database classes.
If this environment variable is set, its value is used as the location of the test database, unless the --tdb (-D) option is used. If this environment variable is not set and the --tdb option is not used, the current directory is used as the test database.
These configuration variables are used by QMTest. You should define them in the [qmtest] section of your QM configuration file.
The default path to use when creating a PID file with the --pid-file. If this entry is not present, an appropriate platform-specific default value is used.