Listener interface

Robot Framework has a listener interface that can be used to receive notifications about test execution. Example usages include external test monitors, sending a mail message when a test fails, and communicating with other systems. Listener API version 3 also makes it possible to modify tests and results during the test execution.

Listeners are classes or modules with certain special methods, and they can be implemented both with Python and Java. Listeners that monitor the whole test execution must be taken into use from the command line. In addition to that, test libraries can register listeners that receive notifications while that library is active.

Taking listeners into use

Listeners are taken into use from the command line with the --listener option so that the name of the listener is given to it as an argument. The listener name is got from the name of the class or module implementing the listener interface, similarly as `test library names`_ are got from classes implementing them. The specified listeners must be in the same `module search path`_ where test libraries are searched from when they are imported. Other option is to give an absolute or a relative path to the listener file `similarly as with test libraries`__. It is possible to take multiple listeners into use by using this option several times:

robot --listener MyListener tests.robot
robot --listener com.company.package.Listener tests.robot
robot --listener path/to/MyListener.py tests.robot
robot --listener module.Listener --listener AnotherListener tests.robot

It is also possible to give arguments to listener classes from the command line. Arguments are specified after the listener name (or path) using a colon (:) as a separator. If a listener is given as an absolute Windows path, the colon after the drive letter is not considered a separator. Starting from Robot Framework 2.8.7, it is possible to use a semicolon (;) as an alternative argument separator. This is useful if listener arguments themselves contain colons, but requires surrounding the whole value with quotes on UNIX-like operating systems:

robot --listener listener.py:arg1:arg2 tests.robot
robot --listener "listener.py;arg:with:colons" tests.robot
robot --listener C:\Path\Listener.py;D:\data;E:\extra tests.robot

Listener interface versions

There are two supported listener interface versions. Listener version 2 has been available since Robot Framework 2.1, and version 3 is supported by Robot Framework 3.0 and newer. A listener must have attribute ROBOT_LISTENER_API_VERSION with value 2 or 3, either as a string or as an integer, depending on which API version it uses. There has also been an older listener version 1, but it is not supported anymore by Robot Framework 3.0.

The main difference between listener versions 2 and 3 is that the former only gets information about the execution but cannot directly affect it. The latter interface gets data and result objects Robot Framework itself uses and is thus able to alter execution and change results. See listener examples for more information about what listeners can do.

Another difference between versions 2 and 3 is that the former supports both Python and Java but the latter supports only Python.

Listener interface methods

Robot Framework creates instances of listener classes when the test execution starts and uses listeners implemented as modules directly. During the test execution different listener methods are called when test suites, test cases and keywords start and end. Additional methods are called when a library or a resource or variable file is imported, when output files are ready, and finally when the whole test execution ends. A listener is not required to implement any official interface, and it only needs to have the methods it actually needs.

Listener versions 2 and 3 have mostly the same methods, but the arguments they accept are different. These methods and their arguments are explained in the following sections. All methods that have an underscore in their name have also camelCase alternative. For example, start_suite method can be used also with name startSuite.

Listener version 2

Listener methods in the API version 2 are listed in the following table. All methods related to test execution progress have the same signature method(name, attributes), where attributes is a dictionary containing details of the event. Listener methods are free to do whatever they want to do with the information they receive, but they cannot directly change it. If that is needed, listener version 3 can be used instead.

Methods in the listener API 2

Method	Arguments	Documentation
start_suite	name, attributes	Called when a test suite starts. Contents of the attribute dictionary: id: Suite id. s1 for the top level suite, s1-s1 for its first child suite, s1-s2 for the second child, and so on. New in RF 2.8.5. longname: Suite name including parent suites. doc: Suite documentation. metadata: `Free test suite metadata`_ as a dictionary/map. source: An absolute path of the file/directory the suite was created from. New in RF 2.7. suites: Names of the direct child suites this suite has as a list. tests: Names of the tests this suite has as a list. Does not include tests of the possible child suites. totaltests: The total number of tests in this suite. and all its sub-suites as an integer. starttime: Suite execution start time.
end_suite	name, attributes	Called when a test suite ends. Contents of the attribute dictionary: id: Same as in start_suite. longname: Same as in start_suite. doc: Same as in start_suite. metadata: Same as in start_suite. source: Same as in start_suite. starttime: Same as in start_suite. endtime: Suite execution end time. elapsedtime: Total execution time in milliseconds as an integer status: Suite status as string PASS or FAIL. statistics: Suite statistics (number of passed and failed tests in the suite) as a string. message: Error message if suite setup or teardown has failed, empty otherwise.
start_test	name, attributes	Called when a test case starts. Contents of the attribute dictionary: id: Test id in format like s1-s2-t2, where the beginning is the parent suite id and the last part shows test index in that suite. New in RF 2.8.5. longname: Test name including parent suites. doc: Test documentation. tags: Test tags as a list of strings. critical: yes or no depending is test considered critical or not. template: The name of the template used for the test. An empty string if the test not templated. starttime: Test execution execution start time.
end_test	name, attributes	Called when a test case ends. Contents of the attribute dictionary: id: Same as in start_test. longname: Same as in start_test. doc: Same as in start_test. tags: Same as in start_test. critical: Same as in start_test. template: Same as in start_test. starttime: Same as in start_test. endtime: Test execution execution end time. elapsedtime: Total execution time in milliseconds as an integer status: Test status as string PASS or FAIL. message: Status message. Normally an error message or an empty string.
start_keyword	name, attributes	Called when a keyword starts. name is the full keyword name containing possible library or resource name as a prefix. For example, MyLibrary.Example Keyword. Contents of the attribute dictionary: type: String Keyword for normal keywords, Setup or Teardown for the top level keyword used as setup/teardown, For for for loops, and For Item for individual for loop iterations. NOTE: Keyword type reporting was changed in RF 3.0. See issue #2248 for details. kwname: Name of the keyword without library or resource prefix. New in RF 2.9. libname: Name of the library or resource the keyword belongs to, or an empty string when the keyword is in a test case file. New in RF 2.9. doc: Keyword documentation. args: Keyword's arguments as a list of strings. assign: A list of variable names that keyword's return value is assigned to. New in RF 2.9. tags: `Keyword tags`_ as a list of strings. New in RF 3.0. starttime: Keyword execution start time.
end_keyword	name, attributes	Called when a keyword ends. name is the full keyword name containing possible library or resource name as a prefix. For example, MyLibrary.Example Keyword. Contents of the attribute dictionary: type: Same as with start_keyword. kwname: Same as with start_keyword. libname: Same as with start_keyword. doc: Same as with start_keyword. args: Same as with start_keyword. assign: Same as with start_keyword. tags: Same as with start_keyword. starttime: Same as with start_keyword. endtime: Keyword execution end time. elapsedtime: Total execution time in milliseconds as an integer status: Keyword status as string PASS or FAIL.
log_message	message	Called when an executed keyword writes a log message. message is a dictionary with the following contents: message: The content of the message. level: `Log level`_ used in logging the message. timestamp: Message creation time in format YYYY-MM-DD hh:mm:ss.mil. html: String yes or no denoting whether the message should be interpreted as HTML or not. Starting from RF 3.0, this method is not called if the message has level below the current `threshold level <Log levels_>`_.
message	message	Called when the framework itself writes a syslog_ message. message is a dictionary with the same contents as with log_message method.
library_import	name, attributes	Called when a library has been imported. name is the name of the imported library. If the library has been imported using the `WITH NAME syntax`_, name is the specified alias. Contents of the attribute dictionary: args: Arguments passed to the library as a list. originalname: The original library name when using the WITH NAME syntax, otherwise same as name. source: An absolute path to the library source. None with libraries implemented with Java or if getting the source of the library failed for some reason. importer: An absolute path to the file importing the library. None when BuiltIn_ is imported well as when using the Import Library keyword. New in Robot Framework 2.9.
resource_import	name, attributes	Called when a resource file has been imported. name is the name of the imported resource file without the file extension. Contents of the attribute dictionary: source: An absolute path to the imported resource file. importer: An absolute path to the file importing the resource file. None when using the Import Resource keyword. New in Robot Framework 2.9.
variables_import	name, attributes	Called when a variable file has been imported. name is the name of the imported variable file with the file extension. Contents of the attribute dictionary: args: Arguments passed to the variable file as a list. source: An absolute path to the imported variable file. importer: An absolute path to the file importing the resource file. None when using the Import Variables keyword. New in Robot Framework 2.9.
output_file	path	Called when writing to an `output file`_ is ready. path is an absolute path to the file.
log_file	path	Called when writing to a `log file`_ is ready. path is an absolute path to the file.
report_file	path	Called when writing to a `report file`_ is ready. path is an absolute path to the file.
xunit_file	path	Called when writing to an `xunit file`_ is ready. path is an absolute path to the file.
debug_file	path	Called when writing to a `debug file`_ is ready. path is an absolute path to the file.
close		Called when the whole test execution ends. With library listeners called when the library goes out of scope.

The available methods and their arguments are also shown in a formal Java interface specification below. Contents of the java.util.Map attributes are as in the table above. It should be remembered that a listener does not need to implement any explicit interface or have all these methods.

public interface RobotListenerInterface {
    public static final int ROBOT_LISTENER_API_VERSION = 2;
    void startSuite(String name, java.util.Map attributes);
    void endSuite(String name, java.util.Map attributes);
    void startTest(String name, java.util.Map attributes);
    void endTest(String name, java.util.Map attributes);
    void startKeyword(String name, java.util.Map attributes);
    void endKeyword(String name, java.util.Map attributes);
    void logMessage(java.util.Map message);
    void message(java.util.Map message);
    void outputFile(String path);
    void logFile(String path);
    void reportFile(String path);
    void debugFile(String path);
    void close();
}

Listener version 3

Listener version 3 has mostly the same methods as listener version 2 but arguments of the methods related to test execution are different. This API gets actual running and result model objects used by Robot Framework itself, and listeners can both directly query information they need and also change the model objects on the fly.

Listener version 3 was introduced in Robot Framework 3.0. At least initially it does not have all methods that the version 2 has. The main reason is that suitable model objects are not available internally. The close method and methods related to output files are called exactly same way in both versions.

Methods in the listener API 3

Method	Arguments	Documentation
start_suite	data, result	Called when a test suite starts. data and result are model objects representing the `executed test suite <running.TestSuite_>`_ and `its execution results <result.TestSuite_>`_, respectively.
end_suite	data, result	Called when a test suite ends. Same arguments as with start_suite.
start_test	data, result	Called when a test case starts. data and result are model objects representing the `executed test case <running.TestCase_>`_ and `its execution results <result.TestCase_>`_, respectively.
end_test	data, result	Called when a test case ends. Same arguments as with start_test.
start_keyword	N/A	Not implemented in RF 3.0.
end_keyword	N/A	Not implemented in RF 3.0.
log_message	message	Called when an executed keyword writes a log message. message is a model object representing the `logged message <result.Message_>`_. This method is not called if the message has level below the current `threshold level <Log levels_>`_.
message	message	Called when the framework itself writes a syslog_ message. message is same object as with log_message.
library_import	N/A	Not implemented in RF 3.0.
resource_import	N/A	Not implemented in RF 3.0.
variables_import	N/A	Not implemented in RF 3.0.
output_file	path	Called when writing to an `output file`_ is ready. path is an absolute path to the file.
log_file	path	Called when writing to a `log file`_ is ready. path is an absolute path to the file.
report_file	path	Called when writing to a `report file`_ is ready. path is an absolute path to the file.
xunit_file	path	Called when writing to an `xunit file`_ is ready. path is an absolute path to the file.
debug_file	path	Called when writing to a `debug file`_ is ready. path is an absolute path to the file.
close		Called when the whole test execution ends. With library listeners called when the library goes out of scope.

Listeners logging

Robot Framework offers a `programmatic logging APIs`_ that listeners can utilize. There are some limitations, however, and how different listener methods can log messages is explained in the table below.

How listener methods can log

Methods	Explanation
start_keyword, end_keyword, log_message	Messages are logged to the normal `log file`_ under the executed keyword.
start_suite, end_suite, start_test, end_test	Messages are logged to the syslog_. Warnings are shown also in the `execution errors`_ section of the normal log file.
message	Messages are normally logged to the syslog. If this method is used while a keyword is executing, messages are logged to the normal log file.
Other methods	Messages are only logged to the syslog.

注釈

To avoid recursion, messages logged by listeners are not sent to listener methods log_message and message.

Listener examples

This section contains examples using the listener interface. There are first examples that just receive information from Robot Framework and then examples that modify executed tests and created results.

Getting information

The first example is implemented as Python module and uses the listener version 2.

"""Listener that stops execution if a test fails."""

ROBOT_LISTENER_API_VERSION = 2

def end_test(name, attrs):
    if attrs['status'] == 'FAIL':
        print 'Test "%s" failed: %s' % (name, attrs['message'])
        raw_input('Press enter to continue.')

If the above example would be saved to, for example, PauseExecution.py file, it could be used from the command line like this:

robot --listener path/to/PauseExecution.py tests.robot

The same example could also be implemented also using the newer listener version 3 and used exactly the same way from the command line.

"""Listener that stops execution if a test fails."""

ROBOT_LISTENER_API_VERSION = 3

def end_test(data, result):
    if not result.passed:
        print 'Test "%s" failed: %s' % (result.name, result.message)
        raw_input('Press enter to continue.')

The next example, which still uses Python, is slightly more complicated. It writes all the information it gets into a text file in a temporary directory without much formatting. The filename may be given from the command line, but also has a default value. Note that in real usage, the `debug file`_ functionality available through the command line option --debugfile is probably more useful than this example.

import os.path
import tempfile


class PythonListener:
    ROBOT_LISTENER_API_VERSION = 2

    def __init__(self, filename='listen.txt'):
        outpath = os.path.join(tempfile.gettempdir(), filename)
        self.outfile = open(outpath, 'w')

    def start_suite(self, name, attrs):
        self.outfile.write("%s '%s'\n" % (name, attrs['doc']))

    def start_test(self, name, attrs):
        tags = ' '.join(attrs['tags'])
        self.outfile.write("- %s '%s' [ %s ] :: " % (name, attrs['doc'], tags))

    def end_test(self, name, attrs):
        if attrs['status'] == 'PASS':
            self.outfile.write('PASS\n')
        else:
            self.outfile.write('FAIL: %s\n' % attrs['message'])

     def end_suite(self, name, attrs):
         self.outfile.write('%s\n%s\n' % (attrs['status'], attrs['message']))

     def close(self):
         self.outfile.close()

The following example implements the same functionality as the previous one, but uses Java instead of Python.

import java.io.*;
import java.util.Map;
import java.util.List;


public class JavaListener {
    public static final int ROBOT_LISTENER_API_VERSION = 2;
    public static final String DEFAULT_FILENAME = "listen_java.txt";
    private BufferedWriter outfile = null;

    public JavaListener() throws IOException {
        this(DEFAULT_FILENAME);
    }

    public JavaListener(String filename) throws IOException {
        String tmpdir = System.getProperty("java.io.tmpdir");
        String sep = System.getProperty("file.separator");
        String outpath = tmpdir + sep + filename;
        outfile = new BufferedWriter(new FileWriter(outpath));
    }

    public void startSuite(String name, Map attrs) throws IOException {
        outfile.write(name + " '" + attrs.get("doc") + "'\n");
    }

    public void startTest(String name, Map attrs) throws IOException {
        outfile.write("- " + name + " '" + attrs.get("doc") + "' [ ");
        List tags = (List)attrs.get("tags");
        for (int i=0; i < tags.size(); i++) {
           outfile.write(tags.get(i) + " ");
        }
        outfile.write(" ] :: ");
    }

    public void endTest(String name, Map attrs) throws IOException {
        String status = attrs.get("status").toString();
        if (status.equals("PASS")) {
            outfile.write("PASS\n");
        }
        else {
            outfile.write("FAIL: " + attrs.get("message") + "\n");
        }
    }

    public void endSuite(String name, Map attrs) throws IOException {
        outfile.write(attrs.get("status") + "\n" + attrs.get("message") + "\n");
    }

    public void close() throws IOException {
        outfile.close();
    }
}

Modifying execution and results

These examples illustrate how to modify the executed tests and suites as well as the execution results. All these examples require using the listener version 3.

Modifying executed suites and tests

Changing what is executed requires modifying the model object containing the executed `test suite <running.TestSuite_>`_ or `test case <running.TestCase_>`_ objects passed as the first argument to start_suite and start_test methods. This is illustrated by the example below that adds a new test to each executed test suite and a new keyword to each test.

ROBOT_LISTENER_API_VERSION = 3

def start_suite(suite, result):
    suite.tests.create(name='New test')

def start_test(test, result):
    test.keywords.create(name='Log', args=['Keyword added by listener!'])

Trying to modify execution in end_suite or end_test methods does not work, simply because that suite or test has already been executed. Trying to modify the name, documentation or other similar metadata of the current suite or test in start_suite or start_test method does not work either, because the corresponding result object has already been created. Only changes to child tests or keywords actually have an effect.

This API is very similar to the `pre-run modifier`_ API that can be used to modify suites and tests before the whole test execution starts. The main benefit of using the listener API is that modifications can be done dynamically based on execution results or otherwise. This allows, for example, interesting possibilities for model based testing.

Although the listener interface is not built on top of Robot Framework's internal `visitor interface`_ similarly as the pre-run modifier API, listeners can still use the visitors interface themselves. For example, the SelectEveryXthTest visitor used in `pre-run modifier`_ examples could be used like this:

from SelectEveryXthTest import SelectEveryXthTest

ROBOT_LISTENER_API_VERSION = 3

def start_suite(suite, result):
    selector = SelectEveryXthTest(x=2)
    suite.visit(selector)

Modifying results

Test execution results can be altered by modifying `test suite <result.TestSuite_>`_ and `test case <result.TestCase_>`_ result objects passed as the second argument to start_suite and start_test methods, respectively, and by modifying the `message <result.Message_>`_ object passed to the log_message method. This is demonstrated by the following listener that is implemented as a class.

class ResultModifier(object):
    ROBOT_LISTENER_API_VERSION = 3

    def __init__(self, max_seconds=10):
        self.max_milliseconds = float(max_seconds) * 1000

   def start_suite(self, data, suite):
       suite.doc = 'Documentation set by listener.'
       # Information about tests only available via data at this point.
       smoke_tests = [test for test in data.tests if 'smoke' in test.tags]
       suite.metadata['Smoke tests'] = len(smoke_tests)

    def end_test(self, data, test):
        if test.status == 'PASS' and test.elapsedtime > self.max_milliseconds:
            test.status = 'FAIL'
            test.message = 'Test execution took too long.'

    def log_message(self, msg):
        if msg.level == 'WARN' and not msg.html:
            msg.message = '<b style="font-size: 1.5em">%s</b>' % msg.message
            msg.html = True

A limitation is that modifying the name of the current test suite or test case is not possible because it has already been written to the `output.xml`_ file when listeners are called. Due to the same reason modifying already finished tests in the end_suite method has no effect either.

This API is very similar to the `pre-Rebot modifier`_ API that can be used to modify results before report and log are generated. The main difference is that listeners modify also the created output.xml file.

Test libraries as listeners

Sometimes it is useful also for `test libraries`_ to get notifications about test execution. This allows them, for example, to perform certain clean-up activities automatically when a test suite or the whole test execution ends.

注釈

This functionality is new in Robot Framework 2.8.5.

Registering listener

A test library can register a listener by using ROBOT_LIBRARY_LISTENER attribute. The value of this attribute should be an instance of the listener to use. It may be a totally independent listener or the library itself can act as a listener. To avoid listener methods to be exposed as keywords in the latter case, it is possible to prefix them with an underscore. For example, instead of using end_suite or endSuite, it is possible to use _end_suite or _endSuite.

Following examples illustrates using an external listener as well as library acting as a listener itself:

import my.project.Listener;

public class JavaLibraryWithExternalListener {
    public static final Listener ROBOT_LIBRARY_LISTENER = new Listener();
    public static final String ROBOT_LIBRARY_SCOPE = "GLOBAL";
    public static final int ROBOT_LISTENER_API_VERSION = 2;

    // actual library code here ...
}

class PythonLibraryAsListenerItself(object):
    ROBOT_LIBRARY_SCOPE = 'TEST SUITE'
    ROBOT_LISTENER_API_VERSION = 2

    def __init__(self):
        self.ROBOT_LIBRARY_LISTENER = self

    def _end_suite(self, name, attrs):
        print 'Suite %s (%s) ending.' % (name, attrs['id'])

    # actual library code here ...

As the seconds example above already demonstrated, library listeners have to specify listener interface versions using ROBOT_LISTENER_API_VERSION attribute exactly like any other listener.

Starting from version 2.9, you can also provide any list like object of instances in the ROBOT_LIBRARY_LISTENER attribute. This will cause all instances of the list to be registered as listeners.

Called listener methods

Library's listener will get notifications about all events in suites where the library is imported. In practice this means that start_suite, end_suite, start_test, end_test, start_keyword, end_keyword, log_message, and message methods are called inside those suites.

If the library creates a new listener instance every time when the library itself is instantiated, the actual listener instance to use will change according to the `test library scope`_. In addition to the previously listed listener methods, close method is called when the library goes out of the scope.

See Listener interface methods section above for more information about all these methods.