i3 testsuite

The i3 testsuite is a collection of files which contain testcases for various i3 features. Some of them test if a certain workflow works correctly (moving windows, focus behaviour, …). Others are regression tests and contain code which previously made i3 crash or lead to unexpected behaviour. They then check if i3 still runs (meaning it did not crash) and if it handled everything correctly.

The goal of having these tests is to automatically find problems and to automatically get a feel for whether a change in the source code breaks any existing feature. After every modification of the i3 sourcecode, the developer should run the full testsuite. If one of the tests fails, the corresponding problem should be fixed (or, in some cases, the testcase has to be modified). For every bugreport, a testcase should be written to test the correct behaviour. Initially, it will fail, but after fixing the bug, it will pass. This ensures (or increases the chance) that bugs which have been fixed once will never be found again.

Also, when implementing a new feature, a testcase might be a good way to be able to easily test if the feature is working correctly. Many developers will test manually if everything works. Having a testcase not only helps you with that, but it will also be useful for every future change.

For several reasons, the i3 testsuite has been implemented in Perl:

Please do not start programming language flamewars at this point.

Learning by example is definitely a good strategy when you are wondering how to write a testcase. Let’s take t/11-goto.t as an easy example and go through it step by step:

This is what we call boilerplate. It exists at the top of every test file (to some extent). The first line is the shebang, which specifies that this file is a Perl script. The second line contains VIM specific settings on how to edit/format this file (use spaces instead of tabs, indent using 4 spaces). Afterwards, the i3test module is used. This module contains i3 testsuite specific functions which you are strongly encouraged to use. They make writing testcases a lot easier and will make it easier for other people to read your tests.

The next line uses the File::Temp module. This is specific to this testcase, because it needs to generate a temporary name during the test. Many testcases use only the i3test module.

The last line opens a connection to X11. You might or might not need this in your testcase, depending on whether you are going to open windows (etc.) or only use i3 commands.

The first line calls i3test’s fresh_workspace function which looks for a currently unused workspace, switches to it, and returns its name. The variable $tmp will end up having a value such as "/tmp/87kBVcHbA9". Note that this is not (necessarily) a valid path, it’s just a random workspace name.

So, now that we are on a new workspace, we ensure that the workspace uses horizontal orientation by issuing the split h command (see the i3 User’s Guide for a list of commands). This is not strictly necessary, but good style. In general, the cmd function executes the specified i3 command by using the IPC interface and returns once i3 acknowledged the command.

In every major section of a testcase, you should put a comment like the one above. This makes it immediately clear how the file is structured.

The open_window function opens a standard window, which will then be put into tiling mode by i3. If you want a floating window, use the open_floating_window function. These functions accept the same parameters as X11::XCB::Window→new, see the i3test documentation at TODO.

This section defines a helper function which will be used over and over in this testcase. If you have code which gets executed more than once or twice (depending on the length of your test, use your best judgement), please put it in a function. Tests should be short, concise and clear.

The focus_after function executes a command and returns the X11 focus after the command was executed. The sync_with_i3 command makes sure that i3 could push its state to X11. See [i3_sync] to learn how this works exactly.

Now, we run the first two real tests. They use Test::More's is function, which compares two values and prints the differences if they are not the same. After the arguments, we supply a short comment to indicate what we are testing here. This makes it vastly more easy for the developer to spot which testcase is the problem in case one fails.

The first test checks that the most recently opened window is focused. Afterwards, the command focus left is issued and it is verified that the middle window now has focus.

Note that this is not a comprehensive test of the focus command — we would have to test wrapping, focus when using a more complex layout, focusing the parent/child containers, etc. But that is not the point of this testcase. Instead, we just want to know if $x→input_focus corresponds with what we are expecting. If not, something is completely wrong with the test environment and this trivial test will fail.

Syntax hint: The qq keyword is the interpolating quote operator. It lets you chose a quote character (in this case the | character, a pipe). This makes having double quotes in our string easy.

In this new major section, a random mark (mark is an identifier for a window, see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we test that trying to focus that mark will not do anything. This is important: Do not only test that using a feature has the expected outcome, but also test that using it without properly initializing it does no harm. This command could for example have changed focus anyways (a bug) or crash i3 (obviously a bug).

Remember: Focus was on the middle window (we verified that earlier in "Test assumptions"). We now mark the middle window with our randomly generated mark. Afterwards, we switch focus away from the middle window to be able to tell if focusing it via its mark will work. If the test works, the goto command seems to be working.

Now we test the same feature, but specifying the mark twice in the command. This should have no effect, but let’s be sure: test it and see if things go wrong.

This part of the test checks that focusing windows by mark works across workspaces. It uses i3test’s focused_ws function to get the current workspace.

The end of every testcase has to contain the done_testing line. This tells complete-run.pl that the test was finished successfully. If it does not occur, the test might have crashed during execution — some of the reasons why that could happen are bugs in the used modules, bugs in the testcase itself or an i3 crash resulting in the testcase being unable to communicate with i3 via IPC anymore.

Consider the following situation: You open two windows in your testcase, then you use focus left and want to verify that the X11 focus has been updated properly. Sounds simple, right? Let’s assume you use this straight-forward implementation:

However, the test fails. Sometimes. Apparantly, there is a race condition in your test. If you think about it, this is because you are using two different pieces of software: You tell i3 to update focus, i3 confirms that, and then you ask X11 to give you the current focus. There is a certain time i3 needs to update the X11 state. If the testcase gets CPU time before X11 processed i3’s requests, the test will fail.

One way to "solve" this would be to add sleep 0.5; after the cmd call. After 0.5 seconds it should be safe to assume that focus has been updated, right?

In practice, this usually works. However, it has several problems:

To illustrate how grave the problem with wasting time actually is: Before removing all sleeps from the testsuite, a typical run using 4 separate X servers took around 50 seconds on my machine. After removing all the sleeps, we achieved times of about 25 seconds. This is very significant and influences the way you think about tests — the faster they are, the more likely you are to check whether everything still works quite often (which you should).

What I am trying to say is: Delays adds up quickly and make the test suite less robust.

The real solution for this problem is a mechanism which I call "the i3 sync protocol". The idea is to send a request (which does not modify state) via X11 to i3 which will then be answered. Due to the request’s position in the event queue (after all previous events), you can be sure that by the time you receive the reply, all other events have been dealt with by i3 (and, more importantly, X11).

Socket activation is a mechanism which was made popular by systemd, an init replacement. It basically describes creating a listening socket before starting a program. systemd will invoke the program only when an actual connection to the socket is made, hence the term socket activation.

The interesting part of this (in the i3 context) is that you can very precisely detect when the program is ready (finished its initialization).