Visualization of Regression and Performance

After updating OpenBSD you might see that something does not work as it used to be. To simplify debugging it is helpful to determine the point in time when the change was introduced and search for the relevant commit.

For functional regressions a test suite that is executed on a daily basis is sufficient. By providing recent test results, the relevant day can be seen and the responsible developer of that area is informed. The OpenBSD regress results can be seen here. http://bluhm.genua.de/regress/results/regress.html

Making statements about performance is more difficult. The requirements for measurements may change, new test programs are needed, test hardware is only available for a limited time. So it is not sufficient to store historic data on a daily basis. You want to change granularity or look back into the past.

For that purpose I have created a system that can create a performance measurement of an OpenBSD kernel in the past. The kernel is compiled from a certain CVS checkout. That may sound easier than it is as the OpenBSD kernel is not self contained. It belongs to a base system, there may be incompatibilities with userland. The performance also depends on the compiler version that changes over an OpenBSD development cycle.

My solution installs an OpenBSD release and updates the kernel from source in fixed time steps. If there is a relevant user land change within the step, the necessary parts of the system are updated. This allows quick progress while compiling the kernel without rebuilding the whole system. With each new kernel the performance is measured.

An overview of all tests results is here: http://bluhm.genua.de/perform/results/perform.html The results of the whole 6.3 release with daily kernel checkouts is displayed here: http://bluhm.genua.de/perform/results/2018-12-24T13%3A28%3A41Z/perform.html The talk will explain how this was implemented and what results where discovered.