Merge lp://qastaging/~vanvugt/mir/compositor-test into lp://qastaging/mir

PASSED: Continuous integration, rev:3484
https://mir-jenkins.ubuntu.com/job/mir-ci/890/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/922
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/961
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/952
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/952
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/932
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/932/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/932
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/932/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/932
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/932/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/932
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/932/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/932
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/932/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/890/rebuild

Should be made a bit more flexible by allowing these params passed via the command line.

89 + mir_demo_client_egltriangle
90 + mir_demo_client_egltriangle
91 + mir_demo_client_egltriangle
92 + mir_demo_client_egltriangle
93 + mir_demo_client_egltriangle
94 + mir_demo_client_flicker
95 + mir_demo_client_progressbar
96 + mir_demo_client_scroll
97 + mir_demo_client_flicker

141 +verify '$frame_rate \>= 55'
142 +verify '$render_time \<= 17'

+$bin_dir/mir_demo_server -f $mir_sock --window-manager=tiling --compositor-report=log > $server_log &

Wow! A use of "--window-manager=tiling". Maybe I shouldn't be proposing to remove it?

What is the motivation for writing this in bash?

It looks like something easy to do using the google test framework we use widely.

Alternatively, there's the python based mir_perf_framework.

Do we really need a third approach?

> Should be made a bit more flexible by allowing these params passed via the
> command line.

> 141 +verify '$frame_rate \>= 55'
> 142 +verify '$render_time \<= 17'

+1, esp the parameters. Not too unreasonable that we might see an android device that's got a goofy vsync (like krillin)

I have considered those things already, so should explain...

Bourne shell: is much more appropriate for simple command-line driven tests. And there will be more in future that are simple external commands, not C/C+ code. More like system tests.

gtest:
https://code.launchpad.net/~vanvugt/mir/generalise-performance-tests/+merge/292600/comments/750962

Parameters: that's actually not important and even not sensible. Because a regression test should be difficult to break. Making the threshold for your regression test a parameter makes it too easy to pass the test by accident.

krillin: Actually when bug 1563287 is present, krillin drops to 20Hz (and the test fails), so a threshold of 55 is fine.

mir_performance_tests: Is the only performance test using real graphics we already have active in CI already. Admittedly it's only run on krillin (see bug 1566743). And looking forward it is grammatically most appropriate that the command 'mir_performance_tests' should run multiple performance tests.

BTW -- I did start out trying to use GTest and threw it away after realizing all of the above.

tiling: I would happily use any old Mir shell, but to ensure there's no occlusion logic active (which would skew performance results) would like to see a shell that cascades windows by default instead. So a little bit of each client must always be visible.

FAILED: Continuous integration, rev:3494
https://mir-jenkins.ubuntu.com/job/mir-ci/933/
Executed test runs:
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-mir/994/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1040
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1031
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1031
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1004
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1004/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1004
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1004/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1004
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1004/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1004
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1004/artifact/output/*zip*/output.zip
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1004/console

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/933/rebuild

Looks like the sort of thing that would be better in a gtest executable; the processing is at the limit of what I'm comfortable bashing.

That would also let you ensure that clients are laid out as expected, replace the “sleep x” with events, and make it easy to add extra checks, like variance and such.

Incorrect...

A gtest executable would not only be more complex (I have tried) but is also impossible right now without forking off the server process in the same way a shell script does. See bug 1546912.

Also, the sleep should not be replaced because it's a real-time delay you need to put the clients visually out of phase.

PASSED: Continuous integration, rev:3495
https://mir-jenkins.ubuntu.com/job/mir-ci/937/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/999
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1045
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1036
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1036
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1009
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1009/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1009
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1009/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1009
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1009/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1009
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1009/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1009
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1009/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/937/rebuild

To which sleep do you refer? The one between sending SIGTERM to the server and sending SIGKILL? Or the one between forking the server and trying to see if the socket is up? Or the one between killing the clients and killing the server? :)

That crash should be fairly easy to work around by pre-forking the children, right?

But, to reiterate, I'm not blocking on doing that.

I was referring to the sleep between clients starting. Although now I've changed the sizes of the triangles that one is not necessary. But it does help visually inform the observer what's going on though... The fact remains this new test takes less time than the existing performance test. I could shave some seconds off, but don't think it matters that much.

I am still not convinced that a gtest executable would be more complex (I haven't tried), nor that e.g. using popen() to detect progress would be less reliable than sleeps.

I am also concerned that we created the mir_perf_framework for performance testing, but appear determined not to use it.

But we do need to test this.

My other reason for avoiding gtest (other than it taking more time, more code, and having to work around crashes on android) is that gtest doesn't like you forking. But I might try and prototype it today anyway...

https://code.launchpad.net/~vanvugt/mir/compositor-test-in-c-minor/+merge/293986