Merge lp://qastaging/~glcompbench-dev/glcompbench/manmower-options-3 into lp://qastaging/glcompbench

Based on suggestions by Jesse, I've pushed some changes:
* Call eglSwapInterval only if --enable-vsync wasn't requestes.
* Moved standard deviation variable to the scope it's used in.
* Use Util::split in libmatrix instead of strtok.

I am still confused about why we need to have multiple substats. In the current code base the SUBSET group ends up holding exactly the same data as the TOTAL group (i.e. Profile::sample_point() samples both substat groups and it's the only public method to sample points).

In case such functionality is indeed needed, I believe that it could be achieved more cleanly by using multiple Profiler instances to hold different data sets.

> I am still confused about why we need to have multiple substats. In the
> current code base the SUBSET group ends up holding exactly the same data as
> the TOTAL group (i.e. Profile::sample_point() samples both substat groups and
> it's the only public method to sample points).

The subsets are temporal - the TOTAL group is quite different from SUBSET after the first iteration. The SUBSET group is what was previously tracked.

The TOTAL group is what I use for pass/fail tests in later commits in the series.

Both contain results accumulated from the same profiler points, but TOTAL is accumulated over the total duration of a test.

> In case such functionality is indeed needed, I believe that it could be
> achieved more cleanly by using multiple Profiler instances to hold different
> data sets.

I think that would result in more convoluted code everywhere that actually uses a profiler point - doubling up every profiler point occurrence via cut and paste, and making some of the resets conditional.

If that's what you'd prefer, I can write it that way instead... Please let me know

Thanks for the explanation, it makes much more sense now. You are right that for this use case using separate Profiler instances would be messy.

Based on your patch I have implemented what is essentially a more generic version of your proposal:

https://code.launchpad.net/~linaro-graphics-wg/glcompbench/profiler-stat-collection

The current code in the branch is not complete yet. It only tracks per-iteration statistics, but adding per-test and total statistics is easy. The code is also missing the 'squares' stats datum, but this is trivial to add, too.

In your proposal the total stats are displayed after each test, but they are not really per-test stats, which I find somewhat confusing. I think it would be better to show per-test stats after each test is completed and total stats only once at the end.

Also, I would rather keep the accumulation logic outside of the profiler class. It's easy to calculate accumulated data outside the profiler, given the statistical data in the 'total' collection (calculating std. dev. needs a bit more care).

What do you think?

Meanwhile I will merge into trunk some of the other changes that don't depend on the substats infrastructure.

> Based on your patch I have implemented what is essentially a more generic
> version of your proposal:
>
> https://code.launchpad.net/~linaro-graphics-wg/glcompbench/profiler-stat-
> collection

Very cool :)

> The current code in the branch is not complete yet. It only tracks per-
> iteration statistics, but adding per-test and total statistics is easy. The
> code is also missing the 'squares' stats datum, but this is trivial to add,
> too.
>
> In your proposal the total stats are displayed after each test, but they are
> not really per-test stats, which I find somewhat confusing. I think it would
> be better to show per-test stats after each test is completed and total stats
> only once at the end.

Yes, that sounds quite sensible.

> Also, I would rather keep the accumulation logic outside of the profiler
> class. It's easy to calculate accumulated data outside the profiler, given the
> statistical data in the 'total' collection (calculating std. dev. needs a bit
> more care).
>
> What do you think?

It's not immediately obvious to me how I'd calculate std. dev. anymore. I could easily calculate the std. dev. of individual profiler points, but since there aren't the same number of each profiler point in an iteration, I can't figure out how to calculate std. dev. of test iterations, which is the more useful stat from my perspective.

Though perhaps there's a way that I'm just not seeing?

> Meanwhile I will merge into trunk some of the other changes that don't depend
> on the substats infrastructure.

Thanks!