Merge lp://qastaging/~nskaggs/ubuntu-test-cases/increase-ap-timeout into lp://qastaging/ubuntu-test-cases/touch

Proposed by Nicholas Skaggs
Status: Merged
Merged at revision: 304
Proposed branch: lp://qastaging/~nskaggs/ubuntu-test-cases/increase-ap-timeout
Merge into: lp://qastaging/ubuntu-test-cases/touch
Diff against target: 11 lines (+1/-0)
1 file modified
scripts/run-autopilot-tests.sh (+1/-0)
To merge this branch: bzr merge lp://qastaging/~nskaggs/ubuntu-test-cases/increase-ap-timeout
Reviewer Review Type Date Requested Status
Leo Arias (community) Approve
Review via email: mp+235706@code.qastaging.launchpad.net

This proposal supersedes a proposal from 2014-09-23.

Commit message

Increases the default timeout for autopilot test runs to 'long', which is currently defined as 30 seconds.

Description of the change

This increases the default timeout for autopilot test runs to 'long', which is currently defined as 30 seconds. This is intended to prevent bug 369990, but it will affect all timeouts for autopilot, including wait_select's, which are currently timing out after the default timeout of 10 seconds. This could cause tests to take an additional 20 seconds to fail (when they do).

Test case authors that require or expect the default timeout would also be affected. In general waiting longer shouldn't impose any additional risk beyond taking longer to fail (any assertions made on a specific time should already be defined inside a test). This is somewhat a heavy handed approach towards fixing the bug, and if possible perhaps it could be applied on a individual testsuite basis.

The underlying causes for why applications don't launch within 10 seconds is concerning and worthy of study, but that is outside of the scope of running these tests. I would rather see more reflective results of the application than using it as a proxy to potential app startup performance concerns. As to tests not reflecting reality and hiding long startup issues, I would only point out the apps incur some additional overhead to load with autopilot (including memory), and thus take longer to launch than launching an application without introspection. Thus this is not reflective of a real world user experience from the start.

To post a comment you must log in.
301. By Nicholas Skaggs

fix unintended whitespace changes

Revision history for this message
Paul Larson (pwlars) wrote :

If the QA team is happy that this doesn't cause unintended side effects, then I'm happy to merge it. Are we sure this isn't just masking a real bug that recently came up though? Have you been able to prove if this actually helps the situation?

Revision history for this message
Leo Arias (elopio) wrote :

I'm happy.

review: Approve
Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

Paul, is is possible for you to kick off a test run of this and link it to help assure us there will not be a regression by introducing this?

A full dashboard run would be lovely, but otherwise, kick off a run of this just using clock, calendar or music which seem to have suffered from this at some point.

Revision history for this message
Paul Larson (pwlars) wrote :
Revision history for this message
Paul Larson (pwlars) wrote :

didn't make it far because it got stuck on the unity8 tests (known problem) but the tests we got through looked somewhat similar to other recent results

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches