Merge lp://qastaging/~nskaggs/ubuntu-test-cases/increase-ap-timeout into lp://qastaging/ubuntu-test-cases/touch
Status: | Merged | ||||
---|---|---|---|---|---|
Merged at revision: | 304 | ||||
Proposed branch: | lp://qastaging/~nskaggs/ubuntu-test-cases/increase-ap-timeout | ||||
Merge into: | lp://qastaging/ubuntu-test-cases/touch | ||||
Diff against target: |
11 lines (+1/-0) 1 file modified
scripts/run-autopilot-tests.sh (+1/-0) |
||||
To merge this branch: | bzr merge lp://qastaging/~nskaggs/ubuntu-test-cases/increase-ap-timeout | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Leo Arias (community) | Approve | ||
Review via email: mp+235706@code.qastaging.launchpad.net |
This proposal supersedes a proposal from 2014-09-23.
Commit message
Increases the default timeout for autopilot test runs to 'long', which is currently defined as 30 seconds.
Description of the change
This increases the default timeout for autopilot test runs to 'long', which is currently defined as 30 seconds. This is intended to prevent bug 369990, but it will affect all timeouts for autopilot, including wait_select's, which are currently timing out after the default timeout of 10 seconds. This could cause tests to take an additional 20 seconds to fail (when they do).
Test case authors that require or expect the default timeout would also be affected. In general waiting longer shouldn't impose any additional risk beyond taking longer to fail (any assertions made on a specific time should already be defined inside a test). This is somewhat a heavy handed approach towards fixing the bug, and if possible perhaps it could be applied on a individual testsuite basis.
The underlying causes for why applications don't launch within 10 seconds is concerning and worthy of study, but that is outside of the scope of running these tests. I would rather see more reflective results of the application than using it as a proxy to potential app startup performance concerns. As to tests not reflecting reality and hiding long startup issues, I would only point out the apps incur some additional overhead to load with autopilot (including memory), and thus take longer to launch than launching an application without introspection. Thus this is not reflective of a real world user experience from the start.
If the QA team is happy that this doesn't cause unintended side effects, then I'm happy to merge it. Are we sure this isn't just masking a real bug that recently came up though? Have you been able to prove if this actually helps the situation?