Merge lp://qastaging/~zyga/checkbox/selftest into lp://qastaging/checkbox
| Status: | Work in progress |
|---|---|
| Proposed branch: | lp://qastaging/~zyga/checkbox/selftest |
| Merge into: | lp://qastaging/checkbox |
| Diff against target: |
784 lines (+738/-0) 9 files modified
.selftest/manage.py (+20/-0) .selftest/units/coverage.pxu (+11/-0) .selftest/units/enum.pxu (+66/-0) .selftest/units/plans.pxu (+12/-0) .selftest/units/provider.pxu (+59/-0) .selftest/units/python.pxu (+114/-0) .selftest/units/smoke.pxu (+11/-0) .selftest/units/support.pxu (+120/-0) selftest.sh (+325/-0) |
| To merge this branch: | bzr merge lp://qastaging/~zyga/checkbox/selftest |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Maciej Kisielewski (community) | Needs Information | ||
|
Review via email:
|
|||
Description of the change
This branch adds selftest, A standalone script that lets anyone use plainbox to test something.
We're going to use it to test checkbox.
There are some strings attached but I wanted to share it early
- it doesn't yet handle package installation, it's easy to do, just not something I did yesterday
- it doesn't build checkbox-gui
What does it do? It's easier to paste the tail of the output:
=======
☑ : Smoke test for selftest
☑ : Create a bare virtualenv
☑ : Populate virtualenv with additional environment variables
☑ : Clone the external-tarballs repository
☑ : Install all python dependencies
☑ : Meta-job for all bootstrap actions
☑ : Enumerate Python Projects
☑ : Enumerate Plainbox Providers
☑ : Develop Plainbox
☑ : Plainbox Integration Tests
☑ : Develop Checkbox Support
☑ : Develop Checkbox NG
☑ : Plainbox Unit Tests
☑ : Checkbox Support Unit Tests
☑ : Checkbox NG Unit Tests
☑ : Build Plainbox Internationaliz
☑ : Build Checkbox Support Internationaliz
☑ : Build Checkbox NG Internationaliz
☑ : Build Plainbox HTML Documentation
☑ : Build Checkbox NG HTML Documentation
☑ : Build Plainbox Manual Pages
☑ : Develop Checkbox Provider for Plainbox
☑ : Develop Resource Provider for Plainbox
☑ : Develop Ubuntu Touch Provider for Plainbox
☑ : Develop Canonical Client Certification Provider for Plainbox
☑ : Develop Canonical Server Certification Provider for Plainbox
☑ : Validate Resource Provider for Plainbox
☑ : Validate Checkbox Provider for Plainbox
☑ : Validate Ubuntu Touch Provider for Plainbox
☑ : Validate Canonical Client Certification Provider for Plainbox
☑ : Validate Canonical Server Certification Provider for Plainbox
☑ : Build Checkbox Provider for Plainbox
☑ : Build Resource Provider for Plainbox
☑ : Build Ubuntu Touch Provider for Plainbox
☑ : Build Canonical Client Certification Provider for Plainbox
☑ : Build Canonical Server Certification Provider for Plainbox
☑ : Build Checkbox Provider for Plainbox Internationaliz
☑ : Build Resource Provider for Plainbox Internationaliz
☑ : Build Ubuntu Touch Provider for Plainbox Internationaliz
☑ : Build Canonical Client Certification Provider for Plainbox Internationaliz
☑ : Build Canonical Server Certification Provider for Plainbox Internationaliz
09:54 <@zyga> this has a chance to replace 90% of support/
09:56 <@zyga> spineau: my goal is to get rid of all the testing scripts
09:57 <@zyga> spineau: and all the support/ directory
09:57 <@zyga> spineau: and all the cruft we have all over the tree
09:57 <@zyga> spineau: and put it all in the .selftest/ provider
09:57 <@zyga> spineau: and run it just like './selftest.sh' in any environment
09:57 <@zyga> spineau: native, in vagrant, in lxc, etc
09:57 <@zyga> spineau: and switch tarmac to that as well
09:58 <@spineau> zyga: we can relocate the release stuff if needed
09:58 <@zyga> spineau: no, I mean I don't mind keeping the support/ directory
09:58 <@zyga> spineau: just the explosion of scripts there that one has to understand
09:58 <@spineau> zyga: ok, I see
09:58 <@zyga> spineau: it also replaces mk-venv
09:59 <@zyga> spineau: though it's all incomplete, it does 90% of the work but not 100%
09:59 <@zyga> spineau: I wanted to get feedback from you guys
09:59 <@zyga> spineau: so that travis and tarmac and local testing is identical
09:59 <@zyga> spineau: and stuff like PROVIDERPATH not being set because it's hard to do so won't be a problem again
10:00 <@zyga> spineau: e.g. the new validation thing only fails because we have that issue
10:00 <@zyga> spineau: it's not broken, the test env is not setup correctly
10:00 <@zyga> spineau: I think to fully replace what we use today two more things need to happen
10:00 <@zyga> spineau: one is easy, add a few more jobs to install ppas, packages, and build checkbox-gui
10:00 <@zyga> spineau: that's really easy
10:01 <@zyga> spineau: the bigger thing is to improve plainbox so that plainbox run can be given a session directory and run (or re-run) tests in an existing session
10:01 <@zyga> spineau: the idea behind that is that now each ./selftest.sh run is standalone and cretes a new test session
10:01 <@zyga> spineau: but that's slow
10:01 <@zyga> spineau: for incremental tests we really want to reuse the session, reuse prior tess there that setup PLAINBOX_
10:02 <@zyga> spineau: e.g. tarballs, venvs, pip packages and more
10:02 <@zyga> spineau: so I'd like to bake it in trunk
10:02 <@zyga> spineau: and to the switch at around 0.22
10:02 <@zyga> spineau: and in the meantime test it in all the ways we can
| Daniel Manrique (roadmr) wrote | # |
| Zygmunt Krynicki (zyga) wrote | # |
| Zygmunt Krynicki (zyga) wrote | # |
Unmerged revisions
- 3623. By Zygmunt Krynicki
-
selftest: add a provider for selftest
Jobs in the selftest provider are dividied into three categories:
- tests for a given component (e.g. plainbox or checkbox-gui)
- configuration for a given component (e.g. installing required
packages and other kinds of dependencies)
- environment probing jobs that can be used by the earlier two,
those can be used to restrict certain tests to a specific platform
(e.g. don't build checkbox-gui on fedora, in travis CI or on precise)Signed-off-by: Zygmunt Krynicki <email address hidden>
- 3622. By Zygmunt Krynicki
-
selftest: add the selftest tool
This patch adds a new tool, selftest.sh, coupld with any provider
(located in .selftest) for self-testing which replaces the ad-hoc
collection of scripts and tiny data files that we used to have for each
of the components. The provider can be invoked using the selftest.sh
shell script.The script has the following modes of operation:
selftest.sh run - run all tests
(run is also implicit so just running the script is good enough)
selftest.sh bootstrap - setup current environment for testing
(bootstrap is also performed implicitly if needed)
selftest.sh manage - run the management script
selftest.sh shell - spawn a shell where plainbox sees the
local self-test provider
selftest.sh validate - validate the selftest provider
selftest.sh info - display information about the selftest providerSigned-off-by: Zygmunt Krynicki <email address hidden>
Cool! I tried this inside a container, just ran ./selftest.sh run, and it failed :( because I hadn't installed intltool, which goes to what you mentioned about package installation.
Another thing I noticed is that the exit code was 0 even though some of the tests failed, I wonder if we should have a proper non-success exit code for this.
I'm also thinking about whether it overlaps a bit with tools like tox, which we use in capomastro and hexr.
But in all, this is very nice and allows expressing the rather complicated setup for plainbox tests in a manner that's familiar to a developer who is already working with plainbox. Thanks!