Merge lp://qastaging/~julian-edwards/launchpad/async-file-uploads-bug-662631 into lp://qastaging/launchpad/db-devel
| Status: | Merged |
|---|---|
| Approved by: | Graham Binns |
| Approved revision: | no longer in the source branch. |
| Merged at revision: | 10021 |
| Proposed branch: | lp://qastaging/~julian-edwards/launchpad/async-file-uploads-bug-662631 |
| Merge into: | lp://qastaging/launchpad/db-devel |
| Diff against target: |
1093 lines (+412/-233) 16 files modified
lib/lp/buildmaster/interfaces/builder.py (+1/-1) lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py (+3/-0) lib/lp/buildmaster/interfaces/packagebuild.py (+6/-1) lib/lp/buildmaster/model/builder.py (+63/-37) lib/lp/buildmaster/model/buildfarmjobbehavior.py (+2/-3) lib/lp/buildmaster/model/packagebuild.py (+107/-84) lib/lp/buildmaster/tests/mock_slaves.py (+20/-11) lib/lp/buildmaster/tests/test_builder.py (+45/-1) lib/lp/buildmaster/tests/test_packagebuild.py (+35/-28) lib/lp/code/model/sourcepackagerecipebuild.py (+6/-4) lib/lp/code/model/tests/test_sourcepackagerecipebuild.py (+3/-1) lib/lp/soyuz/tests/test_binarypackagebuild.py (+3/-1) lib/lp/soyuz/tests/test_binarypackagebuildbehavior.py (+40/-31) lib/lp/translations/model/translationtemplatesbuildbehavior.py (+47/-19) lib/lp/translations/tests/test_translationtemplatesbuildbehavior.py (+26/-10) lib/lp_sitecustomize.py (+5/-1) |
| To merge this branch: | bzr merge lp://qastaging/~julian-edwards/launchpad/async-file-uploads-bug-662631 |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Graham Binns (community) | code | Approve | |
| Jonathan Lange (community) | Abstain | ||
|
Review via email:
|
|||
Commit message
Make file uploads from builders asynchronous so that it doesn't block the rest of the build manager. This is the last part of the build manager to be made asynchronous.
Description of the change
This is the final part of the buildd-manager changes to make it fully asynchronous. Previously it was blocking when grabbing files from the builders - it now does that asynchronously using Twisted's HTTP file transfer.
Most of the changes are quite mechanical which accounts for most of the diff unfortunately - it's where I need to make the code an inner function which is called back from the Deferred. Hopefully you're familiar with this pattern; if not you'll learn it pretty quickly :)
Other changes are to add a new getFiles() function on the BuilderSlave object. This is initially complementing getFile() by calling it repeatedly but the longer term intention is to remove getFile() so that we can call one place and get hold of everything the builder knows about all at once.
The only other major change was a large amount of work to get the translations code fixed and tests passing. In particular, they were trying to test a successful tarball retrieval but not actually doing so - and in fact one test was asserting that nothing was retrieved!
I don't have time to review this properly, but after having a quick skim through it all looks good.
<jml>bigjools: I don't think you need files_downloaded (in packagebuild.py) you could also do:
d = slave.getFiles(
d.addCallbac
d.addCallbac
return d
Is my only comment of any substance.