Launchpad itself

Merge lp://qastaging/~michael.nelson/launchpad/393546-avoid-builder-monopolization into lp://qastaging/launchpad

  1. 393546-avoid-builder-monopolization
  2. Merge into devel
Proposed by Michael Nelson
Status: Merged
Approved by: Eleanor Berger
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp://qastaging/~michael.nelson/launchpad/393546-avoid-builder-monopolization
Merge into: lp://qastaging/launchpad
Diff against target: 143 lines
2 files modified
lib/lp/soyuz/doc/buildd-dispatching.txt (+38/-9)
lib/lp/soyuz/model/builder.py (+18/-2)
To merge this branch: bzr merge lp://qastaging/~michael.nelson/launchpad/393546-avoid-builder-monopolization
Reviewer Review Type Date Requested Status
Eleanor Berger (community) code Approve
Review via email: mp+13230@code.qastaging.launchpad.net
To post a comment you must log in.
Revision history for this message
Michael Nelson (michael.nelson) wrote :

= Summary =

Fix bug 393546 by ensuring that when considering the next candidate for
a builder - we ignore builds for PPAs that already have builds of the
same architecture currently building.

== Proposed fix ==

Add a correllated sub-query that ensures - when considering a build -
that there are no other currently-building builds for the same PPA
archive and architecture.

== Pre-implementation notes ==

See notes on bug 393546.

== Implementation details ==

The _findBuildCandidate() method is an ugly sqlobject query - I didn't
consider updating it - but if I have to I will :)

I'm wondering whether we should restrict the exclusion to
currently-building builds with a lower score than the build being
considered? (ie. so if a new build on the queue for the same
archive/arch has a higher score it will still be considered).

If we do add this then I'll most definitely have to add a separate
test_builder.py to unit test this (as the documentation is already
getting ugly with setup etc.).

== Tests ==

bin/test -vvt buildd-dispatching.txt

== Demo and Q/A ==

None.

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/lp/soyuz/model/builder.py
  lib/lp/soyuz/doc/buildd-dispatching.txt

--
Michael

Revision history for this message
Eleanor Berger (intellectronica) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/soyuz/doc/buildd-dispatching.txt'
2--- lib/lp/soyuz/doc/buildd-dispatching.txt 2009-08-28 06:39:38 +0000
3+++ lib/lp/soyuz/doc/buildd-dispatching.txt 2009-10-12 16:10:27 +0000
4@@ -202,15 +202,20 @@
5
6 == PPA build dispatching ==
7
8-Create new Build record of the same source targeted for a PPA archive:
9+Create two Build records of the same source targeted for a PPA archive:
10
11 >>> from lp.registry.interfaces.person import IPersonSet
12 >>> cprov = getUtility(IPersonSet).getByName('cprov')
13+ >>> warty = getUtility(IDistributionSet)['ubuntu']['warty']
14+ >>> warty_i386 = warty['i386']
15
16+ # One for hoary and one for warty.
17 >>> ppa_build = sprf.sourcepackagerelease.createBuild(
18 ... hoary_i386, PackagePublishingPocket.RELEASE, cprov.archive)
19+ >>> ppa_build_2 = sprf.sourcepackagerelease.createBuild(
20+ ... warty_i386, PackagePublishingPocket.RELEASE, cprov.archive)
21
22-Create BuildQueue record and inspect some parameters:
23+Create BuildQueue records and inspect some parameters:
24
25 >>> ppa_job = ppa_build.createBuildQueueEntry()
26 >>> ppa_job.id
27@@ -219,6 +224,9 @@
28 True
29 >>> ppa_job.buildstart == None
30 True
31+ >>> ppa_job_2 = ppa_build_2.createBuildQueueEntry()
32+ >>> ppa_job_2.id
33+ 4
34
35 The build job's archive requires virtualized builds.
36
37@@ -240,10 +248,10 @@
38 >>> print job
39 None
40
41-In order to enable 'bob' to find and build the PPA job, we have to
42+In order to enable 'bob' to find and build the PPA jobs, we have to
43 change it to virtualized. This is because PPA builds will only build
44-on virtualized builders. We also need to make sure this build's source
45-is published, or it will also be ignored (by superseding it). We can
46+on virtualized builders. We also need to make sure the builds' sources
47+are published, or they will also be ignored (by superseding it). We can
48 do this by copying the existing publication in Ubuntu.
49
50 >>> from lp.soyuz.model.publishing import (
51@@ -253,6 +261,11 @@
52 ... sourcepackagerelease=ppa_job.build.sourcepackagerelease)
53 >>> new_pub = old_pub.copyTo(
54 ... old_pub.distroseries, old_pub.pocket, ppa_job.build.archive)
55+ >>> [old_pub_2] = SourcePackagePublishingHistory.selectBy(
56+ ... distroseries=ppa_job_2.build.distroseries,
57+ ... sourcepackagerelease=ppa_job_2.build.sourcepackagerelease)
58+ >>> new_pub_2 = old_pub_2.copyTo(
59+ ... old_pub_2.distroseries, old_pub_2.pocket, ppa_job_2.build.archive)
60
61 >>> bob_builder.virtualized = True
62 >>> syncUpdate(bob_builder)
63@@ -296,14 +309,30 @@
64 >>> ppa_job.buildstart == get_transaction_timestamp()
65 True
66
67+At this point, the second build job - for the same archive and architecture -
68+will not be found as a build candidate for other builders. This is to
69+avoid nightly builds using all the builders for an arch at once.
70+
71+ >>> print frog_builder.findBuildCandidate()
72+ None
73+
74+But once the first build (for the same archive and architecture) finishes,
75+the second build will be available as a build candidate for both builders.
76+
77+ >>> ppa_job.build.buildstate = BuildStatus.FAILEDTOBUILD
78+ >>> ppa_job.destroySelf()
79+ >>> print frog_builder.findBuildCandidate().id
80+ 4
81+ >>> print bob_builder.findBuildCandidate().id
82+ 4
83+
84 Shutdown builder slave, mark the ppa build record as failed, remove the
85 buildqueue record and make 'bob' builder non-virtual again, so the
86 environment is back to the initial state.
87
88 >>> BuilddSlaveTestSetup().tearDown()
89-
90- >>> ppa_job.build.buildstate = BuildStatus.FAILEDTOBUILD
91- >>> ppa_job.destroySelf()
92+ >>> ppa_job_2.build.buildstate = BuildStatus.FAILEDTOBUILD
93+ >>> ppa_job_2.destroySelf()
94 >>> bob_builder.virtualized = False
95 >>> flush_database_updates()
96
97@@ -326,7 +355,7 @@
98
99 >>> sec_job = sec_build.createBuildQueueEntry()
100 >>> sec_job.id
101- 4
102+ 5
103 >>> print sec_job.builder
104 None
105 >>> print sec_job.buildstart
106
107=== modified file 'lib/lp/soyuz/model/builder.py'
108--- lib/lp/soyuz/model/builder.py 2009-09-04 07:29:56 +0000
109+++ lib/lp/soyuz/model/builder.py 2009-10-12 16:10:27 +0000
110@@ -560,8 +560,8 @@
111 build.buildstate = %s AND
112 distroarchseries.processorfamily = %s AND
113 buildqueue.builder IS NULL
114- """ % sqlvalues(private_statuses,
115- BuildStatus.NEEDSBUILD, self.processor.family)]
116+ """ % sqlvalues(
117+ private_statuses, BuildStatus.NEEDSBUILD, self.processor.family)]
118
119 clauseTables = ['Build', 'DistroArchSeries', 'Archive']
120
121@@ -569,6 +569,22 @@
122 archive.require_virtualized = %s
123 """ % sqlvalues(self.virtualized))
124
125+ # Ensure that if a currently-building build exists for the same
126+ # ppa archive and architecture currently building then we don't
127+ # consider another as a candidate.
128+ clauses.append("""
129+ NOT EXISTS (
130+ SELECT Build.id
131+ FROM Build build2, DistroArchSeries distroarchseries2
132+ WHERE
133+ build2.archive = build.archive AND
134+ archive.purpose = %s AND
135+ build2.distroarchseries = distroarchseries2.id AND
136+ distroarchseries2.processorfamily = %s AND
137+ build2.buildstate = %s)
138+ """ % sqlvalues(
139+ ArchivePurpose.PPA, self.processor.family, BuildStatus.BUILDING))
140+
141 query = " AND ".join(clauses)
142
143 candidate = BuildQueue.selectFirst(