work-items-tracker

Merge lp://qastaging/~salgado/launchpad-work-items-tracker/blueprints-over-time into lp://qastaging/~linaro-automation/launchpad-work-items-tracker/linaro

blueprints-over-time
Merge into linaro

Proposed by Guilherme Salgado on 2011-12-15

Status:	Merged
Merged at revision:	316
Proposed branch:	lp://qastaging/~salgado/launchpad-work-items-tracker/blueprints-over-time
Merge into:	lp://qastaging/~linaro-automation/launchpad-work-items-tracker/linaro
Diff against target:	793 lines (+478/-33) 15 files modified collect (+2/-0) collect_roadmap (+9/-0) html-report (+1/-0) lpworkitems/collect_roadmap.py (+26/-0) lpworkitems/database.py (+17/-3) lpworkitems/factory.py (+32/-1) lpworkitems/models.py (+26/-5) lpworkitems/models_roadmap.py (+15/-0) lpworkitems/tests/test_collect.py (+0/-1) lpworkitems/tests/test_collect_roadmap.py (+23/-5) lpworkitems/tests/test_factory.py (+1/-1) lpworkitems/tests/test_models.py (+17/-2) report_tools.py (+46/-15) roadmap-bp-chart (+254/-0) templates/roadmap_lane.html (+9/-0)
To merge this branch:	bzr merge lp://qastaging/~salgado/launchpad-work-items-tracker/blueprints-over-time
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
Mattias Backman (community)		2011-12-15	Approve on 2011-12-15
Review via email: mp+85921@code.qastaging.launchpad.net

Description of the change

This branch adds a new table which stores the number of blueprints per (day, status, lane). This is then used on the roadmap page to show a graph of blueprints per status/day for the quarter.

Revision history for this message

Mattias Backman (mabac) wrote on 2011-12-15:

Download full text (31.5 KiB)

On Thu, Dec 15, 2011 at 6:58 PM, Guilherme Salgado
<email address hidden> wrote:
> Guilherme Salgado has proposed merging lp:~salgado/launchpad-work-items-tracker/blueprints-over-time into lp:~linaro-infrastructure/launchpad-work-items-tracker/linaro.
>
> Requested reviews:
> Linaro Infrastructure (linaro-infrastructure)
>
> For more details, see:
> https://code.launchpad.net/~salgado/launchpad-work-items-tracker/blueprints-over-time/+merge/85921
>
> This branch adds a new table which stores the number of blueprints per (day, status, lane). This is then used on the roadmap page to show a graph of blueprints per status/day for the quarter.
> --
> https://code.launchpad.net/~salgado/launchpad-work-items-tracker/blueprints-over-time/+merge/85921
> Your team Linaro Infrastructure is requested to review the proposed merge of lp:~salgado/launchpad-work-items-tracker/blueprints-over-time into lp:~linaro-infrastructure/launchpad-work-items-tracker/linaro.
>
> === modified file 'all-projects'
> --- all-projects 2011-12-07 09:03:27 +0000
> +++ all-projects 2011-12-15 17:57:13 +0000
> @@ -117,8 +117,6 @@
>
> if opts.debug:
> extra_collect_args.append("--debug")
> - else:
> - extra_collect_args.append("--mail")

This should only be removed on staging. It's just so we don't email
about errors while testing.

>
> if not collect(source_dir, db_file, config_file, extra_collect_args):
> sys.stderr.write("collect failed for %s" % project_name)
>
> === modified file 'collect'
> --- collect 2011-12-07 09:03:27 +0000
> +++ collect 2011-12-15 17:57:13 +0000
> @@ -6,6 +6,7 @@
> # Copyright (C) 2010, 2011 Canonical Ltd.
> # License: GPL-3
>
> +import datetime
> import urllib, re, sys, optparse, smtplib, pwd, os, urlparse
> import logging
> from email.mime.text import MIMEText
> @@ -755,6 +756,8 @@
>
> # reset status for current day
> collector.clear_todays_workitems()
> + # We can delete all blueprints while keeping work items for previous days
> + # because there's no foreign key reference from WorkItem to Blueprint.
> collector.clear_blueprints()
> collector.clear_metas()
> collector.clear_complexitys()
>
> === modified file 'collect_roadmap'
> --- collect_roadmap 2011-12-15 12:59:35 +0000
> +++ collect_roadmap 2011-12-15 17:57:13 +0000
> @@ -51,9 +51,15 @@
> except urllib2.HTTPError, e:
> print "HTTP error for url '%s': %d" % (url, e.code)
> except urllib2.URLError, e:
> +<<<<<<< TREE
> print "Network error for url '%s': %s" % (url, e.reason.args[1])
> except ValueError, e:
> print "Data error for url '%s': %s" % (url, e.message)
> +=======

On Thu, Dec 15, 2011 at 6:58 PM, Guilherme Salgado
<guilherme.salgado@linaro.org> wrote:
> Guilherme Salgado has proposed merging lp:~salgado/launchpad-work-items-tracker/blueprints-over-time into lp:~linaro-infrastructure/launchpad-work-items-tracker/linaro.
>
> Requested reviews:
>  Linaro Infrastructure (linaro-infrastructure)
>
> For more details, see:
> https://code.launchpad.net/~salgado/launchpad-work-items-tracker/blueprints-over-time/+merge/85921
>
> This branch adds a new table which stores the number of blueprints per (day, status, lane). This is then used on the roadmap page to show a graph of blueprints per status/day for the quarter.
> --
> https://code.launchpad.net/~salgado/launchpad-work-items-tracker/blueprints-over-time/+merge/85921
> Your team Linaro Infrastructure is requested to review the proposed merge of lp:~salgado/launchpad-work-items-tracker/blueprints-over-time into lp:~linaro-infrastructure/launchpad-work-items-tracker/linaro.
>
> === modified file 'all-projects'
> --- all-projects        2011-12-07 09:03:27 +0000
> +++ all-projects        2011-12-15 17:57:13 +0000
> @@ -117,8 +117,6 @@
>
>         if opts.debug:
>             extra_collect_args.append("--debug")
> -        else:
> -            extra_collect_args.append("--mail")

This should only be removed on staging. It's just so we don't email
about errors while testing.

>
>         if not collect(source_dir, db_file, config_file, extra_collect_args):
>             sys.stderr.write("collect failed for %s" % project_name)
>
> === modified file 'collect'
> --- collect     2011-12-07 09:03:27 +0000
> +++ collect     2011-12-15 17:57:13 +0000
> @@ -6,6 +6,7 @@
>  # Copyright (C) 2010, 2011 Canonical Ltd.
>  # License: GPL-3
>
> +import datetime
>  import urllib, re, sys, optparse, smtplib, pwd, os, urlparse
>  import logging
>  from email.mime.text import MIMEText
> @@ -755,6 +756,8 @@
>
>     # reset status for current day
>     collector.clear_todays_workitems()
> +    # We can delete all blueprints while keeping work items for previous days
> +    # because there's no foreign key reference from WorkItem to Blueprint.
>     collector.clear_blueprints()
>     collector.clear_metas()
>     collector.clear_complexitys()
>
> === modified file 'collect_roadmap'
> --- collect_roadmap     2011-12-15 12:59:35 +0000
> +++ collect_roadmap     2011-12-15 17:57:13 +0000
> @@ -51,9 +51,15 @@
>     except urllib2.HTTPError, e:
>         print "HTTP error for url '%s': %d" % (url, e.code)
>     except urllib2.URLError, e:
> +<<<<<<< TREE
>         print "Network error for url '%s': %s" % (url, e.reason.args[1])
>     except ValueError, e:
>         print "Data error for url '%s': %s" % (url, e.message)
> +=======

> +        print "Network error: %s" % e.reason.args[1]
> +    except ValueError, e:
> +        print "Data error: %s" % e.message
> +>>>>>>> MERGE-SOURCE
>
>     return data
>
> @@ -269,11 +275,14 @@
>     store = get_store(opts.database)
>     collector = CollectorStore(store, '', error_collector)
>
> +    collector.clear_todays_blueprint_daily_count_per_state()
>     collector.clear_lanes()
>     collector.clear_cards()
>
>     kanban_import(collector, cfg, opts.board, opts.api_token)
>
> +    collector.store_roadmap_bp_count_per_state()
> +
>     store.commit()
>
>     os.unlink(lock_path)
>
> === modified file 'generate-all'
> --- generate-all        2011-12-13 09:38:14 +0000
> +++ generate-all        2011-12-15 17:57:13 +0000
> @@ -80,15 +80,6 @@
>     else:
>         raise
>
> -groupssubdir = os.path.join(opts.output_dir, 'group')
> -try:
> -    os.mkdir(groupssubdir)
> -except OSError as exc:
> -    if exc.errno == errno.EEXIST:
> -        pass
> -    else:
> -        raise
> -

This was also just removed on staging. Only for improving performance.
We're probably not ready to remove this in production yet.

>  lanessubdir = os.path.join(opts.output_dir, '..', 'lane')
>  try:
>     os.mkdir(lanessubdir)
> @@ -140,12 +131,6 @@
>     basename = os.path.join(opts.output_dir, due_date_str)
>     report_tools.run_reports(my_path, opts.database, basename, opts.config, team=primary_team, root=opts.root, date=due_date_str)
>
> -# groups
> -for g in groups:
> -    basename = os.path.join(groupssubdir, g)
> -    report_tools.group_overview(my_path, opts.database, basename, opts.config, g, root=opts.root)
> -
> -

We should keep this too a while. It's the TR pages.

>  # global report
>  basename = os.path.join(opts.output_dir, 'all')
>  if 'all' in trend_starts:
>
> === modified file 'html-report'
> --- html-report 2011-12-07 09:03:27 +0000
> +++ html-report 2011-12-15 17:57:13 +0000
> @@ -495,6 +495,7 @@
>         data.update(dict(page_type="roadmap_lane"))
>         data.update(dict(lane_title=title))
>         data.update(dict(lanes=lanes))
> +        data.update(dict(chart_url=opts.chart_url))
>         print report_tools.fill_template(
>             "roadmap_lane.html", data, theme=opts.theme)
>
>
> === modified file 'lpworkitems/collect_roadmap.py'
> --- lpworkitems/collect_roadmap.py      2011-12-13 09:24:21 +0000
> +++ lpworkitems/collect_roadmap.py      2011-12-15 17:57:13 +0000
> @@ -1,3 +1,5 @@
> +import datetime
> +
>  from lpworkitems import models_roadmap
>  from utils import unicode_or_None
>
> @@ -24,6 +26,30 @@
>     def store_card(self, card):
>         self.store.add(card)
>
> +    def clear_todays_blueprint_daily_count_per_state(self):
> +        self._clear_all(
> +            models_roadmap.BlueprintDailyCountPerState,
> +            models_roadmap.BlueprintDailyCountPerState.day == datetime.date.today())
> +
> +    def store_roadmap_bp_count_per_state(self):
> +        query = """
> +            SELECT implementation, lane_id, count(*)
> +            FROM specs
> +            JOIN meta on spec = specs.name
> +            JOIN card on roadmap_id = value
> +            WHERE key = 'Roadmap id'
> +            GROUP BY implementation, lane_id
> +            """
> +        day = datetime.date.today()
> +        result = self.store.execute(query)
> +        for status, lane_id, count in result:
> +            obj = models_roadmap.BlueprintDailyCountPerState()
> +            obj.day = day
> +            obj.status = status
> +            obj.lane_id = lane_id
> +            obj.count = count
> +            self.store.add(obj)
> +
>
>  def get_json_item(data, item_name):
>     item = data[item_name]
>
> === modified file 'lpworkitems/database.py'
> --- lpworkitems/database.py     2011-11-03 15:48:13 +0000
> +++ lpworkitems/database.py     2011-12-15 17:57:13 +0000
> @@ -7,13 +7,13 @@
>     store.execute('''CREATE TABLE version (
>         db_layout_ref INT NOT NULL
>         )''')
> -    store.execute('''INSERT INTO version VALUES (14)''')
> +    store.execute('''INSERT INTO version VALUES (15)''')
>
>     store.execute('''CREATE TABLE specs (
>         name VARCHAR(255) PRIMARY KEY,
>         url VARCHAR(1000) NOT NULL,
>         priority CHAR(20),
> -        implementation CHAR(30),
> +        implementation CHAR(30) NOT NULL,
>         assignee CHAR(50),
>         team CHAR(50),
>         status VARCHAR(5000) NOT NULL,
> @@ -25,6 +25,13 @@
>         roadmap_notes VARCHAR(5000)
>         )''')
>
> +    store.execute('''CREATE TABLE spec_daily_count_per_state (
> +        status VARCHAR(5000) NOT NULL,
> +        day DATE NOT NULL,
> +        lane_id REFERENCES lane(lane_id),
> +        count INT NOT NULL
> +        )''')
> +
>     store.execute('''CREATE TABLE work_items (
>         description VARCHAR(1000) NOT NULL,
>         spec VARCHAR(255) REFERENCES specs(name),
> @@ -234,7 +241,14 @@
>         store.execute('ALTER TABLE card ADD COLUMN url VARCHAR(200)')
>         store.execute('ALTER TABLE card ADD COLUMN is_healthy BOOLEAN')
>         store.execute('UPDATE version SET db_layout_ref = 14')
> -
> +    if ver == 14:
> +        store.execute('''CREATE TABLE spec_daily_count_per_state (
> +            status VARCHAR(5000) NOT NULL,
> +            day DATE NOT NULL,
> +            lane_id REFERENCES lane(lane_id),
> +            count INT NOT NULL
> +            )''')
> +        store.execute('UPDATE version SET db_layout_ref = 15')
>
>  def get_store(dbpath):
>     '''Open/initialize database.
>
> === modified file 'lpworkitems/factory.py'
> --- lpworkitems/factory.py      2011-06-14 22:00:21 +0000
> +++ lpworkitems/factory.py      2011-12-15 17:57:13 +0000
> @@ -11,6 +11,7 @@
>     TeamStructure,
>     Workitem,
>     )
> +from lpworkitems.models_roadmap import BlueprintDailyCountPerState, Card
>
>
>  class Factory(object):
> @@ -63,6 +64,8 @@
>             url = self.getUniqueUnicode(prefix=name+"_url")
>         if status is None:
>             status = self.getUniqueUnicode(prefix=name+"_status")
> +        if implementation is None:
> +            implementation = u'Unknown'
>         blueprint.name = name
>         blueprint.url = url
>         blueprint.status = status
> @@ -109,8 +112,11 @@
>             self.store.add(workitem)
>         return workitem
>
> -    def make_meta(self, store=True):
> +    def make_meta(self, key=None, value=None, blueprint=None, store=True):
>         meta = Meta()
> +        meta.key = key
> +        meta.value = value
> +        meta.blueprint = blueprint
>         if store:
>             self.store.add(meta)
>         return meta
> @@ -155,3 +161,28 @@
>         if store:
>             self.store.add(person)
>         return person
> +
> +    def make_blueprint_daily_count_per_state(self, status=None, count=1,
> +                                             day=None, store=True):
> +        if status is None:
> +            status = self.getUniqueUnicode()
> +        if day is None:
> +            day = datetime.date.today()
> +        obj = BlueprintDailyCountPerState()
> +        obj.day = day
> +        obj.status = status
> +        obj.count = count
> +        obj.lane_id = 1
> +        if store:
> +            self.store.add(obj)
> +        return obj
> +
> +    def make_card(self, store=True):
> +        name = self.getUniqueUnicode()
> +        card_id = self.getUniqueInteger()
> +        lane_id = self.getUniqueInteger()
> +        roadmap_id = self.getUniqueUnicode()
> +        card = Card(name, card_id, lane_id, roadmap_id)
> +        if store:
> +            self.store.add(card)
> +        return card
>
> === modified file 'lpworkitems/models.py'
> --- lpworkitems/models.py       2011-12-07 09:03:27 +0000
> +++ lpworkitems/models.py       2011-12-15 17:57:13 +0000
> @@ -2,7 +2,15 @@
>  import re
>  from utils import unicode_or_None
>
> -from storm.locals import Date, Reference, ReferenceSet, Unicode
> +from storm.locals import Date, Int, Reference, ReferenceSet, Unicode
> +
> +ROADMAP_STATUSES_MAP = {
> +    u'Completed': [u'Implemented'],
> +    u'Blocked': [u'Needs Infrastructure', u'Blocked', u'Deferred'],
> +    u'In Progress': [u'Deployment', u'Needs Code Review',
> +                     u'Beta Available', u'Good progress',
> +                     u'Slow progress', u'Started'],
> +    u'Planned': [u'Unknown', u'Not started', u'Informational']}
>
>
>  def fill_blueprint_info_from_launchpad(model_bp, lp_bp):
> @@ -48,6 +56,14 @@
>     project = Unicode()
>
>
> +def get_roadmap_status_for_bp_implementation_status(implementation):
> +    for key in ROADMAP_STATUSES_MAP:
> +        if implementation in ROADMAP_STATUSES_MAP[key]:
> +            return key
> +    # XXX: Is None the appropriate return value here?
> +    return None
> +
> +
>  class Blueprint(object):
>
>     __storm_table__ = "specs"
> @@ -81,6 +97,15 @@
>             lp_bp.whiteboard, "Roadmap\s+Notes")
>         return model_bp
>
> +    @property
> +    def roadmap_status(self):
> +        return get_roadmap_status_for_bp_implementation_status(
> +            self.implementation)
> +
> +
> +def current_date():
> +    return datetime.date.today()
> +
>
>  class Person(object):
>
> @@ -108,10 +133,6 @@
>     superteam_name = Unicode(name="team")
>
>
> -def current_date():
> -    return datetime.date.today()
> -
> -
>  class Meta(object):
>
>     __storm_table__ = "meta"
>
> === modified file 'lpworkitems/models_roadmap.py'
> --- lpworkitems/models_roadmap.py       2011-11-03 15:48:13 +0000
> +++ lpworkitems/models_roadmap.py       2011-12-15 17:57:13 +0000
> @@ -4,6 +4,8 @@
>
>  from storm.locals import Date, Reference, ReferenceSet, Unicode, Int, Bool
>
> +from lpworkitems import models
> +
>
>  class Card(object):
>
> @@ -43,3 +45,16 @@
>     def __init__(self, name, lane_id):
>         self.lane_id = lane_id
>         self.name = name
> +
> +
> +def current_date():
> +    return datetime.date.today()
> +
> +
> +class BlueprintDailyCountPerState(object):
> +    __storm_table__ = 'spec_daily_count_per_state'
> +    __storm_primary__ = 'status', 'day'
> +    day = Date(default_factory=current_date)
> +    status = Unicode()
> +    lane_id = Int()
> +    count = Int()
>
> === modified file 'lpworkitems/tests/test_collect.py'
> --- lpworkitems/tests/test_collect.py   2011-06-14 22:00:21 +0000
> +++ lpworkitems/tests/test_collect.py   2011-12-15 17:57:13 +0000
> @@ -184,7 +184,6 @@
>             self.store.find(
>                 Milestone, Milestone.name == name).one())
>
> -
>     def test_store_blueprint_stores_blueprint(self):
>         blueprint = self.factory.make_blueprint(store=False)
>         ret = self.collector.store_blueprint(blueprint)
>
> === modified file 'lpworkitems/tests/test_collect_roadmap.py'
> --- lpworkitems/tests/test_collect_roadmap.py   2011-12-13 09:24:21 +0000
> +++ lpworkitems/tests/test_collect_roadmap.py   2011-12-15 17:57:13 +0000
> @@ -1,15 +1,13 @@
> +import datetime
> +
>  from lpworkitems.collect_roadmap import (
>     CollectorStore,
>     get_json_item,
> -    lookup_kanban_priority,
>     )
> +from lpworkitems.models_roadmap import BlueprintDailyCountPerState
>  from lpworkitems.error_collector import (
>     ErrorCollector,
>     )
> -from lpworkitems.models_roadmap import (
> -    Card,
> -    Lane,
> -    )
>  from lpworkitems.testing import TestCaseWithFakeLaunchpad
>
>
> @@ -26,6 +24,26 @@
>         fn()
>         self.assertEqual(0, self.store.find(cls).count())
>
> +    def test_clear_todays_blueprint_daily_count_per_state(self):
> +        self.factory.make_blueprint_daily_count_per_state(
> +            day=datetime.date.today())
> +        self.assertClears(
> +            BlueprintDailyCountPerState,
> +            self.collector.clear_todays_blueprint_daily_count_per_state)
> +
> +    def test_store_roadmap_bp_count_per_state(self):
> +        bp = self.factory.make_blueprint()
> +        card = self.factory.make_card()
> +        meta = self.factory.make_meta(
> +            key=u'Roadmap id', value=card.roadmap_id, blueprint=bp)
> +        self.collector.store_roadmap_bp_count_per_state()
> +        self.assertEqual(
> +            1, self.store.find(BlueprintDailyCountPerState).count())
> +        entry = self.store.find(BlueprintDailyCountPerState).one()
> +        self.assertEqual(1, entry.count)
> +        self.assertEqual(card.lane_id, entry.lane_id)
> +        self.assertEqual(bp.implementation, entry.status)
> +
>     # XXX Add tests for the roadmap classes.
>
>
>
> === modified file 'lpworkitems/tests/test_factory.py'
> --- lpworkitems/tests/test_factory.py   2011-06-04 18:48:23 +0000
> +++ lpworkitems/tests/test_factory.py   2011-12-15 17:57:13 +0000
> @@ -162,7 +162,7 @@
>         implementation = u"Implemented"
>         self.assert_with_and_without(
>             self.factory.make_blueprint, "implementation", implementation,
> -            Equals(None))
> +            Equals("Unknown"))
>
>     def test_uses_assignee_name(self):
>         assignee_name = self.factory.getUniqueUnicode(
>
> === modified file 'lpworkitems/tests/test_models.py'
> --- lpworkitems/tests/test_models.py    2011-12-06 15:20:43 +0000
> +++ lpworkitems/tests/test_models.py    2011-12-15 17:57:13 +0000
> @@ -6,8 +6,11 @@
>     extract_last_path_segment_from_url,
>     extract_user_name_from_url,
>     get_whiteboard_section,
> -    )
> -from lpworkitems.testing import TestCaseWithFakeLaunchpad
> +    ROADMAP_STATUSES_MAP,
> +    )
> +from lpworkitems.testing import (
> +    TestCaseWithFakeLaunchpad,
> +    )
>
>
>  class GetWhiteboardSectionTests(TestCase):
> @@ -42,6 +45,18 @@
>
>  class BlueprintTests(TestCaseWithFakeLaunchpad):
>
> +    def test_roadmap_status(self):
> +        roadmap_status = "Completed"
> +        bp_implementation = ROADMAP_STATUSES_MAP[roadmap_status][0]
> +        bp_status = self.factory.make_blueprint(
> +            implementation=bp_implementation)
> +        self.assertEqual(roadmap_status, bp_status.roadmap_status)
> +
> +    def test_roadmap_status_unknown_status(self):
> +        blueprint = self.factory.make_blueprint(
> +            implementation=u"Not Expected")
> +        self.assertEqual(None, blueprint.roadmap_status)
> +
>     def test_from_launchpad_sets_name(self):
>         name = self.factory.getUniqueUnicode(prefix="lpblueprint")
>         lp_bp = self.lp.make_blueprint(name=name)
>
> === modified file 'report_tools.py'
> --- report_tools.py     2011-12-12 07:35:35 +0000
> +++ report_tools.py     2011-12-15 17:57:13 +0000
> @@ -14,7 +14,8 @@
>     Card,
>  )
>  from lpworkitems.models import (
> -    Meta,
> +    Meta, ROADMAP_STATUSES_MAP,
> +    get_roadmap_status_for_bp_implementation_status,
>  )
>
>  valid_states = [u'inprogress', u'blocked', u'todo', u'done', u'postponed']
> @@ -206,12 +207,17 @@
>  def roadmap_pages(my_path, database, basename, config, lane, root=None):
>     cfg = load_config(config)
>     fh = open(basename + '.html', 'w')
> +    # XXX fix this. the intention is to have the chart in the same dir as the
> +    # html. This is not essential but makes linking in roadmap_lane.html easier
> +    chart_name = '/'.join(basename.split('/')[:-1]) + '/current_quarter.svg'
>     try:
>         args = [os.path.join(my_path, 'html-report'), '-d', database]
>         args += ['--report-type', 'roadmap_page']
>         args += ['--lane', lane.name]
>         if root:
>             args += ['--root', root]
> +        if lane.is_current:
> +            args += ['--chart', chart_name]
>         report_args(args, theme=get_theme(cfg))
>         proc = Popen(args, stdout=fh)
>         print basename + '.html'
> @@ -219,6 +225,13 @@
>     finally:
>         fh.close()
>
> +    if lane.is_current:
> +        args = [os.path.join(my_path, 'roadmap-bp-chart'), '-d', database,
> +                '-o', chart_name]
> +        proc = Popen(args)
> +        print chart_name
> +        proc.wait()
> +
>
>  def roadmap_cards(my_path, database, basename, config, card, root=None):
>     cfg = load_config(config)
> @@ -341,6 +354,34 @@
>     return escape(html, True)
>
>
> +def blueprints_over_time(store):
> +    '''Calculate blueprint development over time for the current lane.
> +
> +    We do not need to care about teams or groups since this is intended for the
> +    roadmap overview.
> +
> +    Return date -> state -> count mapping. states are
> +    {planned,inprogress,completed,blocked}.
> +    '''
> +    data = {}
> +    result = store.execute("""
> +        SELECT status, day, count
> +        FROM spec_daily_count_per_state
> +        JOIN lane on lane.lane_id = spec_daily_count_per_state.lane_id
> +        WHERE lane.is_current = 1
> +        """)
> +    for status, day, count in result:
> +        roadmap_status = get_roadmap_status_for_bp_implementation_status(
> +            status)
> +        assert roadmap_status is not None
> +        if day not in data:
> +            data[day] = {}
> +        if roadmap_status not in data[day]:
> +            data[day][roadmap_status] = 0
> +        data[day][roadmap_status] += count
> +    return data
> +
> +
>  def workitems_over_time(store, team=None, group=None, milestone_collection=None):
>     '''Calculate work item development over time.
>
> @@ -978,21 +1019,11 @@
>
>  def card_blueprints_by_status(store, roadmap_id):
>     blueprints = card_blueprints(store, roadmap_id)
> -    statuses = {'Completed': ['Implemented'],
> -                'Blocked': ['Needs Infrastructure', 'Blocked', 'Deferred'],
> -                'In Progress': ['Deployment', 'Needs Code Review',
> -                                'Beta Available', 'Good progress',
> -                                'Slow progress', 'Started'],
> -                'Planned': ['Unknown', 'Not started', 'Informational']}
> -    bp_by_status = {'In Progress': [],
> -                    'Blocked': [],
> -                    'Planned': [],
> -                    'Completed': []}
> +    bp_by_status = {}
> +    for key in ROADMAP_STATUSES_MAP:
> +        bp_by_status[key] = []
>     for bp in blueprints:
> -        for status in statuses.iterkeys():
> -            if bp.implementation in statuses[status]:
> -                bp_by_status[status].append(bp)
> -                break
> +        bp_by_status[bp.roadmap_status].append(bp)
>     return bp_by_status
>
>
>
> === added file 'roadmap-bp-chart'
> --- roadmap-bp-chart    1970-01-01 00:00:00 +0000
> +++ roadmap-bp-chart    2011-12-15 17:57:13 +0000
> @@ -0,0 +1,254 @@
> +#!/usr/bin/python
> +#
> +# Create a blueprint tracking chart from a blueprint database.
> +#
> +# Copyright (C) 2010, 2011 Canonical Ltd.
> +# License: GPL-3
> +
> +import optparse, datetime, sys
> +import report_tools
> +
> +from pychart import *
> +
> +def date_to_ordinal(s):
> +    '''Turn yyyy-mm-dd strings to ordinals'''
> +    return report_tools.date_to_python(s).toordinal()
> +
> +
> +def ordinal_to_date(ordinal):
> +    '''Turn an ordinal date into a string'''
> +    d = datetime.date.fromordinal(int(ordinal))
> +    return d.strftime('%Y-%m-%d')
> +
> +def format_date(ordinal):
> +    d = datetime.date.fromordinal(int(ordinal))
> +    return '/a60{}' + d.strftime('%b %d, %y')
> +
> +def do_chart(data, start_date, end_date, trend_start, title, filename, only_weekdays, inverted):
> +    #set up default values
> +    format = 'svg'
> +    height = 450
> +    width = 1000
> +    legend_x = 700
> +    legend_y = 200
> +    title_x = 300
> +    title_y = 350
> +
> +    if inverted:
> +        legend_x=200
> +
> +    # Tell pychart to use colors
> +    theme.use_color = True
> +    theme.default_font_size = 12
> +    theme.reinitialize()
> +
> +    # turn into pychart data model and calculate maximum number of WIs
> +    max_items = 1 # start at 1 to avoid zero div
> +    lastactive = 0
> +    pcdata = []
> +
> +    for date in xrange(date_to_ordinal(start_date), date_to_ordinal(end_date)+1):
> +        if (not only_weekdays or datetime.date.fromordinal(date).weekday() < 5):
> +            end_date = ordinal_to_date(date)
> +            i = data.get(ordinal_to_date(date), {})
> +            count = i.get('Completed', 0) + i.get('Planned', 0) + i.get('Blocked', 0) + i.get('In Progress', 0)
> +            if max_items < count:
> +                max_items = count
> +            pcdata.append((date, i.get('Planned', 0),0,
> +                           i.get('Blocked', 0),0,
> +                           i.get('In Progress', 0),0,
> +                           i.get('Completed',0),0, count))
> +            if count > 0:
> +                lastactive = len(pcdata) - 1
> +
> +    # add some extra space to look nicer
> +    max_items = int(max_items * 1.05)
> +
> +    x_interval = len(pcdata)/20
> +    if max_items > 500:
> +        y_interval = max_items/200*10
> +    elif max_items < 20:
> +        y_interval = 1
> +    else:
> +        y_interval = max_items/20
> +
> +    # create the chart object
> +    chart_object.set_defaults(area.T, size=(width, height),
> +            y_range=(0, None), x_coord=category_coord.T(pcdata, 0))
> +
> +    # tell the chart object it will use a bar chart, and will
> +    # use the data list for it's model
> +    chart_object.set_defaults(bar_plot.T, data=pcdata)
> +
> +    # create the chart area
> +    # tell it to start at coords 0,0
> +    # tell it the labels, and the tics, etc..
> +    # HACK: to prevent 0 div
> +    if max_items == 0:
> +        max_items = 1
> +    ar = area.T(legend=legend.T(loc=(legend_x,legend_y)), loc=(0,0),
> +            x_axis=axis.X(label='Date', tic_interval=x_interval,format=format_date),
> +            y_axis=axis.Y(label='Blueprints', tic_interval=y_interval),
> +            y_range=(0, max_items))
> +
> +    #initialize the blar_plot fill styles
> +    bar_plot.fill_styles.reset()
> +
> +    # create each set of data to plot
> +    # note that index zero is the label col
> +    # for each column of data, tell it what to use for the legend and
> +    # what color to make the bar, no lines, and
> +    # what plot to stack on
> +
> +    tlabel = ''
> +
> +    if inverted:
> +        plot1 = bar_plot.T(label='Completed' + tlabel, hcol=7)
> +        plot1.fill_style = fill_style.Plain(bgcolor=color.seagreen)
> +
> +        plot3 = bar_plot.T(label='In Progress' + tlabel, hcol=5, stack_on = plot1)
> +        plot3.fill_style = fill_style.Plain(bgcolor=color.gray65)
> +
> +        plot5 = bar_plot.T(label='Blocked' + tlabel, hcol=3, stack_on = plot3)
> +        plot5.fill_style = fill_style.Plain(bgcolor=color.red1)
> +
> +        plot7 = bar_plot.T(label='Planned' + tlabel, hcol=1, stack_on = plot5)
> +        plot7.fill_style = fill_style.Plain(bgcolor=color.darkorange1)
> +    else:
> +        plot1 = bar_plot.T(label='Planned' + tlabel, hcol=1)
> +        plot1.fill_style = fill_style.Plain(bgcolor=color.darkorange1)
> +
> +        plot3 = bar_plot.T(label='Blocked' + tlabel, hcol=3, stack_on = plot1)
> +        plot3.fill_style = fill_style.Plain(bgcolor=color.red1)
> +
> +        plot5 = bar_plot.T(label='In Progress' + tlabel, hcol=5, stack_on = plot3)
> +        plot5.fill_style = fill_style.Plain(bgcolor=color.gray65)
> +
> +        plot7 = bar_plot.T(label='Completed' + tlabel, hcol=7, stack_on = plot5)
> +        plot7.fill_style = fill_style.Plain(bgcolor=color.seagreen)
> +
> +
> +    plot1.line_style = None
> +    plot3.line_style = None
> +    plot5.line_style = None
> +    plot7.line_style = None
> +
> +    plot11 = bar_plot.T(label='total', hcol=9)
> +    plot11.fill_style = None
> +    plot11.line_style = line_style.gray30
> +
> +    # create the canvas with the specified filename and file format
> +    can = canvas.init(filename,format)
> +
> +    # add the data to the area and  draw it
> +    ar.add_plot(plot1, plot3, plot5, plot7)
> +    ar.draw()
> +
> +    # title
> +    tb = text_box.T(loc=(title_x, title_y), text=title, line_style=None)
> +    tb.fill_style = None
> +    tb.draw()
> +
> +#
> +# main
> +#
> +
> +# argv parsing
> +optparser = optparse.OptionParser()
> +optparser.add_option('-d', '--database',
> +    help='Path to database', dest='database', metavar='PATH')
> +optparser.add_option('-t', '--team',
> +        help='Restrict report to a particular team', dest='team')
> +optparser.add_option('-m', '--milestone',
> +        help='Restrict report to a particular milestone', dest='milestone')
> +optparser.add_option('-o', '--output',
> +        help='Output file', dest='output')
> +optparser.add_option('--trend-start', type='int',
> +        help='Explicitly set start of trend line', dest='trendstart')
> +optparser.add_option('-u', '--user',
> +        help='Run for this user', dest='user')
> +optparser.add_option('--only-weekdays', action='store_true',
> +        help='Skip Saturdays and Sundays in the resulting graph', dest='only_weekdays')
> +optparser.add_option('--inverted', action='store_true',
> +        help='Generate an inverted burndown chart', dest='inverted')
> +optparser.add_option('-s', '--start-date',
> +        help='Explicitly set the start date of the burndown data', dest='start_date')
> +optparser.add_option('-e', '--end-date',
> +        help='Explicitly set the end date of the burndown data', dest='end_date')
> +optparser.add_option('--no-foreign', action='store_true', default=False,
> +        help='Do not show foreign totals separate', dest='noforeign')
> +optparser.add_option('--group',
> +        help='Run for this group', dest='group')
> +optparser.add_option('--date',
> +        help='Run for this date', dest='date')
> +
> +(opts, args) = optparser.parse_args()
> +if not opts.database:
> +    optparser.error('No database given')
> +if not opts.output:
> +    optparser.error('No output file given')
> +
> +if opts.user and  opts.team:
> +    optparser.error('team and user options are mutually exclusive')
> +if opts.user and opts.group:
> +    optparser.error('user and group options are mutually exclusive')
> +if opts.team and opts.group:
> +    optparser.error('team and group options are mutually exclusive')
> +if opts.milestone and opts.date:
> +    optparser.error('milestone and date options are mutually exclusive')
> +
> +# The typing allows polymorphic behavior
> +if opts.user:
> +    opts.team = report_tools.user_string(opts.user)
> +elif opts.team:
> +    opts.team = report_tools.team_string(opts.team)
> +
> +store = report_tools.get_store(opts.database)
> +
> +milestone_collection = None
> +if opts.milestone:
> +    milestone_collection = report_tools.get_milestone(store, opts.milestone)
> +elif opts.date:
> +    milestone_collection = report_tools.MilestoneGroup(
> +        report_tools.date_to_python(opts.date))
> +
> +
> +# get date -> state -> count mapping
> +data = report_tools.blueprints_over_time(store)
> +
> +if len(data) == 0:
> +    print 'WARNING: no blueprints, not generating chart (team: %s, group: %s, due date: %s)' % (
> +        opts.team or 'all', opts.group or 'none', milestone_collection and milestone_collection.display_name or 'none')
> +    sys.exit(0)
> +
> +# calculate start/end date if no dates are given
> +if opts.start_date is None:
> +    start_date = sorted(data.keys())[0]
> +else:
> +    start_date=opts.start_date
> +
> +if opts.end_date is None:
> +    if milestone_collection is not None:
> +        end_date = milestone_collection.due_date_str
> +    else:
> +        end_date=report_tools.milestone_due_date(store)
> +else:
> +    end_date=opts.end_date
> +
> +if not start_date or not end_date or date_to_ordinal(start_date) > date_to_ordinal(end_date):
> +    print 'WARNING: empty date range, not generating chart (team: %s, group: %s, due date: %s)' % (
> +        opts.team or 'all', opts.group or 'none', milestone_collection and milestone_collection.display_name or 'none')
> +    sys.exit(0)
> +
> +# title
> +if opts.team:
> +    title = '/20' + opts.team
> +elif opts.group:
> +    title = "/20" + opts.group
> +else:
> +    title = '/20all teams'
> +
> +if milestone_collection is not None:
> +    title += ' (%s)' % milestone_collection.name
> +
> +do_chart(data, start_date, end_date, opts.trendstart, title, opts.output, opts.only_weekdays, opts.inverted)
>
> === modified file 'templates/roadmap_lane.html'
> --- templates/roadmap_lane.html 2011-12-12 09:17:20 +0000
> +++ templates/roadmap_lane.html 2011-12-15 17:57:13 +0000
> @@ -6,18 +6,20 @@
>  import report_tools
>  %>
>
> -Lane to view:
> +<%def name="title()">
> +Progress for ${lane_title}
> +</%def>
> +
> +<h1>${title()}</h1>
> +
> +<p style="text-align: right; color: green; font-size: 13pt">
> +Select calendar quarter to view:
>  <select name="laneselect" onchange="window.location=this.value;">
>  % for lane in lanes:
>  <option value="${lane.name}.html"${' selected="selected"' if lane.name == lane_title else ''}>${lane.name}${' (current)' if lane.is_current else ''}</option>
>  % endfor
>  </select>
>
> -<%def name="title()">
> -Progress for ${lane_title}
> -</%def>
> -
> -<h1>${title()}</h1>
>  <p>
>  <table>
>  <thead><tr><th>Card</th><th>Status</th><th>Team</th><th>Priority</th><th>Blueprints</th><th>Health</th></tr></thead>
> @@ -41,3 +43,12 @@
>  % endfor
>  </table>
>
> +% if chart_url != 'burndown.svg':
> +
> +<div class="overview_graph">
> +<h3>Blueprint progress</h3><p><a href="current_quarter.svg">(enlarge)</a></p>
> +<object
> +       height="500" width="833"
> +               data="current_quarter.svg" type="image/svg+xml">Blueprint progress</object>
> +</div>
> +% endif
>
>

lp://qastaging/~salgado/launchpad-work-items-tracker/blueprints-over-time updated on 2011-12-15

326. By Guilherme Salgado on 2011-12-15: merge linaro branch
327. By Guilherme Salgado on 2011-12-15: Improve a docstring
328. By Guilherme Salgado on 2011-12-15: Rework the tests and fix blueprints_over_time()
329. By Guilherme Salgado on 2011-12-15: Fix blueprints_over_time
330. By Guilherme Salgado on 2011-12-15: Remove some commented out code
331. By Guilherme Salgado on 2011-12-15: Re-add a test assertion I accidentally removed
332. By Guilherme Salgado on 2011-12-15: Remove an unnecessary import

Revision history for this message

Mattias Backman (mabac) wrote on 2011-12-15:

No more staging artifacts that I can find.

review: Approve

lp://qastaging/~salgado/launchpad-work-items-tracker/blueprints-over-time updated on 2011-12-15

333. By Guilherme Salgado on 2011-12-15: Create a new function in collect_roadmap which takes care of deleting and recreating the blueprint status per day