Merge lp://qastaging/~brendan-donegan/hwcert-tools/certificate_check into lp://qastaging/~hardware-certification/hwcert-tools/reporting-tools

Great idea, thanks for going forward with implementing this!

The script takes ages to run with no indication of progress (it's still running as I send this). This is fine for the intended use case of running from a cron script, but I'd still like to see a -v switch (or default verbosity with a -q switch to quiet things for cron runs), otherwise I have no idea what's happening when trying to test it.

Also, I could suggest you have a look at checkbox-message-get-c4, I implemented a caching mechanism so that the reporting parts of the script can be run with pre-fetched data, which speeds up development considerably (i.e. not waiting 15 minutes to test if some modifications or new reporting in the script work as intended). This would also allow, for instance, a single ""fetch" action and then one or several scripts that take that data and report on it in interesting ways.

It's only a suggestion though, so I'll set as Needs Information to see what you think about this.

Fair comment, it is quite slow. At the very least I'll add a verbose mode - if I can't find a way to make it faster. I think it's probably just the sheer volume of data it has to get - every public pass certificate ever! There's the possibility I could do the inverse operation and get all private hardware then check if it has a public certificate - that might be faster. I'll have to check that the API supports querying for all private hardware though.

A suggestion to speed it up: instead of using requests.get to batch each chunk, create a session:

session = requests.Session()

then use the session object's get method instead of requests.get:

blah blah..
result = session.get(blah blah)

the advantage is that using a session reuses the http connection, shaving a bit off each fetched chunk. It won't be twice as fast, but hey, a 5-10% improvement is always welcome.

Another thing that has worked for me is setting expectations: do a timing run, and then just print a message saying "loading all certificates, this will take about 10 minutes". That way the user is not left wondering at a script that doesn't produce anything for a few minutes :) You can print this to stderr so that used from the commandline it behaves in this friendly way, while your cron can pipe stdout to a file while discarding stderr.

Oh - thanks for the tip! Anyway there's still some issues in the implementation (apart from the speed issues) so I'm keeping this on hold until I resolve them.

Okay, the change I wanted to do was to filter out certificates that weren't for precise, since that's the only release we show on the public site now. Using requests.session gives a big speed-up so that was really handy - thanks!

Looks better now (there are no certificates identified by the script?). I'll approve. Thanks!

(er, please set to Needs Review as I don't want to flip to Approved while it's still Work in Progress).