Linaro Android Frontend

Merge lp://qastaging/~dooferlad/linaro-android-frontend/ui_download_links_update into lp://qastaging/linaro-android-frontend

ui_download_links_update
Merge into trunk

Proposed by James Tunnicliffe on 2012-01-17

Status:	Merged
Approved by:	Paul Sokolovsky on 2012-01-25
Approved revision:	253
Merged at revision:	246
Proposed branch:	lp://qastaging/~dooferlad/linaro-android-frontend/ui_download_links_update
Merge into:	lp://qastaging/linaro-android-frontend
Diff against target:	255 lines (+163/-16) 3 files modified android_build/frontend/api.py (+55/-3) android_build/urls.py (+2/-1) static/buildDetails.js (+106/-12)
To merge this branch:	bzr merge lp://qastaging/~dooferlad/linaro-android-frontend/ui_download_links_update
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
Paul Sokolovsky		2012-01-17	Approve on 2012-01-24
Review via email: mp+88914@code.qastaging.launchpad.net

Description of the change

UI will present the user a list of downloads. It will test to see if it can find a build on snapshots.linaro.org. If it finds it it will use that server and point the links there. If it fails to find a build on snapshots, it will look for a locally stored on android-build.linaro.org. If it finds a build there, it will point the download links to it. If it fails to find builds anywhere, it will point links to where it thinks the build should be on snapshots.

In preparation for artefact publishing going away this update also supports using a file called manifest.txt, which is just a list of files (one relative location per line). If it finds manifest.txt in the root of the download location, it will use that. It will fall back to the artefact list and if that fails, display a message saying it couldn't provide links to individual files.

I have tested this using a proxy that would replace bits of the live android-build.linaro.org site (https://code.launchpad.net/~dooferlad/junk/dev_proxy) so I could browse around several build pages to see if this worked. Unfortunately it looks like my ISP is suffering from a routing problem and I just lost connectivity to android-build.linaro.org and I didn't have a chance to test looking at a build that has its downloads on android-build.linaro.org. It should work, but at this point it would be nice to have some feedback on the change.

lp://qastaging/~dooferlad/linaro-android-frontend/ui_download_links_update updated on 2012-01-18

251. By James Tunnicliffe on 2012-01-18: Fixed error - wasn't testing return code when examinining where a build was hosted.
Fixed test_remote_dir_exists description.
Allow test_remote_dir_exists to test android-build.linaro.org.

Revision history for this message

James Tunnicliffe (dooferlad) wrote on 2012-01-18:

A couple of fixes added and now tested in the following conditions:

1. Build exists on android-build.linaro.org only
   - File links point to android-build.linaro.org
2. Build exists on snapshots.linaro.org
   - File links point to snapshots.linaro.org
3. Build exists on shapshots.linaro.org and manifest.txt exists
   - File links point to snapshots.linaro.org and download list comes from
     manifest.txt

So it should be good to go.

Revision history for this message

Paul Sokolovsky (pfalcon) wrote on 2012-01-23:

Well, comments are:

1. Are you sure that making 3 synchronous AJAX requests doesn't hurt page download/interactive performance?

2. create_manifest.py script apparently doesn't belong here, but rather should be part of linaro-android-build-tools and be called from build-android script at the appropriate place.

3. Did you give extra thought what manifest.txt may contain? Just filenames? Maybe file size? What about MD5SUMS file we already create for subset of files (actually, one of the reason why is it created for subset and not all was because that would require faithful duplication of artifact filename patterns - both in Jenkins and in this script, same problem would exist for create_manifest.py too). (I don't have any specific suggestion regarding what manifest.txt should include, just wanted to make sure this is though-out).

4. Ah, last one, naming. Maybe "MANIFEST" is better, it's kind of common practice (that depends on contents format of course).

review: Needs Information

Revision history for this message

James Tunnicliffe (dooferlad) wrote on 2012-01-23:

Hi Paul, thanks for the comments.

> 1. Are you sure that making 3 synchronous AJAX requests doesn't hurt page download/interactive performance?

It hasn't caused any obvious performance problems for me, but if one
or more of the requests took a long time it could be a problem. Will
investigate how easy it is to move them to asynchronous requests.

> 2. create_manifest.py script apparently doesn't belong here, but rather should be part of linaro-android-build-tools and be called from build-android script at the appropriate place.

Ah, thanks. Will move it.

> 3. Did you give extra thought what manifest.txt may contain? Just filenames? Maybe file size? What about MD5SUMS file we already create for subset of files (actually, one of the reason why is it created for subset and not all was because that would require faithful duplication of artifact filename patterns - both in Jenkins and in this script, same problem would exist for create_manifest.py too). (I don't have any specific suggestion regarding what manifest.txt should include, just wanted to make sure this is though-out).

The only thing the web UI will use is file names and that is the only
reason to add the file. If we extend its use later we could update the
UI to cope.

> 4. Ah, last one, naming. Maybe "MANIFEST" is better, it's kind of common practice (that depends on contents format of course).

Not a problem - changing a file name is easy enough.

--
James Tunnicliffe

Revision history for this message

Paul Sokolovsky (pfalcon) wrote on 2012-01-24:

On Mon, 23 Jan 2012 16:42:47 -0000
James Tunnicliffe <email address hidden> wrote:

> Hi Paul, thanks for the comments.
>
> > 1. Are you sure that making 3 synchronous AJAX requests doesn't
> > hurt page download/interactive performance?
>
> It hasn't caused any obvious performance problems for me, but if one
> or more of the requests took a long time it could be a problem. Will
> investigate how easy it is to move them to asynchronous requests.

Well, my idea of that would be done was to have chain of async result
handlers - i.e. result handler of 1st request would check if it got
what's needed and if so, display it, otherwise queue up another request
whose handler would do similarly, etc.

I don't pledge for sync processing to be changed immediately, just want
to make sure it is known as point of potential problem and we're ready
to resolve it.

>
> > 2. create_manifest.py script apparently doesn't belong here, but
> > rather should be part of linaro-android-build-tools and be called
> > from build-android script at the appropriate place.
>
> Ah, thanks. Will move it.
>
> > 3. Did you give extra thought what manifest.txt may contain? Just
> > filenames? Maybe file size? What about MD5SUMS file we already
> > create for subset of files (actually, one of the reason why is it
> > created for subset and not all was because that would require
> > faithful duplication of artifact filename patterns - both in
> > Jenkins and in this script, same problem would exist for
> > create_manifest.py too). (I don't have any specific suggestion
> > regarding what manifest.txt should include, just wanted to make
> > sure this is though-out).
>
> The only thing the web UI will use is file names and that is the only
> reason to add the file. If we extend its use later we could update the
> UI to cope.

Makes sense.

>
> > 4. Ah, last one, naming. Maybe "MANIFEST" is better, it's kind of
> > common practice (that depends on contents format of course).
>
> Not a problem - changing a file name is easy enough.

Ok, so I assume there will be more commits for this merge request
before approving it.

Thanks,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

On Mon, 23 Jan 2012 16:42:47 -0000
James Tunnicliffe <launchpad@nanosheep.org> wrote:

> Hi Paul, thanks for the comments.
> 
> > 1. Are you sure that making 3 synchronous AJAX requests doesn't
> > hurt page download/interactive performance?
> 
> It hasn't caused any obvious performance problems for me, but if one
> or more of the requests took a long time it could be a problem. Will
> investigate how easy it is to move them to asynchronous requests.

I don't pledge for sync processing to be changed immediately, just want
to make sure it is known as point of potential problem and we're ready
to resolve it.

> 
> > 2. create_manifest.py script apparently doesn't belong here, but
> > rather should be part of linaro-android-build-tools and be called
> > from build-android script at the appropriate place.
> 
> Ah, thanks. Will move it.
> 
> > 3. Did you give extra thought what manifest.txt may contain? Just
> > filenames? Maybe file size? What about MD5SUMS file we already
> > create for subset of files (actually, one of the reason why is it
> > created for subset and not all was because  that would require
> > faithful duplication of artifact filename patterns - both in
> > Jenkins and in this script, same problem would exist for
> > create_manifest.py too). (I don't have any specific suggestion
> > regarding what manifest.txt should include, just wanted to make
> > sure this is though-out).
> 
> The only thing the web UI will use is file names and that is the only
> reason to add the file. If we extend its use later we could update the
> UI to cope.

Makes sense.

> 
> > 4. Ah, last one, naming. Maybe "MANIFEST" is better, it's kind of
> > common practice (that depends on contents format of course).
> 
> Not a problem - changing a file name is easy enough.

Ok, so I assume there will be more commits for this merge request
before approving it.

Thanks,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

lp://qastaging/~dooferlad/linaro-android-frontend/ui_download_links_update updated on 2012-01-24

252. By James Tunnicliffe on 2012-01-24

create_manifest.py moved to linaro-android-build-tools project.
Add android-build.linaro.org as a proxy location
Updated static/buildDetails.js so lava-job-info and MANIFEST files are both sourced from the location returned by getDownloadURL.

TODO: remove as many synchronous Y.io calls as possible. It should be possible to save the result of getDownloadURL and reuse it.

253. By James Tunnicliffe on 2012-01-24

getDownloadURL - cache result of last call to reduce remote requests. The cached value is deleted on URL change since the JS doesn't reload, so the same cached value would be returned for different builds of the same project without this step.

Revision history for this message

James Tunnicliffe (dooferlad) wrote on 2012-01-24:

I have just pushed a new version. This doesn't perform asynchronous
requests, but does change the name of manifest.txt to MANIFEST and fix
a couple of bugs.

James

Revision history for this message

Paul Sokolovsky (pfalcon) on 2012-01-24:

review: Approve