Merge lp://qastaging/~liuyq0307/lava-android-test/workload into lp://qastaging/lava-android-test

Before going too far into this, I wanted to mention my general thought:

This seems to be using ADB to generate workload intensive stuff. I think
this is probably the wrong way to be doing this. I think the bigLITTLE
switcher tests is a better approach where its written entirely for the
target. This seems way better, and will ultimately better fit with our
lava-test-shell efforts.

On 12/06/2012 07:06 AM, Yongqin Liu wrote:
> + git_url="ssh://<email address hidden>/srv/linaro-private.git.linaro.org/people/bhoj/workload-automation.git"

That's not going to work for anyone but you.

> Before going too far into this, I wanted to mention my general thought:
>
> This seems to be using ADB to generate workload intensive stuff. I think
> this is probably the wrong way to be doing this. I think the bigLITTLE
> switcher tests is a better approach where its written entirely for the
> target. This seems way better, and will ultimately better fit with our
> lava-test-shell efforts.

not see the detail implementation of workload test, but it seems to download
the apk of nbench, and install it to device, and do other things.

and for this workload test, are all the steps need to be run on host side
able to be done with lava-test-shell?
  git clone
  update ip in config
  run python workload_setup_dependencies.py
  python workload.py ${config_file} ${outputdir}/

> On 12/06/2012 07:06 AM, Yongqin Liu wrote:
> > + git_url="ssh://<email address hidden>/srv/linaro-
> private.git.linaro.org/people/bhoj/workload-automation.git"
>
> That's not going to work for anyone but you.
Or, what you mean is to rewrite the workload test via lava-test-shell?

On 12/06/2012 09:02 PM, Yongqin Liu wrote:
>> Before going too far into this, I wanted to mention my general thought:
>>
>> This seems to be using ADB to generate workload intensive stuff. I think
>> this is probably the wrong way to be doing this. I think the bigLITTLE
>> switcher tests is a better approach where its written entirely for the
>> target. This seems way better, and will ultimately better fit with our
>> lava-test-shell efforts.
>
> not see the detail implementation of workload test, but it seems to download
> the apk of nbench, and install it to device, and do other things.

I think this could be a quick patch to lava-test-shell. We have "deps"
support for Ubuntu. We could do similar for Android.

> and for this workload test, are all the steps need to be run on host side
> able to be done with lava-test-shell?
> git clone
> update ip in config
> run python workload_setup_dependencies.py
> python workload.py ${config_file} ${outputdir}/

We don't have to do the exact steps. git-clone is supported. not sure
about the ip update. The setup deps could be done in an equivalent
manner. Instead of thinking "how do we do this on host in python", I'm
saying ask yourself - "how could I do this as a script that ran on the
target"? I don't think it would be that hard.

If you absolutely think I'm crazy, then continue with this approach, but
doing a "workload" test and requiring ADB seems like a bad idea.

>> On 12/06/2012 07:06 AM, Yongqin Liu wrote:
>>> + git_url="ssh://<email address hidden>/srv/linaro-
>> private.git.linaro.org/people/bhoj/workload-automation.git"
>>
>> That's not going to work for anyone but you.
> Or, what you mean is to rewrite the workload test via lava-test-shell?

No - i just meant that the git url you had there will only work for you
with your ssh keys :)

On 7 December 2012 22:34, Andy Doan <email address hidden> wrote:

> On 12/06/2012 09:02 PM, Yongqin Liu wrote:
> >> Before going too far into this, I wanted to mention my general thought:
> >>
> >> This seems to be using ADB to generate workload intensive stuff. I think
> >> this is probably the wrong way to be doing this. I think the bigLITTLE
> >> switcher tests is a better approach where its written entirely for the
> >> target. This seems way better, and will ultimately better fit with our
> >> lava-test-shell efforts.
> >
> > not see the detail implementation of workload test, but it seems to
> download
> > the apk of nbench, and install it to device, and do other things.
>
> I think this could be a quick patch to lava-test-shell. We have "deps"
> support for Ubuntu. We could do similar for Android.
>
> > and for this workload test, are all the steps need to be run on host side
> > able to be done with lava-test-shell?
> > git clone
> > update ip in config
> > run python workload_setup_dependencies.py
> > python workload.py ${config_file} ${outputdir}/
>
> We don't have to do the exact steps. git-clone is supported. not sure
> about the ip update. The setup deps could be done in an equivalent
> manner. Instead of thinking "how do we do this on host in python", I'm
> saying ask yourself - "how could I do this as a script that ran on the
> target"? I don't think it would be that hard.
>

I get your point here. But the whole benchmark is written by ARM to run
over adb + they even take control of serial terminal similar to LAVA( we
don't execute that path but rely on LAVA to reboot etc).

>
> If you absolutely think I'm crazy, then continue with this approach, but
> doing a "workload" test and requiring ADB seems like a bad idea.
>

adb is used only to transfer the files before and after the benchmark is
run and not while the benchmark is running.

>
> >> On 12/06/2012 07:06 AM, Yongqin Liu wrote:
> >>> + git_url="ssh://
> <email address hidden>/srv/linaro-
> >> private.git.linaro.org/people/bhoj/workload-automation.git"
> >>
> >> That's not going to work for anyone but you.
> > Or, what you mean is to rewrite the workload test via lava-test-shell?
>
> No - i just meant that the git url you had there will only work for you
> with your ssh keys :)
>
>
> --
>
> https://code.launchpad.net/~liuyq0307/lava-android-test/workload/+merge/138448
> You are requested to review the proposed merge of
> lp:~liuyq0307/lava-android-test/workload into lp:lava-android-test.
>

On 12/07/2012 11:13 AM, vishal wrote:
> I get your point here. But the whole benchmark is written by ARM to run
> over adb + they even take control of serial terminal similar to LAVA( we
> don't execute that path but rely on LAVA to reboot etc).

So use lava-android-test.

Its a little frustrating though:

* Android guys complained for over a year about "no blackbox testing".

* Benchmark guys complained about all Android results being invalid
because of the same basic thing.

We now have a solution in place, and you guys don't seem to be
interested in using it.

>> >
>> >If you absolutely think I'm crazy, then continue with this approach, but
>> >doing a "workload" test and requiring ADB seems like a bad idea.
>> >
> adb is used only to transfer the files before and after the benchmark is
> run and not while the benchmark is running.

So that could be done in lava-test-shell :)

> On 12/07/2012 11:13 AM, vishal wrote:
> > I get your point here. But the whole benchmark is written by ARM to run
> > over adb + they even take control of serial terminal similar to LAVA( we
> > don't execute that path but rely on LAVA to reboot etc).
>
> So use lava-android-test.
If this means you aggree this branch for now, please approve this MP.
>
> Its a little frustrating though:
>
> * Android guys complained for over a year about "no blackbox testing".
>
what I know about this is for the network test problem which the adb connection that lava-android-test relies on.
> * Benchmark guys complained about all Android results being invalid
> because of the same basic thing.
not know the details, but from the what I can guess it's not a problem lava can resolve,
normally benchmark tools only displayed the result with GUI where command tools are not able to get.
I don't think lava-test-shell can help on this, if it can do, I am glad to know how it implemented.

> We now have a solution in place, and you guys don't seem to be
> interested in using it.
Actually at least I am interesting in it,
but if you ask me to reimplement an existing tool which works well with adb connection,
then I will want to know why, and know how much time it will take to reimplement.

> >> >If you absolutely think I'm crazy, then continue with this approach, but
> >> >doing a "workload" test and requiring ADB seems like a bad idea.
> >> >
> > adb is used only to transfer the files before and after the benchmark is
> > run and not while the benchmark is running.
>
> So that could be done in lava-test-shell :)

On 12/10/2012 08:18 AM, Yongqin Liu wrote:
>> On 12/07/2012 11:13 AM, vishal wrote:
>>> I get your point here. But the whole benchmark is written by ARM to run
>>> over adb + they even take control of serial terminal similar to LAVA( we
>>> don't execute that path but rely on LAVA to reboot etc).
>>
>> So use lava-android-test.
> If this means you aggree this branch for now, please approve this MP.

yes. it seems fine.

NOTE: we still have to iron out the private git access part of this issue.

Actually wait a bit. I think your git url still won't work

> Actually wait a bit. I think your git url still won't work
I will change it to use the local git repository tomorrow.

And BTW, I will need you to help to deploy this modification tomorrow then.
So in the design of lava-test-shell, can you make it possible that the deployment isno need when add new test?

On 12/10/2012 09:32 AM, Yongqin Liu wrote:
>> Actually wait a bit. I think your git url still won't work
> I will change it to use the local git repository tomorrow.
>
> And BTW, I will need you to help to deploy this modification tomorrow then.
> So in the design of lava-test-shell, can you make it possible that the deployment isno need when add new test?

Yep - lava-test-shell allows you to control your own test definitions
(and even keep them in git if you'd like)