Merge lp://qastaging/~xnox/ubuntu-release-upgrader/lp1409555 into lp://qastaging/ubuntu-release-upgrader

I've search my local archive of bug attachments, specifically sources.list files and VarLogDistupgradeAptclonesystemstate.tar.gz (contains sources.list), and couldn't find the comments you are looking for translated at all so that's good.

To test it "for-real" we could build the package, copy over the vivid.tar.gz to a utopic system and try upgrading.

There was a failure with the test that I've commented on in-line.

On 13 January 2015 at 22:50, Brian Murray <email address hidden> wrote:
> There was a failure with the test that I've commented on in-line.
>
>> +
>> + sources_file = apt_pkg.config.find_file("Dir::Etc::sourcelist")
>
> The test was failing for me and it seems to because the sources_file isn't saved when updateSourcesList is called. This is why the other tests use self._verifySources() whcih I've switched to using for your test.
>

Hm, this could be due to environment changes. I'm developing this on Trusty.

_verifySources() is not sufficient for this test. _verifySources only
checks that "these lines are in the file", I however need to test for
"extras.ubuntu.com lines _are removed from the file_" Thus
_verifySources was passing the test for me, with or without, the
patch. I don't believe wee have done outright removal ever before
(only commenting things out).

I'll check the branch / and test on utopic/vivid in chroot to see what
is going on.

>> + self.assertEqual(open(sources_file).read(),"""deb http://archive.ubuntu.com/ubuntu gutsy main restricted
>> +deb http://archive.ubuntu.com/ubuntu gutsy-updates main restricted
>> +deb http://security.ubuntu.com/ubuntu/ gutsy-security main restricted
>> +
>> +""")
>> +
>> def test_powerpc_transition(self):
>> """
>> test transition of powerpc to ports.ubuntu.com
>>

--
Regards,

Dimitri.

In the process of tracking down the test failure I also found a method for removing entries from sources.list, sources.list.remove(), and that seems like a better idea to me than creating a new list.

On 13 January 2015 at 23:00, Brian Murray <email address hidden> wrote:
> In the process of tracking down the test failure I also found a method for removing entries from sources.list, sources.list.remove(), and that seems like a better idea to me than creating a new list.

That was my original implementation, however it's no good. As it
iterates the list for each removal invokes comparators to remove the
first item with the matching value from the list. Given that list is
"objects" rather than strings, the value for the "commented" string
did not match. Furthermore in python one is not allowed to modify a
list whilst iterating over it, thus a new list (or copy will be
required). This single pass is cumbersome, but most straight-forward
way to implement it. Or e.g. python3 on trusty has some subtle object
comparison error.

If it works and passes the upgrade test with both trusty & utopic
python3, I'm all happy to use whichever algorithm that gives correct
end-result.

--
Regards,

Dimitri.

I've now pushes fixes for all open() calls and logging.warn.

The build ends up using system installation of the ubuntu-release-upgrader.... one cannot remove it as the source code has circular depenency on itself (system update-notifier must be installed, which depends on the system ubuntu-release-upgrader)

...

$ sudo rm /usr/lib/python3/dist-packages/DistUpgrade
$ bzr bd

Above two commands make the build complete including the full test suite, in the current branch.

Note that in the pre-build.sh:

 "(cd utils && ./demotions.py utopic vivid > demoted.cfg)"

passes for me on Trusty, but fails on Vivid, which may be related to changes/regressions in vivid's python-apt. However, I believe that is not related to my changes.

Hi Dimitri, thanks a lot for this work. The branch looks good, whats puzzling is that I see a test failure when running bzr-buildpackage. It looks like nosetests3 is not using the current directory and not honoring PYTHONPATH here which is a bit confusing. Only when python3-distupgrade is not instaleld do the tests work for me when run with nosetest3. Beside this the diff looks fine and I think we should upload it.