lp:~smoser/curtin

Author: Scott Moser
Author Date: 2018-12-14 20:54:47 UTC

vmtest: Allow the explicit disabling of system_upgrade.

When testing with a single proposed upgrade as in:
   CURTIN_VMTEST_ADD_REPOS=proposed
   CURTIN_VMTEST_UPGRADE_PACKAGES=open-iscsi

There was no way to avoid enabling system_upgrade.
The normal behavior if CURTIN_VMTEST_ADD_REPOS has a value should be to
enable system_upgrade, but in this case we only want the single package
upgraded.

So the change here allows the user to turn that off by:
   CURTIN_VMTEST_SYSTEM_UPGRADE=false

Author: Scott Moser
Author Date: 2018-12-14 16:53:57 UTC

fix typo and use 'raises' rather than 'triggers' wrt an exception

Author: Scott Moser
Author Date: 2018-11-30 20:58:33 UTC

simplify a bit.

Author: Scott Moser
Author Date: 2018-11-26 15:59:18 UTC

add comment about fence post / size

Author: Joshua Powers
Author Date: 2018-11-15 21:37:27 UTC

vmtest: add disco tests

Author: Scott Moser
Author Date: 2018-11-16 21:43:40 UTC

unittest: change directory to tmpdir for testing relative files.

Previous commit 104331c3318 changed the behavior of the test
  test_relative_file_url
to test with an absolute path, when the point of the test was to
test relative.

The fix here does what was originally intended, testing that relative
file paths work and utilizing the tmp dir.

Author: Scott Moser
Author Date: 2018-10-29 18:56:07 UTC

add preserve_sources_list doc

Author: Scott Moser
Author Date: 2018-10-18 21:48:54 UTC

add doc about preserve_sources_list.

Author: Scott Moser
Author Date: 2018-10-17 13:36:02 UTC

Fix typo- thanks rharper.

Author: Scott Moser
Author Date: 2018-10-12 20:23:49 UTC

releasing curtin version 18.1-59-g0f993084-0ubuntu1~16.04.1

Author: Scott Moser
Author Date: 2018-10-02 19:28:46 UTC

vmtest: Fix typo in skip-by-date.

The previous commit made one skip by date change go backwards.
Instead of 2019 we had set it to 2018.

Author: Scott Moser
Author Date: 2018-09-28 16:28:12 UTC

bump out another month, after people return from Christmas holiday

Author: Scott Moser
Author Date: 2018-09-21 11:25:15 UTC

tools/jenkins-runner: Error if both filters and tests are given.

Previously if you gave both '--filter' and 'tests/vmtests/' arguments
to tools/jenkins-runner it would just silently ignore the test arguments.
Just fail instead of doing the wrong thing, and possibly leaving the user
believing that the tests they provided correctly ran.

Author: Scott Moser
Author Date: 2018-09-19 08:01:47 UTC

vmtest: boot ephemeral with 'ro' on the kernel command line.

As raised in bug 1792905, when 'ro' is present in on the kernel command
line and the squashfs (rooturl=) parameter does not have a /lib/modules
then copymods fails to copy the modules into the "real root" and
things fail.

Maas is booting with 'ro', so it makes sense for curtin to do the same.
Additionally, 'ro' is present in almost all cases on a command line.

Author: Scott Moser
Author Date: 2018-08-17 21:38:41 UTC

update changelog (New upstream snapshot 18.1-48-g6a776e15).

Author: Scott Moser
Author Date: 2018-08-13 20:21:04 UTC

flake8

Author: Scott Moser
Author Date: 2018-08-09 17:25:50 UTC

Add subcommand 'features'.

The 'features' subcommand just writes the curtin.FEATURES
to standard output, one per line.

Author: Scott Moser
Author Date: 2018-08-09 16:48:46 UTC

Add main so that 'python3 -m curtin' does the right thing.

We could (and did) call curtin as 'PYTHON -m curtin.commands.main'.
That is just overly commplicated. This makes it work as
 PYTHON -m curtin

Author: Scott Moser
Author Date: 2018-08-09 00:46:08 UTC

consistency for logged_time

Author: Scott Moser
Author Date: 2018-08-07 13:35:13 UTC

increase size of root filesystems.

I believe that this change removes the final 2G root (/) filesystems.
These 2G filesystems were too small to handle upgrades sometimes.
Things that lead to this state:
 a.) we started testing with propsed enabled (and upgrade to it)
 b.) the failing cases here were Trusty HWE tests, which end up
     getting 2 kernels installed and 2 update-initramfs and such.

That worst case scenario led to the following output of apt:
  Need to get 2404 kB of archives.
  After this operation, 10.9 MB of additional disk space will be used.
  Need to get 1467 kB of archives.
  After this operation, 4904 kB of additional disk space will be used.
  Need to get 68.3 MB of archives.
  After this operation, 308 MB of additional disk space will be used.
  Need to get 63.9 MB of archives.
  After this operation, 275 MB of additional disk space will be used.

Author: Scott Moser
Author Date: 2018-07-20 19:05:38 UTC

typo locally

Author: Scott Moser
Author Date: 2018-07-16 19:56:41 UTC

vmtest: Skip proposed pocket on dev release when 'proposed' in ADD_REPOS.

The -proposed pocket for an Ubuntu development release is somewhat
expected to be broken. So, when testing proposed with via setting
CURTIN_VMTEST_ADD_REPOS=proposed, this will skip adding the repo for
the development release.

It allows testing of development release with:
  CURTIN_VMTEST_ADD_REPOS=PROPOSED

Author: Scott Moser
Author Date: 2018-07-14 12:26:56 UTC

vmtest: Order the vmtest_pollinate late_command earlier.

The TestInstallUnmount verifies that 'install: {umount: disabled}' does
leaves filesystems mounted, but its late_commands were unmounting
the target. Those conflicted with the late_commands here as we
expected the fs to be mounted.

Fix here is just to move the un-mounting late commands till '99' time
frame and move the vmtest_pollinate one earlier.

Author: Scott Moser
Author Date: 2018-07-13 21:07:50 UTC

vmtests: Increase root fs to give upgrades to -proposed more space.

Our new jenkins job curtin-vmtest-devel-amd64-proposed runs with
CURTIN_VMTEST_ADD_REPOS=proposed . That causes an
'apt-get update && apt-get dist-upgrade' to occur in the installed
system.

The change here simply increases by 500M the size of the root
disk tests that use examples/tests/mirrorboot-msdos-partition.yaml.
We can simply do this because the md0-part1 was on md0 which was a
5G disk, but we were creating 2x2G partitions. So now we still have
some wiggle room there.

This particular build failed because of 2 reasons
a.) the HWE variety meant additional kernels installed
b.) the trusty images are well out of date at thsi point (2018-05-02)
so lots of apt downloads and upgrades.

Author: Scott Moser
Author Date: 2018-07-13 20:07:14 UTC

add a docstring in yaml file

Author: Scott Moser
Author Date: 2018-06-28 21:34:42 UTC

feedback: update doc string per rharper suggestions.

Author: Scott Moser
Author Date: 2018-06-26 13:31:50 UTC

vmtests: skip TrustyTestMdadmBcache until 2019-01-22.

TrustyTestMdadmBcache is not functional due to bug 1754581. This bug
affecting bcache on trusty GA kernels is not likely to be fixed soon.

Author: Scott Moser
Author Date: 2018-06-25 18:22:38 UTC

tox: use simplestreams from git repository rather than bzr.

Simplestreams is moving to git. Use the git repo rather than bzr.

Author: Scott Moser
Author Date: 2018-06-22 20:19:06 UTC

fix order

Author: Scott Moser
Author Date: 2018-06-20 13:17:14 UTC

vmtests: move skip-by date of xfs root and xfs boot out 1 year.

This pushes out the skip-by date of bug 1652822 for a year.
xfs root and xfs boot seems an unlikely use case, and thus a bug
not likely to be fixed. So we do not want to see warnings on it.

Author: Scott Moser
Author Date: 2018-06-07 19:00:19 UTC

releasing curtin version 18.1-25-g9d0e557e-0ubuntu1

Author: Scott Moser
Author Date: 2018-06-07 17:05:17 UTC

flakes

Author: Scott Moser
Author Date: 2018-06-07 15:13:23 UTC

Fix extraction of local filesystem image.

Curtin would fail to extract an image if the image was a
local filesystem path or a file:// url.

Also here is a fix for ensure_dir which would be tripped if you
call write_file("foo.txt"). In that path, ensure_dir would be called
with the result of os.path.dirname("foo.txt") which is "" and
os.makedirs("") would fail.

LP: #1775630

Author: Scott Moser
Author Date: 2018-05-29 16:32:22 UTC

subp: update return value of subp with combine_capture=True.

combine_capture previously would return None in the stderr response.
This was inconsistent with capture=True, combine_capture=False.
It seems more consistent to always return the bytes or string as
documented. Test case and docstring are updated accordingly.

Author: Scott Moser
Author Date: 2018-05-23 19:16:28 UTC

fix use of python3 by xenial-py27. thanks chad.

Author: Scott Moser
Author Date: 2018-05-22 19:11:39 UTC

flip the logic.

patch the source by default.

Author: Scott Moser
Author Date: 2018-05-21 16:12:33 UTC

pyflakes: fix unused variable references identified by pyflakes 2.0.0.

A newer version of pyflakes (2.0.0) was released over the last weekend.
It identifed some unused variables that version 1.6.0 did not identify.
The change here merely fixes those unused variables.

The fact that c-i hit this is not intended. A future fix is needed to
properly pin the version of pyflakes so that it does not happen again.

Author: Ryan Harper
Author Date: 2018-05-18 19:02:29 UTC

releasing curtin version 18.1-17-gae48e86f-0ubuntu1~17.10.1

Author: Scott Moser
Author Date: 2018-05-18 18:28:31 UTC

tox: add a xenial-py3 environment.

A previous commit used mock.assert_called, which is not
present in xenial versions of mock so build would fail on xenial.
Due to bugs that are present in the trusty versions of mock it would
pass there.

So we add a tox environment 'xenial-py3' and add that to the
default set of environments that are run with a 'tox' invocation.

Author: Scott Moser
Author Date: 2018-05-18 14:02:47 UTC

change variable name and also get version from ./bin/curtin.

this way we see the vmtest level and the version of curtin that will run.

Author: Scott Moser
Author Date: 2018-05-17 13:59:03 UTC

get test coverage to 100% of new lines.

Author: Scott Moser
Author Date: 2018-02-13 21:58:50 UTC

vmtest: always boot with BOOTIF and ip=::::BOOTIF:dhcp

This excercises part of boot that was excercised in MAAS but not here.
Cloud-initramfs-dyn-netconf translates BOOTIF to the device name.

ip= is described in klibc-utils package in
 /usr/share/doc/libklibc/README.ipconfig.gz

See bug #1749019 for some more context.

Author: Scott Moser
Author Date: 2018-04-13 01:09:28 UTC

take ryan's suggestion for ua_val replacement

Author: Scott Moser
Author Date: 2018-03-28 19:59:27 UTC

Drop ubuntu branch management script new-upstream-snapshot.

This script was used for managing our ubuntu/ branches.
Instead of maintaining it here, we have moved it to another location.
  https://github.com/cloud-init/qa-scripts/

Author: Scott Moser
Author Date: 2018-03-27 20:34:32 UTC

commit current state

Author: Scott Moser
Author Date: 2018-03-22 15:58:43 UTC

write the install collect file for psuedo test

Author: Scott Moser
Author Date: 2018-02-23 03:03:09 UTC

Remove grub2/xenial specific workarounds for zfsroot.

This removes the workarounds that were put into place for installation
of zfs on xenial. With the fix of bug 1527727, they are no longer
necessary.

Author: Scott Moser
Author Date: 2018-02-06 21:25:55 UTC

vmtest: Add Filesystem Battery test.

This test exercises filesystem creation, mount and unmount of all
supported filesystem types.

Also here is a fix for jfs filesystem creation. mkfs_jfs requires a '-q'
argument to force/quiet. Otherwise it will prompt to ask if you
really want to do that.

Author: Scott Moser
Author Date: 2018-02-12 15:33:00 UTC

vmtest: Remove xenial and trusty specific workaround for rooturl.

xenial and trusty required specifying rc-initrd-dns on the kernel
command line in order to enable dns transition from initramfs to
real root (rooturl). Those changes are now fixed in both releases
so we can remove this code.

The changes made the archive today. We would expect they
are in any daily image with a serial newer than that 20180212.

Author: Scott Moser
Author Date: 2018-03-01 20:59:51 UTC

better search for directory

Author: Scott Moser
Author Date: 2018-02-12 19:22:11 UTC

docs: Add HACKING.rst doc to top level dir and readthedocs.

This adds a general 'HACKING.rst' document to the top level curtin
directory and also publishes that content into readthedocs.io.

Also drops the obsolete 'development.rst'.

Author: Scott Moser
Author Date: 2018-02-07 17:46:17 UTC

Fix error in url_helper and vmtest from previous commit.

Previous commit had 2 issues:
a.) it pushed the majority of tests down "install from url" path,
    which was not intended. The intent was to use 'cp://' whenever
    the booted environment was the same as the target. The problem
    occurred when target_release was the default None.
    The fix here is just to cover target_release == None case.
b.) url_helper.download was broken for python2.7.
    The isue was just that we expected the bytes written to be returned
    from file().write. Python 2 file.write has no return value.
    A unit test is added to test that code.

Author: Scott Moser
Author Date: 2018-02-07 17:22:14 UTC

tools: fix curtainer to specify source version to apt-get source.

We were seeing version skew from the binary package to the source
package, and had recently added code to exit failure if we saw
that case.

This change will make curtainer explicitly request the version
of source that it has the binary for. It is possible that the
ppa would not have that version (if it had been supersceeded),
but hopefully the archive holds onto source for current binary
packages.

Author: Scott Moser
Author Date: 2018-02-06 21:59:13 UTC

address suggested change for _path_from_file usage.

Author: Scott Moser
Author Date: 2018-02-06 00:29:44 UTC

tests: remove xfs filesystem from basic uefi test.

The test_uefi_basic.py file reports to be 'basic' test of uefi.
However, it also tests xfs filesystem support.
In order to make it more basic, just drop the xfs.

The real motivation here is to later enable a precise (12.04)
installation test done inside a 14.04 or 16.04 environment.
Each of those scenarios cause mounting problems of the xfs partition
when the 12.04 kernel boots. That is because the mkfs.xfs command
enabled options that were not supported in the older kernel.

Author: Scott Moser
Author Date: 2018-02-05 21:59:23 UTC

tools: fix curtainer --source with trusty.

Older version of dpkg-parsechangelog in trusty does not seem to
support '--file=<file>' correctly. It does seem to support
--file <file> though. So we'll just use that format.

Author: Scott Moser
Author Date: 2018-02-05 20:17:11 UTC

apply_net: do not write to stderr directly.

Writing to standard error from apply_net method would leak those
messages through to stderr when running tox. There was no value
in writing the messages as 'curtin' would still print them.

Also make apply_net consistent with other sub commands by changing
'apply_net_main' to let exceptions bubble up.
'curtin apply-net' will still write a message with the error
and will exit non-zero. If --showtrace is provided, it will
print the stack trace.

Author: Scott Moser
Author Date: 2018-02-02 14:34:56 UTC

vmtest: Drop use of root-image.gz and maas2roottar.

The root-image.gz and its conversion into a root-tgz file are no
longer necessary. Dropping the legacy code has numerous benefits:
 a.) no longer download root-image.gz files to local mirror
 b.) do not uncompressing these.
 c.) do not use maas2roottar which required sudo to create the root-tgz.

Overall a big win.

Author: Scott Moser
Author Date: 2018-02-02 17:54:12 UTC

in-target: use ChrootableTarget so that unshare pid is used.

Because in-target was using 'chroot' directly, the unshare pid
logic was not getting utilized.

This fixes that up so that we use the ChrootableTarget.subp.
Also it fixes the interactive path to exit with the command's
exit code in python 3.4+.

Author: Scott Moser
Author Date: 2018-02-02 22:08:26 UTC

tools/curtainer: check that binary and source version are the same.

We saw a puzzling failure of vmtest against the daily archive.
The reason for failure was that the source package that was retrieved
via 'apt-get source' differed in version from the version installed.

This adds a check in curtainer that will exit failure if that is the case.

LP: #1747077

Author: Scott Moser
Author Date: 2018-01-31 20:29:37 UTC

fix modeline for shell syntax files.

The header/footer re-work added a vi modline with 'syntax=shell' for
sh or bash files. The correct syntax for vim is 'sh', not 'shell'.

The result was no syntax highlighting in vim on these files.

Author: Scott Moser
Author Date: 2018-01-29 21:26:48 UTC

fix date from string

Author: Scott Moser
Author Date: 2018-01-29 16:54:42 UTC

change /ROOT/ubuntu to be /ROOT/zfsroot

'ubuntu' seems not really right here.

Author: Scott Moser
Author Date: 2018-01-24 19:12:16 UTC

tests: revert workaround for bug in overlayroot that is now fixed.

This just removes a workaround that was put in place until bug 1733183
was fixed in maas images.

Author: Scott Moser
Author Date: 2018-01-25 21:36:34 UTC

tests: Bump skip_by_dates for bug 1652822 and 1671952.

Also here a usability improvement on skip_by_date that is demonstrated
in the shorter calls.

Author: Scott Moser
Author Date: 2018-01-25 19:25:37 UTC

install.py: word changes on a comment.

Author: Scott Moser
Author Date: 2018-01-18 22:12:30 UTC

vmtest: Do not run tests of unsupported Ubuntu releases.

As somewhat expected attempts to install unsupported ubuntu releases
may fail. Example was today the Zesty tests would fail.
An 'apt-get update' would fail as the official mirrors have been
emptied of zesty already.

This uses ubuntu-distro-info (if installed) to skip unsupported
releases.

The user can provide a list of releases to skip with:
  UNSUPPORTED_UBUNTU="artful,zesty,trusty"
(comma or space separated).

Author: Scott Moser
Author Date: 2018-01-19 19:57:09 UTC

few final cleanups.

Author: Scott Moser
Author Date: 2018-01-18 14:25:30 UTC

tests: cleanup the apt custom sources test.

This test must have gone through some iteration in development.
It was testing in a very white-box way, and not really even validating
that the config was as expected (as made evident by the previous
lack of use of EXPECTED_CONVERTED_CONTENT).

The test now asserts that the two expected changes are made.

Author: Scott Moser
Author Date: 2018-01-18 20:38:53 UTC

fix copy/paste bug.

Author: Scott Moser
Author Date: 2018-01-16 21:59:52 UTC

packaging: Do not mention primary contributors in debian/changelog.

new-upstream-snapshot gives credit to committers by putting their
name in square brackets on the debian/changelog entries.

For some time in cloud-init we've stripped myself from that list.
I think it makes sense to strip out primary contributors to avoid
noise, but still give credit to new or less frequent contributors.

Author: Scott Moser
Author Date: 2018-01-16 18:35:32 UTC

packaging: Update new-upstream-snapshot to work with git.

After the move to git, we need to update new-upstream-snapshot.
This version comes from cloud-init, with an extra hunk in it to
handle the transition from bzr.

Commit new-upstream-snapshot to master, where it previously
only lived on ubuntu branches. Then, commit it also to the branches
so that it does not cause conflict.

Author: Scott Moser
Author Date: 2018-01-16 15:24:45 UTC

Bump pylint to 1.8.1.

Increase the version of pylint from 1.7.4 to 1.8.1

Author: Scott Moser
Author Date: 2018-01-09 20:58:03 UTC

move rm to after write for consistency

Author: Scott Moser
Author Date: 2017-12-19 19:13:40 UTC

debian/control: drop conflicts that had bzr version

this is fine as the bzr version (~bzr54) is not in any supported
release.. trusty is at 126-0ubuntu1 in released version even.