lp:~raharper/curtin

Author: Ryan Harper
Author Date: 2021-03-02 21:19:14 UTC

vmtest/centos: fix path to grub config file

Centos images used to always have grub installed at:

/boot/grub/grub.conf

The most recent build of centos70:
  centos/centos70/amd64/20210210_01/root-tgz

Now has grub conf here:

/boot/grub2/grub.cfg

This patch checks for either path before running the sed fixup
command.

Author: Ryan Harper
Author Date: 2021-01-14 21:05:25 UTC

Manually fix up tox issues from automatic adding if Hirsute

The automated tool isn't perfect; mostly issues with indentation.
One more refactor of the really-long-class-names in network disabled
test case as well.

Author: Ryan Harper
Author Date: 2021-01-14 20:32:39 UTC

vmtest: fix image-sync after maas URL stream rename

Image syncing for vmtest has been failing since MAAS renamed their
daily image stream to stable. In an effort to keep the 'daily' URL
valid the website redirects this to 'stable' path. However the content
was also modified and includes references to files and URLs which have
the 'stable' name included. This broke vmtest image sync which expected
the 'daily' string to be present in filenames and content.

The observable result was that curtin would attempt to find content it
had just *synced* to /srv/images but when searching, it would load up
the cached content files which only referenced the 'daily' stream data
and would indicate that the target file was missing.

This branch moves vmtest to using the 'stable' maas image stream
directly. This value is controllable via an environment variable,
MAAS_IMAGE_STREAM, if set will be used to compose the IMAGE_SRC_URL
and the STREAM_BASE variable which is used to find cached content in
/srv/images/.vmtest_data/ directory.

LP: #1908543

Author: Ryan Harper
Author Date: 2020-12-08 18:17:49 UTC

curthooks: verify secondary ESP partitions have fat32 format

Ignore partitions with flag: boot which are not FAT32 formatted as
they are not actual ESPs.

LP: #1907280

Author: Ryan Harper
Author Date: 2020-12-02 17:21:56 UTC

install_grub: Fix bootloader-id for RHEL systems, must be redhat

In 7310b4fe61465, the port from shell dropped a change where the boot-id
value for Redhat family OSes is the value 'redhat' not the os-release ID
value (which is rhel, on RHEL systems).

Add this change back to install_grub and fix unittests that fail with
'rhel' instead of 'redhat' in os-release values. Hardcode 'redhat'
paths in expected values.

LP: #1906543

Author: Ryan Harper
Author Date: 2020-11-22 22:32:57 UTC

vmtests: Replace Eoan test with Groovy

Author: Ryan Harper
Author Date: 2020-10-21 21:52:21 UTC

clear-holders: fix identification of multipath-partitions

identify_partition calls multipath.is_mpath_partition using at sysfs
path to the device but is_mpath_partition expects a /dev/XXX path.
This prevented clear-holders from identifying multipath partitions which
need to be handled differently than regular partitions (or disks).

LP: #1900900

Author: Ryan Harper
Author Date: 2020-10-07 14:54:32 UTC

vmtests/multipath-lvm: dont assume device-mapper block names

- Add comments in the early_command code to document what it is doing
- Use the /device/mapper/mpatha symlink to determine the correct
  device-mapper device name when operating on it
- Add additional commands to capture more information for future
  debugging issues with these tests.

LP: #1898758

Author: Ryan Harper
Author Date: 2020-10-06 16:56:06 UTC

Don't use apt-add

Author: Ryan Harper
Author Date: 2020-09-10 15:11:57 UTC

Skip Groovy IPV6 networking, same issue LP: #1888726

Author: Ryan Harper
Author Date: 2020-09-09 16:51:09 UTC

distro: run apt-get clean after dist-upgrade, install, upgrade operations

Some vmtest scenarios install multiple large packages which consume storage
temporarily and in some cases prevent other install operations, like creating
an initramfs, from completing successfully. This change introduces an
apt-get clean command run after dist-upgrade, install or upgrade actions.
This will clear out the local repositiry of retrieved package files restoring
space back to the filesystem.

Author: Ryan Harper
Author Date: 2020-09-10 14:33:14 UTC

Refactor uefi_remove_duplicates into find/remove functions for reuse

Splitting the function into find/remove functions allows reuse in the
vmtest which exercises the code and then verifies the duplicate
removals.

- Remove skip_by_date on Centos7 in test_reuse_uefi_esp
- Use uefi_find_duplicate_entries to verify vmtest results

Author: Ryan Harper
Author Date: 2020-09-09 15:52:55 UTC

Add unittest for missing BootCurrent value when removing duplicates

Author: Ryan Harper
Author Date: 2020-08-21 17:11:19 UTC

block: disk_to_byid_path handle missing /dev/disk/by-id directory

If a system has disks without any persistent indentifier (like serial)
then udev won't create any /dev/disk/by-id symlinks. This can happen
in VMs which do not provide serial numbers by default (e.g. virtio).

If /dev/disk/by-id did not exist then os.listdir on that path would
raise a FileNotFoundException and prevented disk_to_byid_path from
returning None (or the correct path).

This is fixed by checking if the target prefix exists and if not
return an empty dict.

LP: #1876258

Author: Ryan Harper
Author Date: 2020-08-21 14:47:45 UTC

Add some more details around grub.reorder_uefi configuration

Author: Ryan Harper
Author Date: 2020-08-19 21:50:43 UTC

curthooks: uefi_find_grub_device_ids handle type:mount without path

When scanning a curtin storage config for efi mount entries we failed
to handle the case where an entry does not contain the 'path' key.
Curtin storage config type:mount does not require a 'path' element.
Change to using .get('path') when looking for efi mount path.

LP: #1892242

Author: Ryan Harper
Author Date: 2020-08-18 22:05:44 UTC

mask snapd.seeded.service before container starts

Author: Ryan Harper
Author Date: 2020-08-18 21:52:31 UTC

Only add deb-src if $getsource is set

Author: Ryan Harper
Author Date: 2020-08-14 15:48:50 UTC

netplan openvswitch yaml changed

netplan's openvswitch yaml format has changed and now openvswitch
is not a top-level type but a subkey under various top-level keys.
This change will allow for top-level or subkey matching and retains
the check on renderer type (whether netplan supports that for
openvswitch or not; it's used for NetworkManager, networkd).

LP: #1891608

Author: Ryan Harper
Author Date: 2020-06-23 16:45:42 UTC

Adjust some working in the swap config:force documentation.

Author: Ryan Harper
Author Date: 2020-06-16 16:06:17 UTC

multipath: handle multipath nvme name fields correctly

NVME Multipath devices do not have typical mpath names, instead they
include colons and whitespace. Handle this by adjusting the multipath
maps command output to single quote the name field like we do for the
other fields.

LP: #1878041

Author: Ryan Harper
Author Date: 2020-06-08 18:23:34 UTC

vmtest: Allow testclass to enable 'unsupported releases' on case-by-case basis

Author: Ryan Harper
Author Date: 2020-06-01 18:53:40 UTC

vmtests: fix PreservePartitionWipeVg storage config

This vmtest had both wipe and preserve settings enabled; for disks
this is not correct, we can either wipe a disk or preserve it but
we cannot do both (unlike partitions where we can keep the partition
location, flag, size, etc, but wipe the data). This branch

- Fixes the storage config in the vmtest
- Add a RuntimeException indicating this config is invalid
- Add unittests for get_device_paths_from_storage_config

Author: Ryan Harper
Author Date: 2020-05-28 22:39:36 UTC

releasing curtin version 20.1-2-g42a9667f-0ubuntu1~16.04.1

Author: Ryan Harper
Author Date: 2020-05-28 20:35:30 UTC

releasing curtin version 20.1-2-g42a9667f-0ubuntu1

Author: Ryan Harper
Author Date: 2020-05-28 20:33:28 UTC

releasing curtin version 20.1-2-g42a9667f-0ubuntu1~20.04.1

Author: Ryan Harper
Author Date: 2020-05-28 20:24:56 UTC

releasing curtin version 20.1-2-g42a9667f-0ubuntu1~18.04.1

Author: Ryan Harper
Author Date: 2020-05-28 15:10:36 UTC

Fix flake8

Author: Ryan Harper
Author Date: 2020-05-27 19:44:53 UTC

vmtest: allow installed centos images to reboot, required for selinux relabing

LP: #1881011

Author: Ryan Harper
Author Date: 2020-05-27 19:38:11 UTC

helpers: remove grub-install shell-code and helper functions

Author: Ryan Harper
Author Date: 2020-05-26 21:32:12 UTC

releasing curtin version 20.1-0ubuntu1

Author: Ryan Harper
Author Date: 2020-05-26 17:43:49 UTC

Release 20.1

Bump the version in curtin/__init__.py to be 20.1.

LP: #1880741

Author: Ryan Harper
Author Date: 2020-05-22 22:22:25 UTC

Handle multiple separators which were found in TestAllindata vmtest

Author: Ryan Harper
Author Date: 2020-05-21 00:09:01 UTC

Remove unneeded local reboot_arg

Author: Ryan Harper
Author Date: 2020-05-20 17:19:49 UTC

Add unittest data file

Author: Ryan Harper
Author Date: 2020-05-18 15:44:05 UTC

doc/config: Document grub:update_nvram default value to match code.

Author: James Falcon
Author Date: 2020-05-14 16:53:03 UTC

Replace references to old team manager with new team manager

Author: Ryan Harper
Author Date: 2019-11-06 17:13:32 UTC

vmtest: bump default nr_cpus to 2 for more unsquashfs performance!

Author: Ryan Harper
Author Date: 2020-05-12 19:42:40 UTC

Replace 'l' for 'L' to make tip flake8 happy

LP: #1878236

Author: Ryan Harper
Author Date: 2020-05-07 20:44:00 UTC

releasing curtin version 19.3-68-g6cbdc02d-0ubuntu1

Author: Ryan Harper
Author Date: 2020-05-06 21:26:28 UTC

block-discover: ignore invalid id_serial/id_wwn values

When obtaining unique ids for identifying a block storage
device, block-discover looks for ID_WWN and ID_SERIAL values
and will use them in the rendered storage config. In some cases
the value of these attributes may be invalid. For example, some
consumer disks might have an invalid WWN number:

0x00000000000000000000

Curtin will also ingnore empty values, like
ID_SERIAL=' '

Excluded ids will not be emitted in the final storage config.

LP: #1876848

Author: Ryan Harper
Author Date: 2020-05-06 19:56:23 UTC

Drop bogus top-level vmtests directory from target_dirs

Author: Ryan Harper
Author Date: 2020-05-05 20:13:55 UTC

Simplfy config by dropping secondary disk and lvm config

Author: Ryan Harper
Author Date: 2020-04-29 13:58:35 UTC

Drop --mknode use in early_commands as well

Author: Ryan Harper
Author Date: 2020-04-28 21:03:58 UTC

Rework unittest and document expected result, being careful with quoting

Author: Ryan Harper
Author Date: 2020-04-27 01:02:31 UTC

Initial block-meta-async

Author: Ryan Harper
Author Date: 2020-04-23 20:40:35 UTC

Drop all use of dname in TestScsiBasic and use wwn instead, drop centos66 bits on scsi, not needed

Author: Ryan Harper
Author Date: 2020-04-23 00:18:31 UTC

Add unittests for CalcDmPartitionInfo, tox fixes

Author: Ryan Harper
Author Date: 2020-04-22 21:38:02 UTC

Don't use udevadm test (which does trigger rules, btw), be explicit with trigger

Author: Ryan Harper
Author Date: 2020-04-22 16:09:33 UTC

block-meta: device mapper partitions may be block devices not links

In some cases the multipath library may create a block device which
is not a symbolic link to the device-mapper dev[1]. Curtin expects the
symlink. Remove the non-symlink file and allow kpartx to create it.

1. https://bugzilla.redhat.com/show_bug.cgi?id=869253

Author: Ryan Harper
Author Date: 2020-04-21 23:46:59 UTC

block-meta: skip wipe device paths if not present

Curtin extracts the devices from storage config with wipe enabled.
These devices may not exist yet (curtin will be creating them) so
there's no need to clear them, instead drop them from the list of
devices to send to clear-holders.

Other changes:

multipath:
 - Clarify multipath logging on device type checking
block-meta:
 - Stop iterating through disk lookup keys after we find one
 - Use udevadm_settle after parted partition creation before
   calling kpartx to prevent race with kernel partition update
clear-holders:
 - Log the inputs to clear-holders
block:
 - Log a warning when sys_block_path is called on a non-existent
   device path.
 - Log the return value from block.lookup_disk()

LP: #1869075

Author: Ryan Harper
Author Date: 2020-04-21 22:12:55 UTC

Mock out zfs_supported to prevent attempting to load kernel modules

Author: Ryan Harper
Author Date: 2020-04-21 17:19:58 UTC

vmtest: rsync don't cross filesystem boundaries when copying

In some scenarios /var/lib/lxcfs is mounted with kernel bits
and rsync complains; instead we now copy using the --one-file-system
flag which avoids crossing these boundaries.

LP: #1873909

Author: Ryan Harper
Author Date: 2020-04-21 00:53:57 UTC

Fix new calls of util.subp and multipath.is_mpath_member

Author: Ryan Harper
Author Date: 2020-04-20 15:47:43 UTC

lvm-over-multipath: handle lookups of multipath members

Subiquity provides a 'path' value for disks in storage config.
This triggered an edge case where curtin attempted to wipe the
underlying scsi disk *while* multipath is enabled resulting in
an error when attempting to get exclusive access to the disk.

This branch resolves this by checking if a disk or partition
is related to multipath and if so returning the device mapper
disk path.

LP: 1837214

Author: Ryan Harper
Author Date: 2020-04-20 20:53:44 UTC

Fix reuse-esp on ubuntu, mark ESP with grub_device, fix missing part flags

Author: Ryan Harper
Author Date: 2020-04-20 19:58:19 UTC

vmtest: basic use dname to lookup disk with multiple partitions

Author: Ryan Harper
Author Date: 2020-04-20 19:01:30 UTC

block-discover: handle missing multipath 'path' data, use DM_NAME

Probert's multipath data (maps and paths) is not always complete.
In some systems, the paths output may not include the required
mpath friendly name and instead use a serial number. In such
cases we can also check DM_NAME in the multipath device udev data
to extract the multipath_id that we can use to search for a member
device.

LP: #1873728

Author: Ryan Harper
Author Date: 2020-04-20 16:10:51 UTC

Adjust docstring, remove filter comment.

Author: Ryan Harper
Author Date: 2020-04-16 17:17:18 UTC

Don't check the ptable type of a disk with no ptable

Author: Ryan Harper
Author Date: 2020-04-16 16:58:04 UTC

block: check for fdasd command before using

Author: Ryan Harper
Author Date: 2020-04-15 16:51:25 UTC

curthooks: always use ChrootableTarget.subp when calling efibootmgr

There was a bug recently introduced when adding the feature to remove
duplicate UEFI boot entries which removed use of ChrootableTarget
subp method. The assumption was that util.subp calls would also get
the target parameter set but this was not the case. The ephemeral
environment does not usually have efibootmgr package installed but
we ensure that it is present in the target system. This branch
replaces calls to util.subp with the ChrootableTarget subp method.

Author: Ryan Harper
Author Date: 2020-04-15 16:24:58 UTC

Fix multipath boot issues

- When writing multipath bindings file, use the disk name only
  (mpatha) not the target device (mpatha-part1) or boot will fail.
- When possible, copy the ephemeral multipath bindings file to target; it is
  autogenerated with the correct value.
- Fix ephemeral kernel/initrd kflavor selection when booting Centos; the
  'lowlatency' does not provide all of the required modules; we now default
  to 'generic'.
- Fix centos initramfs config issue when using lvm *and* multipath
- Refactor multipath filter, used | intead of / for delimiter and allow
  multiple filters to be combined.

Author: Ryan Harper
Author Date: 2020-04-09 22:44:18 UTC

Change unittest to ensure we call the blockdevparser.parse() path

Author: Ryan Harper
Author Date: 2020-04-08 17:06:00 UTC

Adjust setup script to recreate failure

Author: Ryan Harper
Author Date: 2020-04-07 21:11:19 UTC

Fix assertEqual on call_count

Author: Ryan Harper
Author Date: 2020-04-07 16:05:38 UTC

Rename file to match conf_file change

Author: Ryan Harper
Author Date: 2020-04-06 21:11:28 UTC

vmtest: shuffle test-cases to randomize load on host

Author: Ryan Harper
Author Date: 2020-03-30 18:46:42 UTC

format: extra_options should be a list type

Switch to a list for extra_options to allow for specifying whitespace
characters in values passed to mkfs command.

- tox: add block-schema validate env

Author: Ryan Harper
Author Date: 2020-03-27 19:24:35 UTC

Don't enable the vmtest by default, images are not published

Author: Ryan Harper
Author Date: 2020-03-26 21:45:05 UTC

Refactor how extra_options value is added to mkfs command

- Split extra_options by whitespace
- Update unittest
- Implement unittest verification in filesystem_battery vmtest

Author: Ryan Harper
Author Date: 2020-03-23 21:40:02 UTC

tox.ini: Fix issues with newer tox on focal

Newer tox on focal is not happy when inhieriting dependencies.
First in py27 testenv, it does not pickup the 'testenv' deps
which includes test-depedndences and dependencies files. Fix
this by explicitly referencing these deps fir testenv:py27.

Second, in testenv:trusty-p27, which recreates specific package
versions to model the trusty build environment we cannot pull in
the default testenv deps, as this breaks with different versions
of mock and other settings. Instead, just bring in the setuptools
dep to ensure we can keep running on py27 for now.

Author: Ryan Harper
Author Date: 2020-03-11 01:34:32 UTC

Skip Centos tests on non amd64

Author: Ryan Harper
Author Date: 2020-03-11 17:21:54 UTC

Add vmtest unittest to verify id_serial contents

Author: Ryan Harper
Author Date: 2020-03-11 17:04:14 UTC

vmtest: use util.load_file for loading collect files

In some cases the content inside collect files may need to be
decoded. Instead of open-coded read and decode, use the common
util.load_file() which already handles decoding data.

Author: Ryan Harper
Author Date: 2020-03-10 20:48:39 UTC

Add fixes from Chad

Author: Ryan Harper
Author Date: 2020-03-06 23:29:59 UTC

Skip bionic probert, it's not new enough

Author: Ryan Harper
Author Date: 2020-03-04 20:42:40 UTC

tox: all py27 environments should use the base py27 deps

We fixed base py27 deps to pin setuptools < 45 to fix an issue
with newer setuptools and py27 deprecation. We need source the
base py27 env deps in all py27 related environments.

Author: Ryan Harper
Author Date: 2020-03-04 20:31:09 UTC

Add non-duplicate entry with different path to verify we compare both path and name in entries before removing

Author: Ryan Harper
Author Date: 2020-03-03 23:16:17 UTC

Add github issue related to the change

Author: Ryan Harper
Author Date: 2020-02-25 23:17:09 UTC

block-discover: dasd disks need device_id, vtoc if present

Refactor discover to include device_id in type: disk so
partitions can lookup the path to the disk. If the dasd
device is formatted, include ptable: vtoc. Add additional
unittests to verify this behavior.

Author: Ryan Harper
Author Date: 2020-02-14 01:09:47 UTC

Avoid the copy, detect via user-data script, use makecache to refresh metadata before install

Author: Ryan Harper
Author Date: 2020-02-07 17:44:07 UTC

Change baseclass name

Author: Ryan Harper
Author Date: 2020-02-06 19:50:30 UTC

Address feedback, add docstring, handle exception, add check on pnum

Author: Ryan Harper
Author Date: 2020-02-06 19:30:26 UTC

Fix up class names

Author: Ryan Harper
Author Date: 2020-02-04 20:29:14 UTC

unittest: fix clear-holders multipath remove parent mapping

Author: Ryan Harper
Author Date: 2020-02-04 21:13:50 UTC

vmtests: raid5_bcache convert to get_fstab_expected format

Author: Ryan Harper
Author Date: 2020-02-04 21:16:10 UTC

vmtests: bump fixby date out and fix false positive when date passes

The fix for vlan is still pending, however we've observed a false
positive where vmtest would indicate that the test was passing when
it actually was not. This is because skip_by_date will run the
decorated function *after* the fixby date has passed. The implemention
was 'pass' which does pass but did not actually run the test case.

Fix this by calling to the super class.

LP: #1855148

Author: Ryan Harper
Author Date: 2020-02-04 20:03:38 UTC

remove-vmtest-release: remove 'class' match case, fix calling clean-file

Author: Ryan Harper
Author Date: 2020-01-30 19:55:39 UTC

releasing curtin version 19.3-17-g50ffca46-0ubuntu1~16.04.1

Author: Ryan Harper
Author Date: 2020-01-30 19:52:53 UTC

releasing curtin version 19.3-17-g50ffca46-0ubuntu1~18.04.1

Author: Ryan Harper
Author Date: 2020-01-30 19:50:52 UTC

releasing curtin version 19.3-17-g50ffca46-0ubuntu1~19.10.1

Author: Ryan Harper
Author Date: 2020-01-30 17:35:46 UTC

clear-holders: add multipath related scenarios

Author: Ryan Harper
Author Date: 2020-01-29 02:28:31 UTC

Add comment on disk/part s390x by-path choice.

Author: Ryan Harper
Author Date: 2020-01-24 13:08:23 UTC

multipath: try harder when removing multipath devices, add verbosity

Author: Ryan Harper
Author Date: 2020-01-16 21:19:23 UTC

releasing curtin version 19.3-14-ga45a54ca-0ubuntu1

Author: Ryan Harper
Author Date: 2020-01-16 17:21:33 UTC

Skip the SCSI test, don't install

Author: Ryan Harper
Author Date: 2020-01-14 17:04:58 UTC

Revert 'The number of sync shall be three, no less, no more'

Author: Ryan Harper
Author Date: 2019-12-18 17:44:10 UTC

Bump XFS /boot skip-by date out a while

Author: Ryan Harper
Author Date: 2019-12-12 19:22:32 UTC

Move skip_by date for vlan bug out since we're already past the date and it's not fixed