…ical#5449)

…onical#5450)

Avoid integration test dependencies on PPAs that we don't own.

The canonical-kernel-team PPA will trail publication of packages
to the devel series sometimes by many weeks. Also, their process
may change in the future and the daily canonical-kernel-team PPA
may not be used at some point in the future.

Some of our apt-related integration tests are only asserting that
different formats of user-data can setup and assert valid PPA
signing keys to 3 separate PPAs on a single instance launch to avoid
the costs of launching three separate instances to test different
user-data formats.

For these tests, we can hard-code a known series and PPA without
degrading the value of the simple test.

Shift our canonical-kernel-team PPA reference to curtin-dev daily
PPA for two reasons:
- we have tighter coupling with the curtin dev team to keep in step
  with any process changes
- ppa:curtin-dev/daily already has a 4096-bit signing key instead of
  the stale 1024-bit key in canonical-kernel-team's PPA which may
  change in the near future.

'metalink' config can be specified instead or along with 'baseurl' in the yum
repository config. Add support for specifying metalink instead of 'baseurl'.

Fixes canonicalGH-5359

Signed-off-by: Ani Sinha <[email protected]>
Co-authored-by: Ben Gray <[email protected]>

klibc's ipconfig format [1] states that PROTO values 'none', 'off',
'static' and blank all mean no autoconfiguration, but cloud-init parser
is too strict and accepts only the first.

LP: #2065787

[1] https://git.kernel.org/pub/scm/libs/klibc/klibc.git/plain/usr/kinit/ipconfig/README.ipconfig

…nical#5435)

get_connection() is deprecated in requests 2.32.2+ so this will
allow for the LXDSocketAdapter to avoid using a deprecated api.

Fixes canonicalGH-5434

Signed-off-by: eaglegai <[email protected]>

Update based on 2c09f69

LXD instances don't have a defined datasource_list and therefore
implicitly exercise ds-identify. Add coverage for all other platforms.

Fixes canonicalGH-5368

Do not raise an error when userdata files are missing
in cloud-init query operations.

Fixes canonicalGH-5396

This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.

…#5411)

This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.

…l#5411)

Convert the remaining half of the module docs to be rendered by the new template
system.

Includes the documentation on how to document modules under this system.

Add additional developer script tools/format_yaml_doc to reformat either
data.yaml or example.yaml cloud-config files for module documentation.

Co-authored-by: Chad Smith <[email protected]>

…nical#5365)

This field is required by the OpenStack network_data.json parsing code
but is not supported by the VLAN object in the cloudinit network schema v1.

Fixes canonicalGH-5364

…ical#5456)

Both hyphenated and underscore delimited key names are permitted
by cloudinit/distros/ug_util.py#L114 due to magic replacement
of key names.

Since this is still valid json schema, add the necessary hyphenated
aliases for all users/groups keys. Because the goal in the future is
to only support one config key for a given configuraion option, add
deprecated keys to those schema definitions.

Also drop the description key from the deprecates lock-passwd schema
key.

Any deprecated schema key which provides a suggested replacement should
not provide duplicated key descriptions as the preferred replacement
will provided the necessary context.

Fixes canonicalGH-5454

…anonical#5367)

There is a discrepancy between the properties key name formatting in
the OpenStack network_data.json and cloudinit network-config.json
specifications. Ensure `bond_` is translated to `bond-` when the
OpenStack configuration is parsed by cloudinit.

Fixes canonicalGH-5366

Co-authored-by: Alberto Contreras <[email protected]>

After 70f7e78, mac_address is not
rendered for in openstack for network config v1. Fix the unit test to
reflect that.

8906e17 is causing some test failures on older releases.
Fix them

Commit 8470af0 changes the integration test keyserver PPA to
ppa:curtin-dev/daily. Update the assertion key fingerprint.

Adapt util.should_log_deprecation to expect two params:
- version: where the key/feature was deprecated
- boundary_version: at which deprecation level logs occur

Per [1], org's correct type is string. Added as new type and deprecated
integer.

References:
[1] https://github.com/candlepin/subscription-manager/blob/b6fad11e7783ae414fe88fdecee57d8db5c8e292/man/subscription-manager.8#L589

Fixes canonicalGH-5382

Co-authored-by: pneigel-ca <[email protected]>

…l#5474)

…anonical#5475)

The ubuntu_advantage key is deprecated in 24.1 in favor of ubuntu_pro.

Any version less than 24.1 will have only an info level log resulting in
no DEPRECATED keys from cloud-init status --format=yaml.

Parse log level messages in /var/log/cloud-init.log instead.

- Update deprecation test to include focal.
- Update test using `cloud-id` appropriately

Remove code for detecting a virtualenv. This has been broken since ~3.7.

Invalid type annotations confuse mypy.

When it sees a type annotation that doesn't conflict with other
annotations, it is trusted. This means that future attempts to
conditionally check a variable's type will cause mypy to produce an
unreachable code path warning.

Later use of the same variable name leads mypy to report unreachable code
paths.

Mypy infers an unannotated variable's type from its initial value.
When a variable's type may change (such as from None to non-None),
the variable's type needs to be annotated, otherwise mypy may warn
of unreachable code paths.

Enable mypy's "warn_unreachable" checks.
Remove adjacent unused mocks.

Note: mypy can incorrectly report typing errors as "unused code paths"

Python's provides type annotation features which allow validating types
throughout the code base. Use, rather than ignore, these features.

Note that since a dict's .get() may return None, using this method
when the only possible return value is a literal will confuse mypy.

Enable positional arguments for various environments.
Add missing deps for tip-mypy.
The -tip envs were broken when a non-tip env existed. In these cases,
{[testenv:mypy]commands} resolved to the non-tip environment. Fix it.

Fix the current network v2 schema as well.

This doesn't add anything useful on its own, but it defines a custom
type which can easily be overridden.

See canonicalGH-5398 for more details.

This also satisfies mypy via type narrowing, which hasattr() does not
accomplish.

jsonschema is still a soft dependency, so don't fail it not present.

Update test to reduce expected deprecation count when
below deprecation boundary. changed_version messages are still logged.

Also use ca_certs instead of ca-certs deprecated key

Bump the version in cloudinit/version.py to 24.2 and
update ChangeLog.

The /etc/hostname.* files should have the mtu on
a separate line otherwise it gives error:

  ifconfig: mtu: bad value

The lines are executed in order by ifconfig and
mtu should be on it's own line.

Fixes: canonicalGH-5413

)

When there is no IPv6 addr given in the customization configuration,
we shall set IPv6 type to dhcp6, then customized Linux network will be
set to dhcp IPv6 explicitly.

* Remove `unittest` constructs and remove base classes.
* Replace tests that don't test things with tests that do
* Add fstab and mounts combinations test

canonicalGH-5445

There are no call sites requesting not decoding the environment vars.
This change decodes then always, simplifying typing and logic.

Instead of a broad try/except, do properly check for conditions that
invalidate a mount location.

Support for jumbo frames requires that the underlying physical interfaces
and the parent bond interface all have the larger MTU configured, not just
the physical interfaces.

…canonical#5501)

At least one of (or both) 'baseurl' or 'metalink' should be provided for yum
repository specification. Add schema changes to enforce it. Without this,
with just 'metalink' property set, one would get the schema validator error

\---
Error: Cloud config schema errors: yum_repos.epel-release: 'baseurl' is a required property
\---

Signed-off-by: Ani Sinha <[email protected]>

On systemd, services are started by PID 1. When this doesn't happen, cloud-init
is in an unknown run state and should warn the user.

Reorder pid log to be able to reuse Distro information.

Add docstring deprecating util.is_Linux().

…5209)

Implement verify_clean_boot() to ignore certain expected logs
in a platform-specific way.

…anonical#5209)

Ensure ignore_warnings=True or ignore_errors=True is honored and
not overridden by supplemental warning texts appended.

…l#5209)

Avoid using warning level messages as there may be some
use-cases in the wild that need to invoke cloud-init boot
stages after boot for some reason unknown to upstream.

Provide a detailed warning message informing admins to file
issues against cloud-init to better represent those feature
needs before dropping this feature altogether.

…onical#5504)

Commit 604d80e introduced assertions expecting exit 2 from the
CLI when calling cloud-init init --local. Revert this test assertion
as only cloud-init status command exits (2) on deprecations/warnings.

Invoking cloud-init's boot stages on the commmand line will only exit
1 if critical errors are encountered to avoid degrading overall
systemd health as seen from cloud-init systemd units. When cloud-init
boot stages encounter recoverable_errors of any type, there is no
need to exit non-zero as those deprecation logs are not-critical to
the health of the system as a whole.

It is pretty consistently failing due to canonical#5373 with no fix in
sight.

Ensure DNS server addresses are parsed from the proper location
of network_data.json

Fixes canonical#5386

Co-authored-by: Alberto Contreras <[email protected]>

If DNS information is added to a NetworkManager managed interface where
the given protocol family is disabled, NetworkManager will be unable to
activate the interface.

canonical#5387

'mirrorlist' config can be specified instead or along with 'baseurl' in the yum
repository config. Add support for specifying mirrorlist instead of 'baseurl'.

Fixes canonicalGH-5520
Signed-off-by: Ani Sinha <[email protected]>

9929a00 added the ability to used a cached datasource when none is
found. This was supposed to be per-datasource, but the lack of cache
cleaning got applied universally. This commit makes it so cache will be
cleaned as it was before if fallback isn't implemented in datasource.

Fixes canonicalGH-5486

This is useful for logs we want hidden by default but can be turned
on via configuration.

Switch to pathlib where appropriate and call consistently