Vendored modules

.github/workflows/tests.yaml

@@ -22,7 +22,7 @@ jobs:
 
     runs-on: ${{ matrix.os }}
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
       - uses: ruby/setup-ruby@v1
         with:
           ruby-version: ${{ matrix.ruby }}

Gemfile

@@ -11,6 +11,7 @@ end
 group :test do
   # Required for the final controlrepo tests
   gem 'toml-rb'
+  gem 'rexml'
 end
 
 group :development do

README.md

@@ -21,6 +21,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others
     - [Puppetfile](#puppetfile)
   - [Spec testing](#spec-testing)
     - [Adding your own spec tests](#adding-your-own-spec-tests)
+    - [Vendored Modules](#vendored-modules)
   - [Using Workarounds](#using-workarounds)
   - [Extra tooling](#extra-tooling)
     - [Plugins](#plugins)
@@ -33,6 +34,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others
     - [Ruby Warnings](#ruby-warnings)
     - [Rake tasks](#rake-tasks)
       - [generate_fixtures](#generate_fixtures)
+      - [generate_vendor_cache](#generate_vendor_cache)
 
 ## Overview
 
@@ -605,6 +607,30 @@ If you want to see Puppet's output, you can set the `SHOW_PUPPET_OUTPUT` environ
 
 `SHOW_PUPPET_OUTPUT=true onceover run spec`
 
+### Vendored Modules
+
+As of Puppet 6.0 some resource types were removed from Puppet and repackaged as individual modules. These supported type modules are still included in the `puppet-agent` package, so you don't have to download them from the Forge. However, this does not apply to the `puppet` gem used when spec testing. This frequently results in users wondering why their Puppet manifests apply just fine on a node, but their tests fail with messages like `Unknown resource type: cron_core` for example. A common workaround for this problem was to add said modules into your Puppetfile, thus requiring manual management.
+
+Onceover now has the ability to remove that manual process for you by querying Github's API to determine which versions are in use by the version of the [puppet-agent package](https://github.com/puppetlabs/puppet-agent/tree/main/configs/components) you are testing against.
+
+This functionality is opt in, so to use it configure the following:
+
+```yaml
+# onceover.yaml
+opts:
+  auto_vendored: true
+```
+
+or on the cli:
+
+```shell
+bundle exec onceover run spec --auto_vendored=true
+```
+
+Essentially what this is doing is resolving any of these [supported type modules](https://www.puppet.com/docs/puppet/8/type#supported-type-modules-in-puppet-agent) that are not already specified in your Puppetfile, and adding them to the copy Onceover uses to deploy into its working directory structure.
+
+CI/CD pipeline users are encouraged to provide Onceover with a cache of the module versions to test against in order to avoid hitting Githubs API ratelimit. To do so, the [generate_vendor_cache](#generate_vendor_cache) rake task can be used to populate the cache into your `spec/vendored_modules` directory.
+
 ## Using workarounds
 
 There may be situations where you cannot test everything that is in your puppet code, some common reasons for this include:
@@ -879,6 +905,12 @@ fixtures:
 
 Notice that the symlinks are not the ones that we provided in `environment.conf`? This is because the rake task will go into each of directories, find the modules and create a symlink for each of them (This is what rspec expects).
 
+#### generate_vendor_cache
+
+`bundle exec rake generate_vendor_cache`
+
+This task will query Github's API to determine the versions of the vendored modules in use by the version of the puppet agent you are testing against, and cache that information in `control-repo/spec/vendored_modules`. This way your pipelines won't need to reach out for this information each time Onceover is ran with `auto_vendored` enabled.
+
 ## Developing Onceover
 
 Install gem dependencies:

features/auto_vendored.feature

@@ -0,0 +1,27 @@
+@vendored @puppet6
+Feature: Automatically resolve modules vendored with puppet-agent package
+  Onceover should optionally attempt to resolve these vendored modules so that
+  users do not need to maintain these in their Puppetfile's unless they have a reason
+  to do so
+
+  Background:
+    Given onceover executable
+
+  Scenario: Auto resolve disabled and Puppetfile empty
+    Given initialized control repo "vendored"
+    When I run onceover command "run spec" with class "role::cron"
+    Then I should see error with message pattern "Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron'"
+
+  Scenario: Auto resolve enabled and Puppetfile empty
+    Given existing control repo "vendored"
+    When I run onceover command "run spec --auto_vendored=true" with class "role::cron"
+    Then the temporary Puppetfile should contain /mod 'puppetlabs-cron_core',\n.*git: 'https://github.com/puppetlabs\/puppetlabs-cron_core.git',\n.*ref: 'refs\/tags\/.*'/
+    And I should not see any errors
+
+  Scenario: Auto resolve enabled and cron_core specified in Puppetfile
+    Given existing control repo "vendored"
+    When I run onceover command "run spec --auto_vendored=true" with --puppetfile Puppetfile.cron
+    Then I should see message pattern "cron_core found in Puppetfile. Using the specified version"
+    Then the temporary Puppetfile should contain /mod 'puppetlabs\/cron_core'/
+    And I should not see any errors
+

lib/onceover/cli/run.rb

@@ -44,6 +44,7 @@ def self.command
             optional nil, :format, 'Which RSpec formatter to use, valid options are: documentation, progress, FailureCollector, OnceoverFormatter. You also specify this multiple times', multiple: true, default: :defaults
             optional nil, :no_workarounds, 'Disables workarounds that have been added for convenience to get around common RSPec issues such as https://github.com/rodjek/rspec-puppet/issues/665'
             optional :ff, :fail_fast, 'Abort the run after the first failure'
+            optional nil, :auto_vendored, 'Attempt to resolve vendored puppet modules. Ex: puppetlabs/cron_core', default: false
 
             run do |opts, args, cmd|
               repo = Onceover::Controlrepo.new(opts)

lib/onceover/deploy.rb

@@ -13,6 +13,10 @@ def deploy_local(repo = Onceover::Controlrepo.new, opts = {})
       skip_r10k_default = !(File.file?(repo.puppetfile))
       skip_r10k = opts[:skip_r10k] || skip_r10k_default
       force     = opts[:force] || false
+      # Only attempt to resolve vendored modules if configured to do so
+      auto_vendored = opts[:auto_vendored] || false
+
+      require 'onceover/vendored_modules' if auto_vendored
 
       if repo.tempdir == nil
         repo.tempdir = Dir.mktmpdir('r10k')
@@ -90,7 +94,26 @@ def deploy_local(repo = Onceover::Controlrepo.new, opts = {})
         if /:control_branch/.match(puppetfile_contents)
           logger.debug "replacing :control_branch mentions in the Puppetfile with #{git_branch}"
           new_puppetfile_contents = puppetfile_contents.gsub(":control_branch", "'#{git_branch}'")
-          File.write("#{temp_controlrepo}/Puppetfile", new_puppetfile_contents)  
+          File.write("#{temp_controlrepo}/Puppetfile", new_puppetfile_contents)
+        end
+
+        if auto_vendored
+          tmp_puppetfile = File.join(temp_controlrepo, 'Puppetfile')
+          tmp_puppetfile_contents = File.read(tmp_puppetfile)
+          vm = Onceover::VendoredModules.new({repo: repo})
+          puppetfile = R10K::ModuleLoader::Puppetfile.new(basedir: temp_controlrepo)
+          vm.puppetfile_missing_vendored(puppetfile)
+          unless vm.missing_vendored.empty?
+            missing_slugs = vm.missing_vendored.map do |missing_mod|
+              missing_mod.keys[0]
+            end
+            logger.debug "Adding #{missing_slugs} to #{tmp_puppetfile}"
+            modlines = vm.missing_vendored.map do |missing_mod|
+              mod_slug = missing_mod.keys[0]
+              "mod '#{mod_slug}',\n  git: '#{missing_mod[mod_slug][:git]}',\n  ref: '#{missing_mod[mod_slug][:ref]}'"
+            end.join("\n")
+            File.write(tmp_puppetfile, tmp_puppetfile_contents + "\n# Onceover Managed Vendored Modules\n" + modlines)
+          end
         end
       end
 

lib/onceover/rake_tasks.rb

@@ -88,3 +88,12 @@
   end
 
 end
+
+desc 'Create cache for vendored modules'
+task :generate_vendor_cache do
+  require 'onceover/controlrepo'
+  require 'onceover/vendored_modules'
+
+  repo = Onceover::Controlrepo.new(debug: true)
+  Onceover::VendoredModules.new({repo: repo, cachedir: File.join(repo.spec_dir, 'vendored_modules'), force_update: true})
+end

lib/onceover/vendored_modules.rb

@@ -0,0 +1,191 @@
+require 'puppet/version'
+require 'net/http'
+require 'uri'
+require 'multi_json'
+require 'r10k/module_loader/puppetfile'
+require 'onceover/logger'
+
+### operations
+#
+# 1. resolve all the component json files in the puppet-agent repo for vendored modules
+# 2. parse each json file and determine vendored modules repo + ref
+#
+###
+
+## Example
+#
+# vm = Onceover::VendoredModules.new
+# puts vm.vendored_references
+# puppetfile = R10K::ModuleLoader::Puppetfile.new(basedir: '.')
+# vm.puppetfile_missing_vendored(puppetfile)
+# puts vm.missing_vendored.inspect
+
+class Onceover
+  class VendoredModules
+
+    attr_reader :vendored_references, :missing_vendored
+
+    def initialize(opts = {})
+      @repo = opts[:repo] || Onceover::Controlrepo.new
+      @cachedir = opts[:cachedir] || File.join(@repo.tempdir, 'vendored_modules')
+      @puppet_version = Gem::Version.new(Puppet.version)
+      @puppet_major_version = Gem::Version.new(@puppet_version.segments[0])
+      @force_update = opts[:force_update] || false
+
+      @missing_vendored = []
+
+      # This only applies to puppet >= 6 so bail early
+      raise 'Auto resolving vendored modules only applies to puppet versions >= 6' unless @puppet_major_version >= Gem::Version.new('6')
+
+      # Create cachedir
+      unless File.directory?(@cachedir)
+        logger.debug "Creating #{@cachedir}"
+        FileUtils.mkdir_p(@cachedir)
+      end
+
+      # Location of user provided caches:
+      #   control-repo/spec/vendored_modules/<component>-puppet_agent-<agent version>.json
+      @manual_vendored_dir = File.join(@repo.spec_dir, 'vendored_modules')
+
+      # Get the entire file tree of the puppetlabs/puppet-agent repository
+      # https://docs.github.com/en/rest/git/trees?apiVersion=2022-11-28#get-a-tree
+      puppet_agent_tree = query_or_cache(
+        "https://api.github.com/repos/puppetlabs/puppet-agent/git/trees/#{@puppet_version}",
+        { :recursive => true },
+        component_cache('repo_tree')
+      )
+      # Get only the module-puppetlabs-<something>_core.json component files
+      vendored_components =  puppet_agent_tree['tree'].select { |file| /configs\/components\/module-puppetlabs-\w+\.json/.match(file['path']) }
+      # Get the contents of each component file
+      # https://docs.github.com/en/rest/git/blobs?apiVersion=2022-11-28#get-a-blob
+      @vendored_references = vendored_components.map do |component|
+        mod_slug = component['path'].match(/.*(puppetlabs-\w+).json$/)[1]
+        mod_name = mod_slug.match(/puppetlabs-(\w+)/)[1]
+        query_or_cache(
+          component['url'],
+          nil,
+          component_cache(mod_name)
+        )
+      end
+    end
+
+    def component_cache(component)
+      # Ideally we want a cache for the version of the puppet agent used in tests
+      desired_name = "#{component}-puppet_agent-#{@puppet_version}.json"
+      # By default look for any caches created during previous runs
+      cache_file = File.join(@cachedir, desired_name)
+
+      unless @force_update
+        # If the user provides their own cache
+        if File.directory?(@manual_vendored_dir)
+          # Check for any '<component>-puppet_agent-<puppet version>.json' files
+          dg = Dir.glob(File.join(@manual_vendored_dir, "#{component}-puppet_agent*"))
+          # Check if there are multiple versions of the component cache
+          if dg.size > 1
+            # If there is the same version supplied as whats being tested against use that
+            if dg.any? { |s| s[desired_name] }
+              cache_file = File.join(@manual_vendored_dir, desired_name)
+            # If there are any with the same major version, use the latest supplied
+            elsif dg.any? { |s| s["#{component}-puppet_agent-#{@puppet_major_version}"] }
+              maj_match = dg.select { |f| /#{component}-puppet_agent-#{@puppet_major_version}.\d+\.\d+\.json/.match(f) }
+              maj_match.each do |f|
+                if (version_from_file(cache_file) == version_from_file(desired_name)) || (version_from_file(f) >= version_from_file(cache_file))
+                  # if the current cache version matches the desired version, use the first matching major version in user cache
+                  # if there are multiple major version matches in user cache, use the latest
+                  cache_file = f
+                end
+              end
+            # Otherwise just use the latest supplied
+            else
+              dg.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) }
+            end
+          # If there is only one use that
+          elsif dg.size == 1
+            cache_file = dg[0]
+          end
+        end
+      end
+
+      # Warn the user if cached version does not match whats being used to test
+      cache_version = version_from_file(cache_file)
+      if cache_version != @puppet_version
+        logger.warn "Cache for #{component} is for puppet_agent #{cache_version}, while you are testing against puppet_agent #{@puppet_version}. Consider updating your cache to ensure consistent behavior in your tests"
+      end
+
+      cache_file
+    end
+
+    def version_from_file(cache_file)
+      version_regex = /.*-puppet_agent-(\d+\.\d+\.\d+)\.json/
+      Gem::Version.new(version_regex.match(cache_file)[1])
+    end
+
+    # Currently expects to be passed a R10K::Puppetfile object.
+    # ex: R10K::ModuleLoader::Puppetfile.new(basedir: '.')
+    def puppetfile_missing_vendored(puppetfile)
+      puppetfile.load
+      @vendored_references.each do |mod|
+        # Extract name and slug from url
+        mod_slug = mod['url'].match(/.*(puppetlabs-\w+)\.git/)[1]
+        mod_name = mod_slug.match(/^puppetlabs-(\w+)$/)[1]
+        # Array of modules whos names match
+        existing = puppetfile.modules.select { |e_mod| e_mod.name == mod_name }
+        if existing.empty?
+          # Change url to https instead of ssh to allow anonymous git clones
+          # so that users do not need to have an ssh keypair associated with a Github account
+          url = mod['url'].gsub('[email protected]:', 'https://github.com/')
+          @missing_vendored << {mod_slug => {git: url, ref: mod['ref']}}
+          logger.debug "#{mod_name} found to be missing in Puppetfile"
+        else
+          logger.debug "#{mod_name} found in Puppetfile. Using the specified version"
+        end
+      end
+    end
+
+    # Return json from a query whom caches, or from the cache to avoid spamming github
+    def query_or_cache(url, params, filepath)
+      if (File.exist? filepath) && (@force_update == false)
+        logger.debug "Using cache: #{filepath}"
+        json = read_json_dump(filepath)
+      else
+        logger.debug "Making GET request to: #{url}"
+        json = github_get(url, params)
+        logger.debug "Caching response to: #{filepath}"
+        write_json_dump(filepath, json)
+      end
+      json
+    end
+
+    # Given a github url and optional query parameters, return the parsed json body
+    def github_get(url, params)
+      uri = URI.parse(url)
+      uri.query = URI.encode_www_form(params) if params
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = (uri.scheme == 'https')
+      request = Net::HTTP::Get.new(uri.request_uri)
+      request['Accept'] = 'application/vnd.github.raw+json'
+      request['X-GitHub-Api-Version'] = '2022-11-28'
+      response = http.request(request)
+
+      case response
+      when Net::HTTPOK # 200
+        MultiJson.load(response.body)
+      else
+        # Expose the ratelimit response headers
+        # https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#checking-the-status-of-your-rate-limit
+        ratelimit_headers = response.to_hash.select { |k, v| k =~ /x-ratelimit.*/ }
+        raise "#{response.code} #{response.message} #{ratelimit_headers}"
+      end
+    end
+
+    # Returns parsed json of file
+    def read_json_dump(filepath)
+      MultiJson.load(File.read(filepath))
+    end
+
+    # Writes json to a file
+    def write_json_dump(filepath, json_data)
+      File.write(filepath, MultiJson.dump(json_data))
+    end
+  end
+end

spec/fixtures/controlrepos/vendored/Puppetfile

@@ -0,0 +1,3 @@
+forge "https://forge.puppet.com"
+
+# Just an empty Puppetfile with no vendored modules listed

spec/fixtures/controlrepos/vendored/Puppetfile.cron

@@ -0,0 +1,5 @@
+forge "https://forge.puppet.com"
+
+# Puppetfile where cron_core is already being managed
+
+mod 'puppetlabs/cron_core'

spec/fixtures/controlrepos/vendored/environment.conf

@@ -0,0 +1 @@
+modulepath     = site-modules:modules:$basemodulepath

spec/fixtures/controlrepos/vendored/site-modules/role/manifests/cron.pp

@@ -0,0 +1,9 @@
+class role::cron {
+  # Resources shouldn't be in roles... but demonstrating
+  cron { 'logrotate':
+    command => '/usr/sbin/logrotate',
+    user    => 'root',
+    hour    => 2,
+    minute  => 0,
+  }
+}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/augeas_core-puppet_agent-7.30.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-augeas_core.git","ref":"refs/tags/v1.5.0"}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/augeas_core-puppet_agent-8.6.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-augeas_core.git","ref":"refs/tags/v1.5.0"}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/cron_core-puppet_agent-7.30.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-cron_core.git","ref":"refs/tags/v1.3.0"}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/cron_core-puppet_agent-8.6.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-cron_core.git","ref":"refs/tags/v1.3.0"}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/host_core-puppet_agent-7.30.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-host_core.git","ref":"refs/tags/v1.3.0"}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/host_core-puppet_agent-8.6.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-host_core.git","ref":"refs/tags/v1.3.0"}

spec/fixtures/controlrepos/vendored/spec/vendored_modules/mount_core-puppet_agent-7.30.0.json

@@ -0,0 +1 @@
+{"url":"[email protected]:puppetlabs/puppetlabs-mount_core.git","ref":"refs/tags/v1.3.0"}