Follow me on Twitter Twitter_16
Subscribe to my RSS feed Rss_16
Posted over 6 years ago

Puppet Best Practices: Environment specific configs

I’m about a year into a migration to puppet and have learned quite a bit along the way. In this post I’m going to talk about what I’ve learned about managing environment specific configurations. I’ll briefly go through some of the different ways that we’ve managed these configurations along the way and then talk about where we are now. If you just want to know the end results feel free to skip to the end, but I think it’s valuable to see the problems with some of the other approaches we tried.

Case statements

In our first iteration we used selectors and case statements to set environment specific configs. Each environment had it’s own domain (i.e. web1.qa, web1.stg, web1.prod), and it seemed obvious to key off of that domain to set the configs for each environment.

modules/motd/manifests/init.pp
class motd {
   $motd = $environment ? {
     'test'       => 'This is the QA environment',
     'staging'    => 'This is the Staging environment'
     'production' => 'This is the production environment'
  }
  file { '/etc/motd' :
    content => $motd,
  }
}

This worked well functionally, but the issue manifested itself when we wanted to build a new environment, or change a particular config. When adding a new environment it was necessary to search through the entire code base for each switch statement and add a new case for the new environment. This first manifested itself when we began using Vagrant for testing our manifests. Vagrant VM’s by default are in the .local domain and thus did not match any of the domains we had coded for.

Lesson learned: Centralize the configs.

Environment specific directories

The idea here was to create environment specific templates within each module. I realize this a bit of a lousy example, but it gets the point across.

Directory Structure
`-- modules
    `-- motd
        |-- manifests
        |   `-- init.pp
        `-- files
            |-- production
            |   `-- motd
            |-- staging
            |   `-- motd
            `-- test
                `-- motd
init.pp
class motd {
  file { '/etc/motd' :
    source => ''puppet:///modules/openssl/$::environment/motd'
  }
}
modules/motd/files/production/motd
This is the production environment

This also worked for templates. Now we knew exactly where to go, when we wanted to make a config change. Each module had this structure. However, it was still pretty annoying to make a change or add a new environment. The process was copy the production directory to a directory with the environment name, then update the configs. The other problem was that when you needed to make a simple config change, it was necissary to push the entire code base. We realized that pushing the entire code base for a simple config change was too much overhead.

Lessons Learned: Centralize even more and move the configs out of the code base

ENC

Puppet comes with an ENC (Graphical UI for managing nodes). If you’re not using this check it out. There are many ENC’s available, some are command line, and some are GUI based. The important thing here is that the ENC supports setting parameters. The ENC enabled us to create groups, set parameters on those groups, assign nodes to a group, and then access those parameters as variable in the code base. The ENC we used is the puppet-dashboard
Using this strategy we could update configs through the ENC without pushing out new puppet code. This was a big win!

The code to lookup a param set in the ENC is…

lookupvar("::#{module_name}_#{var_name}")

However, the ENC does not provide an audit trail. No source control for configurations is a pretty big problem. We had no audit trail for what changes were being made, who made them, and why? We quickly needed a way to get the configurations back into source control. …but didn’t we say earlier that we didn’t want them in source control so that we could change configs without pushing new code.

Hiera

The solution is to move your configs to a separate repository and use hiera to do config lookups based on the environment. Here is an example of our setup.

On the puppetmaster:

/etc/puppet/hiera.yaml (simple configuration)
---
:backends: - yaml
:hierarchy: - %{environment}
            - common
:yaml:
  :datadir: /mnt/puppetmanifests/configuration
/etc/puppet/hiera.yaml (advanced configuration)
---
:backends: - yaml
:hierarchy: - %{environment}/%{fqdn}
            - %{environment}/%{calling_module}
            - %{environment}
            - common/%{calling_module}
            - common
:yaml:
  :datadir: /mnt/puppetmanifests/configuration

Hiera iterates over it hierarchy lists in order looking for a matching file and then for a matching param until the one you are requesting is found. For example, when requesting a parameter called motd from a module called motd and the server is web1.colo which has ‘production’ set as its environment in /etc/puppet/puppet.conf. Hiera will take the following actions:

  1. Look for file /mnt/pupppetmanifests/configuration/production/web1.colo.yaml
  2. If found look for parameter motd
  3. If parameter found return value
  4. If file or parameter not found look for file /mnt/pupppetmanifests/configuration/production/motd.yaml
  5. If found look for parameter motd
  6. If parameter found return value
  7. If file or parameter not found look for file /mnt/pupppetmanifests/configuration/production.yaml
  8. If found look for parameter motd
  9. If parameter found return value
  10. If file or parameter not found look for file /mnt/pupppetmanifests/configuration/common/motd.yaml
  11. If found look for parameter motd
  12. If parameter found return value
  13. If file or parameter not found look for file /mnt/pupppetmanifests/configuration/common.yaml
  14. If found look for parameter motd
  15. If parameter found return value
  16. If file or parameter not found fail resource
/mnt/puppetmanifests/configuration/production/production.yaml
---
motd: I'm a production server
/mnt/puppetmanifests/configuration/common.yaml
---
motd: I don't know what environment I'm in.

Motd module

class motd {
  file{'/etc/motd':
    ensure => present,
    content => function_hiera("motd",'')
  }
}

Hiera is included as part of puppet 3.0, but if you’re running puppet 2.7 you’ll need to install the following packages (for debian): hiera, hiera-puppet

Now you can push config changes without pushing new puppet code.

comments powered by Disqus