Posted over 8 years ago
How to organize puppet manifests (directory structure)
I really haven’t been able to find any documented methodology or best practices around how to organize puppet manifests. After several discussions on th’is topic with my colleagues we’ve identified a list of principles that should be considered when defining the directory structure. I’ll outline what these are and then propose an implementation that follows the principles.
The directory structure should attempt to follow conventions created by non-puppet coding standards
To support a devOps culture we must create a directory and code organization structure that both traditional developers and systems people can understand and use. This means that the code structure for puppet needs to follow patterns that developers are familiar with, yet use a vocabulary that relate to systems.
The directory structure should be as flat as possible
A flat directory structure presents information to a user faster. As a directory structure deepens, information becomes more difficult to find. To clarify what I mean here, imagine a developer wanting to determine if a manifest exists before writing one. Code duplication is bad right? So, we want to make it easy for a developer to quickly find a manifest. By burying manifests in deep directory structures, finding that manifest becomes a bit more difficult, but by using a flat directory structure the developer can quickly see if that manifest exists. So, why not put everything in one directory. I’ll get into that next.
Clearly define what goes where
There should be no ambiguity as to what manifest belongs in what directory. This could be easily be solved by putting all your manifests in one directory, but this leads to another problem… information overload. Sorting through a lot of manifests in one place could becomes just as difficult as digging through a deep directory structure to find information. The key here is to separate as much as possible, but clearly define those separations and make the naming of those separations as intuitive as possible.
We can change it
Write your manifests so that refactoring and moving manifests around is as easy as possible. Our needs will change, and we will have discoveries that may prompt us to revisit our directory structure. This doesn’t mean we got it wrong. Iterating is okay!
The MVC pattern is a common and well known development pattern for web applications. I plan to create a vocabulary around infrastructure that will allow us to create groupings of manifests. Then I will draw similarities between these groupings and their counterpart in the MVC framework. Hopefully this will culminate in a pattern that can be used for infrastructure code development. I’ll call this pattern the Service-Role-Capability (SRC) pattern for the purposes of this post.
Next, I’ll give a brief introduction to the MVC pattern, but more information can be found here.
The model is the most basic building block of the MVC pattern. This is an object or class that represents a real-world element. If you ware developing software for a soda machine, soda would be a model. It would contain methods that can be used to define that object (like name, price). It also would contain methods to interact with that object (like removeOne, that might be used to subtract one from the inventory of that soda when one is sold). A model should never interact with a controller directly, it is at the bottom of the MVC stack. However, it can interact with other models, and it may even contain other models (see ‘this article’:http://javapapers.com/oops/association-aggregation-composition-abstraction-generalization-realization-dependency/#&slider1=1). Generally a model is backed by a database which is used to maintain state. As a best practice, no other element in the MVC framework should interact with the database.
So, a model…
- represents a real world object.
- interacts with the database directly.
- contains business logic needed to manipulate that object.
The view is at the top of the MVC stack. Think of the view as the piece that puts all the pieces together in a way that is easy to interpret. The view in a web application is generally HTML and defines [only] the structure of the page. There is no business logic in the view. The view may call other elements of the MVC stack, but only to get data for rendering. It should not call anything that manipulates or changes data. The view should be completely static and easy to read.
So, a view…
- has no business logic.
- clearly defines the structure of then end product (a web page)
- can call into other elements of the framework, but only to retrieve information (not to manipulate it).
Think of the controller as the machine that orchestrates everything else. It is the glue that handles the interactions between models, manipulates data, and prepares everything for the view to render. It ties the business logic that is handled by the models together in a way that provides the view with data it needs to render.
So, a controller…
- interacts with multiple models in order to handle a specific request.
- provides data to the view.
Services are atomic items that correspond to the model in the MVC pattern. Services are the most basic element of the SRC element. A service will contain a very basic element (a product or package) and its configuration and dependencies. Those dependencies may be other services. An example of a service might be apache or snmp. If you think of service configuration as business logic and package repositories as a database the relationship (perhaps even package versions as state) the similarities between a service and a model in the MVC pattern become apparent. Also, in the MVC pattern it is a best practice to define your models as the most basic building blocks possible of that define the system under development. It is important to think of the service in this way as well.
So, a service…
- is a basic building block for your system.
- is the only element in the SRC pattern that interacts with external systems.
- defines the configuration and dependencies that pertain to the service.
A role defines the structure of an end-system. It should define the services and the capabilities needed to provision an end server. The structure of this manifest should be very clear and easy to read. The capabilities that are included in this manifest should clearly relate to the role otherwise they should be encapsulated in a capability. No logic (configuration, variables, etc.) should be in the role. A role can also include other roles.
So, a role…
- has no business logic.
- clearly defines the end system.
- uses other roles, capabilities, and services to define itself.
A capability is a grouping of services or other capabilities that puts basic elements together in a logical way to build complex functionality. Capabilites provide a means to present a grouping of manifests together in a way that makes it easy to consume by roles without needing to understand the details of what makes up that functionality.
So a capability…
- interacts with multiple services and capabilities.
- is consumed by roles to define system functionality.
Now, where to put custom types, facts, providers, etc. Generally web application products have a concept of utilities or libraries that exist outside the MVC framework. Developers will often organize these under a lib or util directory. The same concept works for puppet.
An example directory structure:
modules/ |-- capabilities | |-- php_webserver | | |-- manfiests | | | |-- init.pp | | |-- templates | | | |-- rails_webserver | | |-- manfiests | | | |-- init.pp | | |-- templates | | | |-- unit_tests | |-- manifests | |-- init.pp |-- lib | |-- roles | |-- build_server01 | | |-- manifests | | |-- init.pp | | | |-- web_server_for_product01 | | |-- manifests | | |-- init.pp | | | |-- web_server_for_product02 | |-- manifests | |-- init.pp | |-- services |-- apache | |-- manifests | |-- dependencies | | |-- init.pp | | |-- php.pp | | |-- rails.pp | | | |-- init.pp | |-- php | |-- manifests | |-- dependencies | | |-- init.pp | |-- init.pp | |-- php-xdebug | |-- manifests | |-- init.pp | |-- product01 | |-- manifests | |-- dependencies | | |-- common.pp | | |-- runtime.pp | | |-- test.pp | | | |-- init.pp | |-- product02 | |-- manifests | |-- dependencies | | |-- common.pp | | |-- runtime.pp | | |-- test.pp | | | |-- init.pp | |-- rails |-- manifests |-- dependencies | |-- init.pp | |-- init.pp
This post is intended as a building block and a means to prompt more discussion around this topic. I’d be very interested in others ideas and experience related to organizing puppet manifests. What has worked? What hasn’t?
I plan on having a series of posts related to this topic. Other posts I’m considering are:
- Convention for a service directory structure: creating notify, config, and dependency classes for each service.
- The advantages of defining a class for each package.
Leave a comment if you have any other puppet related topics that you would like to discuss.