Filling in the gaps in your Chef knowledge base

First things first. Let's have a quick, high-level review of Chef. A Chef 101 class, if you will. The plan here is to make sure that all readers are on the same page with me so that the rest of the book will be easier to follow and benefit from. If you are already a Chef ninja, you can probably skip right on past this section and start exploring the ChefDK in the next section. Otherwise, let's audit our 101 class now.

Major pieces of a Chef puzzle

Chef is a ruby framework, and its main purpose is to facilitate the automation, reuse, and documentation of server configurations. Chef allows you to treat server configuration just like any other type of software code.

There are three major components of this framework for almost all corporate-level Chef deployments—the workstation, the Chef Server, and the nodes. Let's talk a little about each of these components:

• Workstations: The Chef workstation is, as the name would suggest, where the real "work" is done. This is where developers will create their infrastructure code. It's also where they will test their creations. The workstation is where developers will interact with their source code control systems. It is where genius is born. It is where the "desired state" of nodes is defined via code.
• Chef servers: The Chef server is a database-backed web API server with a browsable user interface. It is like the matrix housing the hive mind of the Chef universe. The "work" created in the workstation is uploaded into the Chef server so that it can be used to automate the configuration of the many nodes. It stands ready to provide the desired state information to the Chef client on a node so that it can bring that node into alignment or convergence with the desired state of configuration.
• Nodes: Nodes are the ultimate targets of the "work" that's been created on the workstation and uploaded to the Chef server. A node is where the automation occurs via the Chef client. Nodes are transformed into the desired state configuration. Commonly, nodes are servers, either virtual or physical, but a node can be anything that needs to be configured, such as a Docker container, or a network device such as a switch or router. Every node has a unique name, such as the FQDN of a server.

The Chef client

The Chef client is the tool that is deployed to all nodes and used to configure or reconfigure the node to its desired state. The Chef client is what executes the automation. It is the magician that, through the magic of Chef, transforms an ordinary server into the desired state node that the developer defined back on his workstation.

The Ohai Chef tool

Ohai is the tool that gathers information about a node. Information such as platform details, operating system data, and processor information is made available to the Chef client so that the latter can have the know-how to bring the node to the desired state. Ohai is executed at the beginning of a Chef client run to gather the state of the node. At the end of a Chef client run, all of the data gathered by Ohai (usually) is shared as node data with the Chef server, and the shared data is available for searches done against the Chef server. Ohai can be extended via plugins, and we'll take a look at that in a later chapter.

Recipes and cookbooks

Recipes are the building blocks used to define the desired states. Recipes are files of ruby code that define the commands to be run on nodes. They are like blueprints used to "build" a node. Recipes are collections of Chef resources. We will explain more about Chef resources shortly.

Cookbooks are collections of related recipes, templates, files, and custom resources. They provide organization and versioning for recipes. Each unique version of a cookbook represents unique sets of functionality, such as bug fixes or added features.

Cookbooks define a scenario, such as everything needed to install and configure apt-docker or Sublime text, and they contain all the elements needed to support the defined scenario.

Recipes and cookbooks provide modularity and let you easily reuse code.

Chef run lists

A "run list" is, as the name would suggest, a list of, and the sequence for, the recipes, cookbooks, and policies (spoiler alert!) to be applied to a node. A run list contains all of the information required to configure a node to a desired state. That is, a Chef run list describes the desired final state of the node. It is important to note that, if the node's state already matches what the resources in the run list describe, then no action will be taken to change the node's state.

Chef roles

Roles are functional groupings of recipes and cookbooks used to describe the full blueprint needed for a node to become everything it is intended to be. Roles are reusable configurations, and they can be applied to multiple nodes to make functionally identical servers, such as a farm of web servers.

Chef resources

Chef resources are statements of configuration policy. They are defined in recipes and take actions through the Chef client to put the node into the desired state. Chef resources have some types: Package, Template, Service, and so on. They have a name and parameters. Also, Chef resources can send notifications to other resources.

Chef resources define what we want to happen to a node. They don't say how to do it. The how to do it is left to the providers which are platform-specific. That is to say, the way you install a package will be different depending on the OS, and the provider determines the correct way to do it—the "how". The Chef resource simply defines the "what," such as "install ntp".

• Package: This contains software or applications, such as apache, ntp, and cron, and the action to be performed on that software or application, such as "install".
• Template: These are files with place-holders for attributes that are transformed into configuration files for package installation and execution.
• Service: This is the installed executable of the package and the actions that the executable can perform, such as start, stop, or restart. Service also defines whether the software or application is launched at node startup.

More things to consider

Apart from the three types of Chef resources we saw earlier, let's look at some other important aspects of recipes and cookbooks.

• Attributes: These are variables used in recipes and templates. Generally speaking, a recipe will represent the pattern used in configuration. The attributes provide the specific details for the recipes. For example, the recipe will say "configure the port" and the attribute will say "which port?". Attributes can be provided in a large variety of places, such as cookbooks, roles, and environments. As such, there is a necessity for an order of precedence. There is a complete description of attribute precedence on the chef.io site. You can find it at https://docs.chef.io/attributes.html.
• Order matters: When creating recipes, the resources need to be listed in order. First, the package, then the template, and finally the service; when creating run lists, the order of policies, roles, cookbooks, and recipes is the order in which they are evaluated. Any recipe that appears in a run list (or as a dependency of a policy, role, or cookbook) more than once will only be evaluated the first time.
• Convergence and idempotence: A Chef client run converges the node into the desired state. What this means is that only things that do not equal the desired state on the node are modified during the Chef client run. If, for example, the recipe says that the desired state of the node is to have the ntp demon running, the ntp package is already installed on the node, and the ntp demon is already running, then Chef client will take no action to install or start ntp.

Chef resources are idempotent. That is, applying them more than once results in the same outcome every time. If no inputs related to the resource have changed, then applying that resource won't change anything after the first application. In fact, if none of the inputs have changed, the corresponding commands don't even get run.

This is actually one of the most important concepts and features of Chef.

Data bags

Data bags are containers for information that is not tied to a specific node. Data bags are the global variables of a Chef server. They can be used in recipes and can be searched like node data via the Chef server. One common use case for data bags is for user and group information. Items in a data bag can be encrypted. This allows secret information to be stored in them, for example, passwords.

Environments

Environments allow you to define specific cookbook versions that are applied to a given set of nodes. They permit you to model the stages of your infrastructure workflow, that is, Development, Test, Stage, and Production. By identifying specific nodes as Development, and other nodes as Production, for example, you can apply different versions of your cookbooks to the nodes based on their environment membership.

The Chef supermarket

The supermarket is a site that provides shared cookbooks. There is a public supermarket that contains community-created and -maintained cookbooks. This site is hosted by Chef and is available at https://supermarket.chef.io. In addition to the public supermarket, anyone can create and manage a private supermarket that can host cookbooks intended for the private consumption of you and your organization.

Tip

It is always best practice to do a thorough code review of any cookbooks obtained from the public community supermarket before using them in your production environments.

Chef Development Kit

The ChefDK contains everything you need to start working with Chef on a workstation. It provides all the tools that a developer needs to create and modify cookbooks and upload them to a Chef server. We are going to go into a lot of detail regarding the ChefDK later in this chapter.

You can jump ahead to the ChefDK sections now, or you can read on and learn how to set up your very own private Chef server.

Setting up an on-premise (private) Chef Server

Many companies will have strong reasons to keep their infrastructure configuration data within the firewalls of their datacenters, and Chef server has a mode for that. It's called on-premise or private Chef server.

There are some real benefits to using an in-house solution, including control. With a private installation, you have full control of your system, and for some that reason is enough. But there are additional considerations such as performance. Since a private Chef server will likely be physically closer to the node's network, Chef client runs will be faster. For example, if Chef is used in a load-based on-demand server deployment solution, then the extra speed the private Chef server can provide when converging your new Nodes can make all the difference in meeting the load demand in time.

With all the reasons to use an on-premise Chef server, there is a downside: you are responsible for the deployment, configuration, and maintenance of your Chef servers.

To get you started, let's take a look at the deployment and initial configuration of a standalone private Chef server now. We'll go through the steps to get a new Chef server set up on an Ubuntu 14.04 server.

The first step is downloading the required Chef server installer version. Visit the Chef server page at this link https://downloads.chef.io/chef-server/ to find and download the version you will use for your installation.

Next, you will want to transfer the downloaded installer package to your soon to be Chef server.

Once the installer is available on your server, you will want to install the package. The command to do this is:

sudo dpkg -i chef-server-core_12.2.0-1_amd64.deb

Once the Chef server package has been installed onto your server, the first thing to do is start the Chef server services. The command to do this is:

sudo chef-server-ctl reconfigure

The Chef server startup does a lot of work and as such you will see a lot of activity displayed as the setup progresses. When the startup command finally finishes successfully (a few minutes later; you might want to get a cup of coffee), the result will be a running Chef server.

Tip

Do you get an error in the nginx.rb file during the reconfigure?

The private Chef server has a dependency on the nginx cookbook, and in some cases you may experience an error when this dependency is resolved. The error may look something like this:

"common_name nil currently does not overwrite the value of common_name."

What this error message is probably indicating is that there is no valid host entry in the /etc/hosts file. You should check that file, and if there isn't a valid entry, create one.

Are we there yet? Well, we do have a Chef server up-and-running, but it is not much good to us yet. If you browse to your new Chef server, you will see something like this:

That is a good start, but there is still work to be done. Next up is creating your first Chef user on the Chef server with the user-create command. That user will be the initial Chef admin for the server. The format of the user-create command is:

sudo chef-server-ctl user-create user_name first_name last_name email password --filename FILE_NAME

Here is an example of what that command might look like:

sudo chef-server-ctl user-create earlwaud Earl Waud [email protected] MyPassword99 --filename /home/earlwaud/earlwaud.pem

Here is what it looks like when you issue the command on your Chef server (along with the user-show command):

The filename parameter used in the user-create command provides the location at which to store the user's private key. The contents of this key file should be securely stored so that they can be provided to the user whose account has been created.

The next step is to create the first organization in your Chef server. The format of that command is as follows:

sudo chef-server-ctl org-create short_name "full_organization_name" --association_user user_name --filename ORGANIZATION-validator.pem

Here is the command I used to create the organization on my example Chef server:

sudo chef-server-ctl org-create sdearl "San Diego Earl" --association_user earlwaud --filename /home/earlwaud/sdearl-validator.pem

You can see that, in the example, the key file was saved at /home/earlwaud/sdearl-validator.pem.

If the filename parameter is not provided in the org-create command, the validator key will be displayed to stdout.

Either way, it is vital that the key information be captured and securely stored since it is not saved in the Chef server database and cannot be recovered if lost. The only option available when an organization's private key is completely lost is to reset the validation key.

Next, we need to install a package that will allow management of the Chef server via a web interface. The package is named opscode-manage. The command to install the package is:

sudo chef-server-ctl install opscode-manage

Now we need to restart the Chef server services by issuing the reconfigure command again:

sudo chef-server-ctl reconfigure

And finally, we need to start the opscode-manage services. This is done by issuing the command:

sudo opscode-manage-ctl reconfigure

If everything went as planned, you should now be able to browse to the new Chef server, and log in with your newly created admin user. Of course, there is still more work to do, such as integrating your new Chef server with your corporate domain to allow domain user access control. Still, this is a good start, right?

References

• Explore the Chef community Supermarket at this link: https://supermarket.chef.io
• Download Chef server from https://downloads.chef.io/chef-server/
• Learn more about installing Chef server at https://docs.chef.io/release/server_12-2/install_server.html
• Learn more about attributes and attribute precedence at https://docs.chef.io/attributes.html
• Read more about the "common name" error at https://github.com/chef-cookbooks/chef-server/issues/108