Bcfg2 Manual

The Bcfg2 client performs all client configuration or reconfiguration operations. It renders a declarative configuration specification, provided by the Bcfg2 server, into a set of configuration operations which will, if executed, attempt to change the client's state into that described by the configuration specification. Conceptually, the Bcfg2 client serves to isolate the Bcfg2 server and specification from the imperative operations required to implement configuration changes. This isolation allows declarative specifications to be manipulated symbolically on the server, without needing to understand the properties of the underlying system tools. In this way, the Bcfg2 client acts as a sort of expert system that "knows" how to implement declarative configuration changes.

The operation of the Bcfg2 client is intended to be as simple as possible. The normal configuration process consists of four main steps:

The Bcfg2 server is responsible for taking a network description and turning it into a series of configuration specifications for particular clients. It also manages probed data and tracks statistics for clients.

The Bcfg2 server takes information from two sources when generating client configuration specifications. The first is a pool of metadata that describes clients as members of an aspect-based classing system. That is, clients are defined in terms of aspects of their behavior. The other is a file system repository that contains mappings from metadata to literal configuration. These are combined to form the literal configuration specifications for clients.

Literal configuration specifications are served to clients by the Bcfg2 server. This is a differentiating factor for Bcfg2; all other major configuration management systems use a non-literal configuration specification. That is, the clients receive a symbolic configuration that they process to implement target states. We took the literal approach for a few reasons:

This section will discuss several aspects of the design of bcfg2, and the particular use cases that motivated them. Initially, this will consist of a discussion of the system metadata, and the intended usage model for package indices as well.

Bcfg2 uses an aspect-based classing mechanism to describe configuration patterns in its specifications. Each class describes particular aspects of client configurations. The Bcfg2 metadata mechanism has two types of information, client metadata and group metadata. Client metadata describes what top level group a client is associated with. Its configuration is derived from a combination of its host information and this group. Group definitions describe groups in terms of what bundles they include and other groups they include. Groups have a set of properties that describe how they can be used. (Starred values are defaults)

When writing bcfg2 specification, administrators primarily perform one of two operations: addition of new configuration entities or the modification of existing entries. If new entities need to be added, then they must be added to the abstract configuration. This is the inventory of configuration entities that should be installed on a client. Two plugins provide the basis for the abstract configuration, the bundler and base. The bundler builds descriptions of interrelated configuration entities. These are typically used for the representation of services, or other complex groups of entities. Base provides a laundry list of configuration entities that need to be installed on hosts. These entities are independent from one another, and can be installed individually without worrying about the impact on other entities.

Entities in the abstract configuration (and correspondingly in the literal configuration) can have one of several types. In the abstract configuration, each of these entities only has a tag and the name attribute set.

During the construction of the literal configuration, first the abstract configuration is built, and then explicit data is bound in to each entity. The previous section describes how the first stage works, and this section describes the second stage. Each entity will be served by one plugin. This plugin will decide what explicit data should be bound in to a particular entity for a given client. Each of these plugins has a specific area of the configuration repository, corresponding to its name. This section describes how several of the basic plugins works.

	  owner:root
	  group:root
	  perms:0755

	  $ ls
	  passwd.H_adenine     passwd            passwd.G99_chiba
	  passwd.H_bio-debian  passwd.H_cvstest  passwd.H_foxtrot  
	  passwd.H_reboot      passwd.H_rudy2    passwd.G98_netserv
	  passwd.G99_tacacs-server.cat           :info

<Services priority='0'>
  <Service name='ssh' status='on'/>
  <Group name='a'>
    <Service name='ssh' status='off'/>
    <Group name='b'>
      <Service name='ssh' status='on'/>
    </Group>
  </Group>
</Services>

Any configuration change will apply to some set of clients. Often, repository changes can have unintended consequences to clients not included in the target group. To address this issue, consider the changes performed, and if they can affect clients in unexpected ways.

Bcfg2 includes a repository validation tool that will check all XML files in the repository against included XML schemas. It is critical to run this command, bcfg2-repo-valdate after any modifications to XML files. If all files validate properly, then no output will be returned. It takes a "-v" option that prints out a line for each file that is validated. This can be used to ensure that all files are checked.

In addition to the description of the abstract process above, we present several examples of the thought process and actions taken to achieve a particular configuration goal. These will start simple, but become more complex.

The Bcfg2 server will build configurations based on a set of high-level specifications that use class-based abstractions to provide reusability. This approach works pretty well; however, it can be hard to deploy and may be too complicated to solve simple problems.

This issue can be addressed through the use of the Bcfg2 client with a static configuration specification. This method works as follows: important configuration details are statically specified in a file on each system. The Bcfg2 client runs periodically, and ensures that all aspects of configuration included in the static specification are correct. It then performs any update operations needed on the client.

The format of static specifications is identical to that provided by the Bcfg2 server, when it is used. It consists of a series of "Bundle" and "Independent" clauses. Independent clauses contain a series of configuration elements that can be installed without any install time dependence on other configuration elements. Bundles are series of dependent clauses. This means that configuration elements may interfere with one another, or that services may need to be restarted upon configuration update.

Each of these containers consists of a series of configuration elements. The same elements may appear in either type of clauses. These are basic types that are the same across all OS ports.

The next easiest method to deploy is one where the configuration specification is as simple and literal as possible. This style of configuration specification can be characterized as near copies of parts of the system.

This style of deployment uses the stock generators: Cfg, Pkgmgr, and Svcmgr. These manage configuration files, packages and services, respectively. Copies of configuration files are placed in the Cfg repository, in as generic a location as possible.

In my computing environment there are quite a diverse set of machines and requirements for their operation. What this meant was that I needed to devise a build system for machines that would allow me to easily customize the software and services on the machine while still being able to easily manage them and keep them secure. What I came up with that solved this problem was that the initial install needed to be the smallest subset of software that all machines had in common and install this with whatever automated install system fit the OS. The goal being that the OS automated installer( ie: kickstart, or systemimager ) would put the initial bits on disk and take care of hardware stuff and then as part of the postinstall process I run bcfg2 to insure that the rest of the software and configuration occurs based on the machines metadata. The overall goal was met. I could now build any type of machine that I needed just by using the common buildsystem and let bcfg2 determine what was different machine to machine.

My current build process is centered around systemimager and bcfg2. I have done some small enhancements to systemimager so that with one floppy or cdrom any administrator can build any number of machine profiles automatically. This is all done with some of the new features that allow the encoding of the profile and image in the clientside command so that the back end metadata can be asserted from the client, which overrides the defaults specified in the metadata.xml file.

Bcfg2 plugins are loadable python modules that the Bcfg2 server loads at initialization time. These plugins can contribute to the functions already offered by the Bcfg2 server or can extend its functionality. In general, plugins will provide some portion of the configuration for clients, with a data representation that is tuned for a set of common tasks. Much of the core functionality of Bcfg2 is implemented by several plugins, however, they are not special in any way; new plugins could easily supplant one or all of them.

Bcfg2 plugins are python classes that subclass from Bcfg2.Server.Plugin.Plugin. Several plugin-specific values must be set in the new plugin. These values dictate how the new plugin will behave with respect to the above four functions.

import socket, Bcfg2.Server.Plugin

class Chiba(Bcfg2.Server.Plugin.Plugin):
    '''the Chiba plugin builds the following files:
      -> /etc/network/interfaces'''

    __name__ = 'Chiba'
    __version__ = '$Id: chiba.py 1702 2006-01-19 20:20:51Z desai '
    __author__ = 'bcfg-dev@mcs.anl.gov'
    Entries = {'ConfigFile':{}}

    def __init__(self, core, datastore):
        Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore)
        self.repo = Bcfg2.Server.Plugin.DirectoryBacked(self.data, 
	  self.core.fam)
        self.Entries['ConfigFile']['/etc/network/interfaces'] \ 
	  = self.build_interfaces

    def build_interfaces(self, entry, metadata):
        '''build network configs for clients'''
        entry.attrib['owner'] = 'root'
        entry.attrib['group'] = 'root'
        entry.attrib['perms'] = '0644'
        try:
            myriaddr = socket.gethostbyname("%s-myr" % \
	  metadata.hostname)
        except socket.gaierror:
            self.LogError("Failed to resolve %s-myr"% metadata.hostname)
            raise Bcfg2.Server.Plugin.PluginExecutionError, ("%s-myr" \
            % metadata.hostname, 'lookup')
        entry.text = self.repo.entries['interfaces-template'].data % \
	  myriaddr
      

The BCFG2 Reporting System consists of a number of elements including the bcfg2-build-reports Executable, a configuration file, and XSLT transform files. bcfg2-build-reports reads a default configuration file (or a config file specified on the command line) then prepares and delivers the reports according to the format defined in the transform files. It is expected that this executable will be run by the adminstrator periodically via cron or similar facility. The executable can also be run manually on demand for a special sort of report that needs to be generated immediately.

bcfg2-build-reports gets the data it reports from a number of sources. Metadata/clients.xml contains information about if a host is currently pingable or not. bcfg2-ping-sweep will be run automatically by bcfg2-build-reports if the Metadata/clients.xml file is out of date with pingability information.

The next place bcfg2-build-reports gets data from is the statistics.xml file. This file is maintained by bcfgd, and is updated whenever a client updates, therefore is always up to date and no maintainance is required on this file. Most of the information in the predefined reports come from this file.

Finally bcfg2-build-reports is able to pull information from the Metadata/groups.xml file as well. This allows reports to describe the configured profile for each client.

There are a number of report types and delivery styles to present and transmit the reported data. The reporting structure lends itself best to structuring reports around groups of machines. For any group of machines any number of reports are generated. Each report may be delivered via Mail, WWW, or RSS (or any combination of the three.) In the future additional report types will be added, and if necessary, additional types of deliveries will be created. It is easy to create your own custom report using XSLT. Tables describing report types and report delivery mechanisms follow:

The report-configuration.xml file is the standard file that the bcfg2-build-reports executable uses when it is run without any command line arguments. Alternate configuration files, formatted identically, can be used by specifing -c flag. This can be useful for running different types of reports at different intervals. For example:

      Run this hourly: bcfg2-build-reports -c WebAndRssReport-config.xml
      Run this daily: bcfg2-build-reports -c emailReports-config.xml
    

The report-configuration.xml file is structured with a root <Reports/> tag at the top level. Within this tag any number of <Report/> tags can be inserted. Each report is structured around a group of machines. <Machine/> tags may individually reference a machine by hostname, or by a Python Regular Expression describing a group of hostnames. ".*" is especially helpful to describe all hosts. More information can be found about such Regexes at: http://docs.python.org/lib/re-syntax.html.

Any number of <Delivery/> elements can be defined for a given report. A delivery consists of a mechanism and type. The mechanism would be something like Mail or Web, and the type would describe the intended content of the report. Some are tailored to overall machine health, while others could be best fit for auditing purposes

Finally, each <Delivery/> element contains one or more <Destination/> elements. In the case of an RSS or WWW report, the destination should be a complete path to the output file including the file's name. In e-mail based reports the destination should be a valid e-mail address.

The following configuration will generate two separate reports and deliver them in a number of different ways. For more information on exactly what each section does, please refer to the Configuration section above.

<Reports>
      <Report name='core_stats' good='Y' modified='Y'>
          <Delivery mechanism='mail' type='nodes-digest'>
              <Destination address='user@domain.tld'/>
              <Destination address='user@otherdomain.tld'/>           
          </Delivery>
          <Delivery mechanism='www' type='nodes-digest'>
              <Destination address='/var/www/core_stats.html'/>
          </Delivery>
          <Machine name='.*'/>
      </Report>

      <Report name='stats_for_a_machines' good='N' modified='Y'>
          <Delivery mechanism='mail' type='nodes-digest'>
              <Destination address='user@domain.tld'/>
          </Delivery>
          <Delivery mechanism='mail' type='overview-stats'>
              <Destination address='user@otherdomain.tld'/>
          </Delivery>
          <Machine name='a.*'/>
          <Machine name='x-aim'/>
      </Report>
    </Reports>
  

Once configured correctly, just wait for the e-mail or view the outputed html files with a web browser. bcfg2-build-reports to recieve your reports immediately, or configure cron to run it perodically. E-mail reports will deliver the appropriate content directly to your mail client, and the html files should be viewable with any web browser. It is suggested those files be accessable via a webserver for convenience to other interested parties.

Examples of the performance and overview reports.