Staying DRY with value substitutions, extending, and macros

A buildout configuration is a collection of sections, each holding a collection of options. It’s common for option values to be repeated across options. For examples, many file-path options might start with common path prefixes. Configurations that include clients and servers might share server-address options. This topic presents various ways you can reuse option values without repeating yourself.

Value substitutions

When supplying values in a configuration, you can include values from other options using the syntax:

${SECTION:OPTION}

For example: ${buildout:directory} refers to the value of the directory option in the in the buildout section of the configuration. The value of the referenced option will be substituted for the referencing text.

You can simplify references to options in the current section by omitting the section name. If we wanted to use the buildout directory option from within the buildout section itself, we could use ${:directory}. This convenience is especially useful in macros, which we’ll discuss later in this topic.

There’s a special value that’s also useful in macros, named _buildout_section_name_, which has the name of the current section. We’ll show how this is used when we discuss macros.

Default and computed option values

Many sections have option values that can be used in substitutions without being defined in a configuration.

The buildout section, where settings for the buildout as a whole are provided has many default option values. For example, the directory where scripts are installed is configurable and the value is available as ${buildout:bin-directory}. See the Buildout options reference for a complete list of Buildout options that can be used in substitutions.

Many recipes also have options that have defaults or that are computed and are available for substitutions.

Sources of configuration options

Configuration option values can come from a number of sources (in increasing precedence):

software default values
These are defined by buildout and recipe sources.
user default values
These are set in per-user default configuration files and override default values.
options from one or more configuration files
These override user defaults and each other, as described below.
option assignments in the buildout command line
These override configuration-file options.

Extending configuration files

The extends option in a buildout section can be used to extend one or more configuration files. There are a number of applications for this. For example, common options for a set of projects might be kept in a common base configuration. A production buildout could extend a development buildout, or they could both extend a common base.

The option values in the extending configuration file override those in the files being extended. If multiple configurations are named in the extends option (separated by whitespace), then the configurations are processed in order from left/top to right/bottom, with the later (right/bottom) configurations overriding earlier (left/top) ones. For example, in:

extends = base1.cfg base2.cfg
          base3.cfg

The options in the configuration using the extends option override the options in base3.cfg, which override the options in base2.cfg, which override the options in base1.cfg.

Base configurations may be extended multiple times. For example, in the example above, base1.cfg might, itself, extend base3.cfg, or they might both extend a common base configuration. Of course, cycles are not allowed.

Configurations may be named with URLs in the extends option, in which case they may be downloaded from remote servers. See The extends-cache buildout option.

When a relative path is used in an extends option, it’s interpreted relative to the path of the extending configuration.

Conditional configuration sections

Sometimes, you need different configuration in different environments (different operating systems, or different versions of Python). To make this easier, you can define environment-specific options by providing conditional sections:

[ctl]
suffix =

[ctl:windows]
suffix = .bat

In this tiny example, we’ve defined a ctl:suffix option that’s .bat on Windows and an empty string elsewhere.

A conditional section has a colon and then a Python expression after the name. If the Python expression result is true, the section options from the section are included. If the value is false, the section is ignored.

Some things to note:

  • If there is no exception, then options from the section are included.
  • Sections and options can be repeated. If an option is repeated, the last value is used. In the example above, on Windows, the second suffix option overrides the first. If the order of the sections was reversed, the conditional section would have no effect.

In addition to the normal built-ins, the expression has access to global variables that make common cases short and descriptive as shown below

Name Value
sys sys module
os os module
platform platform module
re re module
python2 True if running Python 2
python3 True if running Python 3
python26 True if running Python 2.6
python27 True if running Python 2.7
python32 True if running Python 3.2
python33 True if running Python 3.3
python34 True if running Python 3.4
python35 True if running Python 3.5
python36 True if running Python 3.6
python37 True if running Python 3.7
python38 True if running Python 3.8
python39 True if running Python 3.9
python310 True if running Python 3.10
sys_version sys.version.lower()
pypy True if running PyPy
jython True if running Jython
iron True if running Iron Python
cpython True if not running PyPy, Jython, or Iron Python
sys_platform str(sys.platform).lower()
linux True if running on Linux
windows True if running on Windows
cygwin True if running on Cygwin
solaris True if running on Solaris
macosx True if running on Mac OS X
posix True if running on a POSIX-compatible system
bits32 True if running on a 32-bit system.
bits64 True if running on a 64-bit system.
little_endian True if running on a little-endian system
big_endian True if running on a big-endian system

Expressions must not contain either the # or the ; character.

User-default configuration

A per-user default configuration may be defined in the default.cfg file in the .buildout subdirectory of a user’s home directory (~/.buildout/default.cfg on Mac OS and Linux). This configuration is typically used to set up a shared egg or cache directory, as in:

[buildout]
eggs-directory = ~/.buildout/eggs
download-cache = ~/.buildout/download-cache
abi-tag-eggs = true

See the section on optimizing buildouts with shared eggs and download caches for an explanation of the options used in the example above.

Merging, rather than overriding values

Normally, values in extending configurations override values in extended configurations by replacing them, but it’s also possible to augment or trim overridden values. If += is used rather than =, the overriding option value is appended to the original. So, for example if we have a base configuration, buildout.cfg:

[buildout]
parts =
  py
  test
  server
...

And a production configuration prod.cfg, we can add another part, monitor, like this:

[buildout]
extends = buildout.cfg
parts += monitor
...

In this example, we didn’t have to repeat (or necessarily know) the base parts to add the monitor part.

We can also subtract values using -=, so if we wanted to exclude the test part in production:

[buildout]
extends = buildout.cfg
parts += monitor
parts -= test
...

Something to keep in mind is that this works by lines. The += form adds the lines in the new data to the lines of the old. Similarly, -= removes lines in the overriding option from the original lines. This is a bit delicate. In the example above, we were careful to put the base values on separate lines, in anticipation of using -=.

Merging values also works with option assignments provided via the buildout command line. For example, if you want to temporarily use a development version of another project, you can augment the buildout develop option on the command-line when running buildout:

buildout develop+=/path/to/other/project

Although, if you’ve pinned the version of that project, you’ll need to unpin it, which you can also do on the command-line:

buildout develop+=/path/to/other/project versions:projectname=

Extending sections using macros

We can extend other sections in a configuration as macros by naming then using the < option. For example, perhaps we have to create multiple server processes that listen on different ports. We might have a base server section, and some sections that use it as a macro:

[server]
recipe = zc.zdaemonrecipe
port = 8080
program =
  ${buildout:bin-directory}/serve
     --port ${:port}
     --name ${:_buildout_section_name_}

[server1]
<= server
port = 8081

[server2]
<= server
port = 8082

In the example above, the server1 and server2 sections use the server section, getting its recipe and program options. The resulting configuration is equivalent to:

[server]
recipe = zc.zdaemonrecipe
port = 8080
program =
  ${buildout:bin-directory}/serve
     --port ${:port}
     --name ${:_buildout_section_name_}

[server1]
recipe = zc.zdaemonrecipe
port = 8081
program =
  ${buildout:bin-directory}/serve
     --port ${:port}
     --name ${:_buildout_section_name_}

[server2]
recipe = zc.zdaemonrecipe
port = 8082
program =
  ${buildout:bin-directory}/serve
     --port ${:port}
     --name ${:_buildout_section_name_}

Value substitutions in the base section are applied after its application as a macro, so the substitutions are applied using data from the sections that used the macro (using the < option).

You can extend multiple sections by listing them in the < option on separate lines, as in:

[server2]
<= server
   monitored
port = 8082

If multiple sections are extended, they’re processed in order, with later ones taking precedence. In the example above, if both server and monitored provided an option, then the value from monitored would be used.

A section that’s used as a macro can extend another section.