User configuration¶

User configuration and preferences can be specified in a user provided configuration file, and usually also on the command line.

Values specified in a user provided configuration file override the defaults, while command line options take precedence over any other specified configurations.

Configuration file¶

Users can provide a configuration file to override parameters in the default configuration.

Unless a configuration file is explicitly specified on the command line when invoking bst, an attempt is made to load user specific configuration from $XDG_CONFIG_HOME/buildstream.conf. On most Linux based systems, the location will be ~/.config/buildstream.conf

Note

If you have have multiple major versions of BuildStream installed, you can have separate configuration files in your ${XDG_CONFIG_HOME}.

You can do this by naming them according to the major versions of BuildStream you have installed. BuildStream 1 will load it’s configuration from $XDG_CONFIG_HOME/buildstream1.conf and BuildStream 2 will load it’s configuration from $XDG_CONFIG_HOME/buildstream2.conf, while any version will fallback to $XDG_CONFIG_HOME/buildstream.conf.

Project specific value¶

The projects key can be used to specify project specific configurations, the supported configurations on a project wide basis are listed here.

Artifact server¶

Although project’s often specify a remote artifact cache in their project.conf, you may also want to specify extra caches.

Assuming that your host/server is reachable on the internet as artifacts.com (for example), there are two ways to declare remote caches in your user configuration:

Adding global caches:

#
# Artifacts
#
artifacts:
  # Add a cache to pull from
  - url: https://artifacts.com/artifacts:11001
    server-cert: server.crt
  # Add a cache to push/pull to/from
  - url: https://artifacts.com/artifacts:11002
    server-cert: server.crt
    client-cert: client.crt
    client-key: client.key
    push: true
  # Add another cache to pull from
  - url: https://anothercache.com/artifacts:8080
    server-cert: another_server.crt

Note

Caches declared here will be used by all BuildStream project’s on the user’s machine and are considered a lower priority than those specified in the project configuration.

Specifying caches for a specific project within the user configuration:

projects:
  project-name:
    artifacts:
      # Add a cache to pull from
      - url: https://artifacts.com/artifacts:11001
        server-cert: server.crt
      # Add a cache to push/pull to/from
      - url: https://artifacts.com/artifacts:11002
        server-cert: server.crt
        client-cert: client.crt
        client-key: client.key
        push: true
      # Add another cache to pull from
      - url: https://ourprojectcache.com/artifacts:8080
        server-cert: project_server.crt

Note

Caches listed here will be considered a higher priority than those specified by the project. Furthermore, for a given list of URLs, earlier entries will have higher priority.

Notice that the use of different ports for the same server distinguishes between pull only access and push/pull access. For information regarding this and the server/client certificates and keys, please see: Key pair for the server.

Source cache server¶

Similarly global and project specific source caches servers can be specified in the user configuration.

Global source caches

#
# Source caches
#
source-caches:
  # Add a cache to pull from
  - url: https://cache.com/sources:11001
    server-cert: server.crt
  # Add a cache to push/pull to/from
  - url: https://cache.com/sources:11002
    server-cert: server.crt
    client-cert: client.crt
    client-key: client.key
    push: true
  # Add another cache to pull from
  - url: https://anothercache.com/sources:8080
    server-cert: another_server.crt

Project specific source caches

projects:
  project-name:
    artifacts:
      # Add a cache to pull from
      - url: https://cache.com/sources:11001
        server-cert: server.crt
      # Add a cache to push/pull to/from
      - url: https://cache.com/sources:11002
        server-cert: server.crt
        client-cert: client.crt
        client-key: client.key
        push: true
      # Add another cache to pull from
      - url: https://ourprojectcache.com/sources:8080
        server-cert: project_server.crt

Remote execution¶

The configuration for remote execution in project.conf can be provided in the user configuation. The global configuration also has a pull-artifact-files option, which specifies when remote execution is being performed whether to pull file blobs of artifacts, or just the directory trees required to perform remote builds.

There is only one remote execution configuration used per project.

The project overrides will be taken in priority. The global configuration will be used as fallback.

Global remote execution fallback:

remote-execution:
  execution-service:
    url: http://execution.fallback.example.com:50051
    instance-name: main
  storage-service:
    url: https://storage.fallback.example.com:11002
    server-cert: /keys/server.crt
    client-cert: /keys/client.crt
    client-key: /keys/client.key
    instance-name: main
  action-cache-service:
    url: http://cache.flalback.example.com:50052
    instance-name: main
  pull-artifact-files: True

Project override:

projects:
  some_project:
    remote-execution:
      execution-service:
        url: http://execution.some_project.example.com:50051
        instance-name: main
      storage-service:
        url: http://storage.some_project.example.com:11002
        instance-name: main
      action-cache-service:
        url: http://cache.some_project.example.com:50052
        instance-name: main

Strict build plan¶

The strict build plan option decides whether you want elements to rebuild when their dependencies have changed. This is enabled by default, but recommended to turn off in developer scenarios where you might want to build a large system and test it quickly after modifying some low level component.

Example

projects:
  project-name:
    strict: False

Note

It is always possible to override this at invocation time using the --strict and --no-strict command line options.

Default Mirror¶

When using mirrors, a default mirror can be defined to be fetched first. The default mirror is defined by its name, e.g.

projects:
  project-name:
    default-mirror: oz

Note

It is possible to override this at invocation time using the --default-mirror command-line option.

Local cache expiry¶

BuildStream locally caches artifacts, build trees, log files and sources within a cache located at ~/.cache/buildstream (unless a $XDG_CACHE_HOME environment variable exists). When building large projects, this cache can get very large, thus BuildStream will attempt to clean up the cache automatically by expiring the least recently used artifacts.

By default, cache expiry will begin once the file system which contains the cache approaches maximum usage. However, it is also possible to impose a quota on the local cache in the user configuration. This can be done in two ways:

By restricting the maximum size of the cache directory itself.

For example, to ensure that BuildStream’s cache does not grow beyond 100 GB, simply declare the following in your user configuration (~/.config/buildstream.conf):

cache:
  quota: 100G

This quota defines the maximum size of the artifact cache in bytes. Other accepted values are: K, M, G or T (or you can simply declare the value in bytes, without the suffix). This uses the same format as systemd’s resource-control.

By expiring artifacts once the file system which contains the cache exceeds a specified usage.

To ensure that we start cleaning the cache once we’ve used 80% of local disk space (on the file system which mounts the cache):

cache:
  quota: 80%

Default configuration¶

The default BuildStream configuration is specified here for reference:

# Default BuildStream user configuration.

#
#    Work Directories
#
#
# Note that BuildStream forces the XDG Base Directory names
# into the environment if they are not already set, and allows
# expansion of '~' and environment variables when specifying
# paths.
#

# Location to store sources
sourcedir: ${XDG_CACHE_HOME}/buildstream/sources

# Root location for other directories in the cache
cachedir: ${XDG_CACHE_HOME}/buildstream

# Location to store build logs
logdir: ${XDG_CACHE_HOME}/buildstream/logs

# Default root location for workspaces, blank for no default set.
workspacedir: .

#
#    Cache
#
cache:
  # Size of the artifact cache in bytes - BuildStream will attempt to keep the
  # artifact cache within this size.
  # If the value is suffixed with K, M, G or T, the specified memory size is
  # parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base
  # 1024), respectively.
  # Alternatively, a percentage value may be specified, which is taken relative
  # to the isize of the file system containing the cache.
  quota: infinity

  # Whether to pull build trees when downloading element artifacts
  pull-buildtrees: False

  # Whether to cache build trees on artifact creation:
  #
  #  always  - Always cache artifact build tree content
  #  auto    - Only cache build trees when necessary, e.g., for failed builds
  #  never   - Never cache artifact build tree content. This is not recommended
  #            for normal users as this breaks core functionality such as
  #            debugging failed builds and may break additional functionality
  #            in future versions.
  #
  cache-buildtrees: auto


#
#    Scheduler
#
scheduler:

  # Maximum number of simultaneous downloading tasks.
  fetchers: 10

  # Maximum number of simultaneous build tasks.
  builders: 4

  # Maximum number of simultaneous uploading tasks.
  pushers: 4

  # Maximum number of retries for network tasks.
  network-retries: 2

  # What to do when an element fails, if not running in
  # interactive mode:
  #
  #  continue  - Continue queueing jobs as much as possible
  #  quit      - Exit after all ongoing jobs complete
  #  terminate - Terminate any ongoing jobs and exit
  #
  on-error: quit


#
# Build related configuration
#
build:

  #
  # Maximum number of jobs to run per build task.
  #
  # The default behavior when this is set to 0, is to use the
  # maximum number of threads available, with a maximum of 8.
  #
  max-jobs: 0

  #
  # Control which dependencies to build:
  #
  #  plan - Only dependencies required for the build plan
  #  all  - All dependencies
  #
  dependencies: plan


#
#    Logging
#
logging:

  # The abbreviated cache key length to display in the UI
  key-length: 8

  # Whether to show extra detailed messages
  verbose: True

  # Maximum number of lines to print from the
  # end of a failing build log
  error-lines: 20

  # Maximum number of lines to print in a detailed
  # message on the console or in the master log (the full
  # messages are always recorded in the individual build
  # logs)
  message-lines: 20

  # Whether to enable debugging messages
  debug: False

  # Format string for printing the pipeline at startup, this
  # also determines the default display format for `bst show`
  #
  # See `bst show --help` for details on which symbols can be
  # used in the format string.
  element-format: |

    %{state: >12} %{full-key} %{name} %{workspace-dirs}

  # Format string for all log messages.
  #
  # Following symbols can be used in the format string:
  #
  # elapsed       - Time spent on this operation
  # elapsed-us    - Time spent on this operation (with microsecond precision)
  # wallclock     - Current wallclock time
  # wallclock-us  - Current wallclock time (with microsecond precision)
  # key           - The abbreviated cache key (if all sources are consistent)
  # element       - Name of the element
  # action        - Type of the message
  # message       - The main message text
  #
  message-format: |

    [%{elapsed}][%{key}][%{element}] %{action} %{message}

  # Limit bst console output update rate to 1Hz where applicable
  throttle-ui-updates: True