5. Optionality and directives¶
In this chapter we’re going to go over some of the more flexible constructs which BuildStream offers for optionality, and show how we can use directives in the BuildStream YAML format.
Note
This example is distributed with BuildStream in the doc/examples/directives subdirectory.
5.1. Overview¶
This chapter’s example will build another hello.c program which much
resembles the program in the running commands example,
but here we’re going to make the greeting string configurable using the C
preprocessor.
We’ll be compiling the following C file:
5.1.1. files/src/hello.c¶
/*
* hello.c - Simple hello world program
*/
#include <stdio.h>
int main(int argc, char *argv[])
{
printf(GREETING_MESSAGE);
return 0;
}
And we’re going to build it using make, using the following Makefile:
5.1.2. files/src/Makefile¶
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -DGREETING_MESSAGE="\"${GREETING}\"" -Wall -o $@ $<
Notice the addition of -DGREETING_MESSAGE="\"${GREETING}\"" in the above
Makefile, this will allow us to configure the greeting message from the
hello.bst element declaration.
We will need to add support to our project for optionality, and we’ll have to make conditional statements to resolve what kind of greeting we want from the hello world program.
5.2. Project structure¶
Since this project has much the same structure as the
running commands chapter did, we won’t go over all of
these elements in detail. Instead let’s focus on the addition of the new
project options in project.conf, the added
file in the include/ project subdirectory, and how these come together
in the the hello.bst element.
5.2.1. project.conf¶
# Unique project name
name: directives
# Minimum required BuildStream version
min-version: 2.0
# Subdirectory where elements are stored
element-path: elements
# Define an alias for our alpine tarball
aliases:
alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
# Define an option for this project
#
options:
flavor:
type: enum
description: Flavor of the greeting in the hello world program
values:
- normal
- somber
- excited
default: normal
Here, our project.conf declares a project option called flavor, and this
will inform what kind of greeting message we want to use when building the project.
5.2.2. elements/hello.bst¶
kind: manual
description: |
A hello world program with a custom greeting message
# Depend on the base system
depends:
- base.bst
# Stage the files/src directory for building
sources:
- kind: local
path: files/src
# This include file defines the %{greeting} variable used below
variables:
(@): include/greeting.bst
# Now configure the commands to run
config:
# This time we inform the Makefile of which greeting we want
build-commands:
- make PREFIX="%{prefix}" GREETING="%{greeting}"
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Notice the (@) symbol we’ve added in the variables: section, this
symbol is used to invoke the include directive,
which can be useful for code sharing between elements or simply to improve readability.
In this case, we are compositing the content of include/greeting.bst into the
variables section of the element declaration, directives
can however be used virtually anywhere in the BuildStream YAML format.
5.2.3. include/greeting.bst¶
# We define the greeting message here conditionally
(?):
- flavor == "normal":
greeting: |
Hello world !
- flavor == "somber":
greeting: |
Hey world.
- flavor == "excited":
greeting: |
Howdy there, and what a world it is !
Here we can see the dictionary which will be composited into the variables:
section of the hello.bst element described above.
Note the usage of the (?) symbol at the toplevel of the YAML dictionary,
this is how we perform conditional statements
in the BuildStream YAML format.
This include file uses the flavor project option we declared in project.conf to
decide what value will end up being assigned to the %{greeting} variable, which
will ultimately be used in the hello.bst element.
5.3. Using the project¶
Now that we have a project which uses options and conditional statements, lets build the project with a few different options and observe the outputs.
5.3.1. Building hello.bst element with options¶
Since the flavor option we’ve declared above has a default, we can build it the first time using bst build without any special command line options:
user@host:~/directives$ bst build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:53:50
Project: directives (/home/user/directives)
Targets: hello.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: directives
Project Options
flavor: normal
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
buildable 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 base/alpine.bst
waiting c556a766f75b4d44070ae7eb39b344142736f5505b535299a6d4f5f1376de8eb base.bst
waiting cad7581043070b047af4834845fc584abd88ef51040aa6584186eeed10fea068 hello.bst
===============================================================================
[--:--:--][179c6ae9][ build:base/alpine.bst ] START directives/base-alpine/179c6ae9-build.1710.log
[--:--:--][179c6ae9][ build:base/alpine.bst ] START Staging sources
[00:00:00][179c6ae9][ build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][179c6ae9][ build:base/alpine.bst ] START Caching artifact
[00:00:00][179c6ae9][ build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:00][179c6ae9][ build:base/alpine.bst ] SUCCESS directives/base-alpine/179c6ae9-build.1710.log
[--:--:--][c556a766][ build:base.bst ] START directives/base/c556a766-build.1716.log
[--:--:--][c556a766][ build:base.bst ] START Caching artifact
[00:00:00][c556a766][ build:base.bst ] SUCCESS Caching artifact
[00:00:00][c556a766][ build:base.bst ] SUCCESS directives/base/c556a766-build.1716.log
[--:--:--][cad75810][ build:hello.bst ] START directives/hello/cad75810-build.1720.log
[--:--:--][cad75810][ build:hello.bst ] START Staging dependencies
[00:00:00][cad75810][ build:hello.bst ] SUCCESS Staging dependencies
[--:--:--][cad75810][ build:hello.bst ] START Staging sources
[00:00:00][cad75810][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][cad75810][ build:hello.bst ] START Running commands
make PREFIX="/usr" GREETING="Hello world !"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][cad75810][ build:hello.bst ] SUCCESS Running commands
[--:--:--][cad75810][ build:hello.bst ] START Caching artifact
[00:00:00][cad75810][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][cad75810][ build:hello.bst ] SUCCESS directives/hello/cad75810-build.1720.log
[00:00:00][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 3
Fetch Queue: processed 0, skipped 3, failed 0
Build Queue: processed 3, skipped 0, failed 0
If we want to build the somber flavor, we just need to specify the
additional --option command line option to bst
in order to inform BuildStream of the options we want.
user@host:~/directives$ bst --option flavor somber build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:53:51
Project: directives (/home/user/directives)
Targets: hello.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: directives
Project Options
flavor: somber
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
cached 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 base/alpine.bst
cached c556a766f75b4d44070ae7eb39b344142736f5505b535299a6d4f5f1376de8eb base.bst
buildable 028e738455de72766973477172442f4c6d89eaff445894e69a29ed7d850cfb15 hello.bst
===============================================================================
[--:--:--][028e7384][ build:hello.bst ] START directives/hello/028e7384-build.1767.log
[--:--:--][028e7384][ build:hello.bst ] START Staging dependencies
[00:00:00][028e7384][ build:hello.bst ] SUCCESS Staging dependencies
[--:--:--][028e7384][ build:hello.bst ] START Staging sources
[00:00:00][028e7384][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][028e7384][ build:hello.bst ] START Running commands
make PREFIX="/usr" GREETING="Hey world."
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][028e7384][ build:hello.bst ] SUCCESS Running commands
[--:--:--][028e7384][ build:hello.bst ] START Caching artifact
[00:00:00][028e7384][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][028e7384][ build:hello.bst ] SUCCESS directives/hello/028e7384-build.1767.log
[00:00:00][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 1
Fetch Queue: processed 0, skipped 1, failed 0
Build Queue: processed 1, skipped 0, failed 0
Note that the --option option can be specified many times on the
bst command line, so as to support projects which have multiple
options.
Finally lets get the excited flavor built as well:
user@host:~/directives$ bst --option flavor excited build hello.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:53:51
Project: directives (/home/user/directives)
Targets: hello.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: directives
Project Options
flavor: excited
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
cached 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 base/alpine.bst
cached c556a766f75b4d44070ae7eb39b344142736f5505b535299a6d4f5f1376de8eb base.bst
buildable 3d93bafadd3d0b00d16a558c46af4c7b3ca8d81ddcd0d0a184faee2c876ca12c hello.bst
===============================================================================
[--:--:--][3d93bafa][ build:hello.bst ] START directives/hello/3d93bafa-build.1816.log
[--:--:--][3d93bafa][ build:hello.bst ] START Staging dependencies
[00:00:00][3d93bafa][ build:hello.bst ] SUCCESS Staging dependencies
[--:--:--][3d93bafa][ build:hello.bst ] START Staging sources
[00:00:00][3d93bafa][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][3d93bafa][ build:hello.bst ] START Running commands
make PREFIX="/usr" GREETING="Howdy there, and what a world it is !"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][3d93bafa][ build:hello.bst ] SUCCESS Running commands
[--:--:--][3d93bafa][ build:hello.bst ] START Caching artifact
[00:00:00][3d93bafa][ build:hello.bst ] SUCCESS Caching artifact
[00:00:00][3d93bafa][ build:hello.bst ] SUCCESS directives/hello/3d93bafa-build.1816.log
[00:00:00][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 3
Session: 1
Fetch Queue: processed 0, skipped 1, failed 0
Build Queue: processed 1, skipped 0, failed 0
If you observe the cache keys above, you will notice that while
we have only three elements in the pipeline, counting base/alpine.bst,
base.bst and hello.bst, we have actually built five artifacts,
because the hello.bst is built differently each time, it has a
different cache key and is stored separately in the artifact cache.
5.3.2. Run the hello world program with options¶
Since the --option command line option to bst
is a main option, it can be used in any command.
Let’s run the hello program using bst shell
three times in a row, each time using a different option so we can
observe the results.
user@host:~/directives$ bst shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello.bst ] START Staging dependencies
[00:00:00][ ][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello.bst ] START Integrating sandbox
[00:00:00][ ][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello.bst ] STATUS Running command
hello
2020-08-19T14:53:52.795+0000 [1869:139836381059136] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:52.796+0000 [1869:139836381059136] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello world !
user@host:~/directives$ bst --option flavor somber shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello.bst ] START Staging dependencies
[00:00:00][ ][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello.bst ] START Integrating sandbox
[00:00:00][ ][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello.bst ] STATUS Running command
hello
2020-08-19T14:53:53.509+0000 [1905:139888999401536] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:53.509+0000 [1905:139888999401536] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hey world.
user@host:~/directives$ bst --option flavor excited shell hello.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello.bst ] START Staging dependencies
[00:00:00][ ][ main:hello.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello.bst ] START Integrating sandbox
[00:00:00][ ][ main:hello.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello.bst ] STATUS Running command
hello
2020-08-19T14:53:54.283+0000 [1941:140716832364608] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:54.284+0000 [1941:140716832364608] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Howdy there, and what a world it is !
5.4. Summary¶
In this chapter we’ve demonstrated how to declare project options, how to use conditional directives, and also how to use include directives.
To get more familliar with these concepts, you may want to explore the remaining directives in the BuildStream YAML format, and also take a look at the various types of project options that are also supported.