2. Running commands

In the first chapter we only imported a file to create an artifact, this time lets run some commands inside the isolated build sandbox.

Note

This example is distributed with BuildStream in the doc/examples/running-commands subdirectory.

2.1. Overview

In this chapter, we’ll be running commands inside the sandboxed execution environment and producing build output.

We’ll be compiling the following simple C file:

2.1.1. files/src/hello.c

/*
 * hello.c - Simple hello world program
 */
#include <stdio.h>

int main(int argc, char *argv[])
{
  printf("Hello World\n");
  return 0;
}

And we’re going to build it using make, using the following Makefile:

2.1.2. files/src/Makefile

# Sample makefile for hello.c
#
.PHONY: all install

all: hello

install:
	install -d ${DESTDIR}${PREFIX}/bin
	install -m 755 hello ${DESTDIR}${PREFIX}/bin

hello: hello.c
	$(CC) -Wall -o $@ $<

We’ll be using the most fundamental build element, the manual build element.

The manual element is the backbone on which all the other build elements are built, so understanding how it works at this level is helpful.

2.2. Project structure

In this project we have a project.conf, a directory with some source code, and 3 element declarations.

Let’s first take a peek at what we need to build using bst show:

user@host:~/running-commands$ bst show hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
   buildable 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 base/alpine.bst 
     waiting e2d966e322d011d1eeafd8e77fc7257e72cce29a813b67c66b087358e785873e base.bst 
     waiting 0d8e771194a9b434f43d1cd44355b2e992136fd0941ca5366405ad8f9d4e3e05 hello.bst 

This time we have loaded a pipeline with 3 elements, let’s go over what they do in detail.

2.2.1. project.conf

# Unique project name
name: running-commands

# Minimum required BuildStream version
min-version: 2.0

# Subdirectory where elements are stored
element-path: elements

# Define an alias for our alpine tarball
aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

Our project.conf is very much like the last one, except that we have defined a source alias for alpine.

Tip

Using source aliases for groups of sources which are generally hosted together is encouraged. This allows one to globally change the access scheme or URL for a group of repositories which belong together.

2.2.2. elements/base/alpine.bst

kind: import
description: |

    Alpine Linux base runtime

sources:
- kind: tar

  # This is a post doctored, trimmed down system image
  # of the Alpine linux distribution.
  #
  url: alpine:integration-tests-base.v1.x86_64.tar.xz
  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639

This import element uses a tar source to download our Alpine Linux tarball to create our base runtime.

This tarball is a sysroot which provides the C runtime libraries and some programs - this is what will be providing the programs we’re going to run in this example.

2.2.3. elements/base.bst

kind: stack
description: Base stack

depends:
- base/alpine.bst

This is just a symbolic stack element which declares that anything which depends on it, will implicitly depend on base/alpine.bst.

It is typical to use stack elements in places where the implementing logical software stack could change, but you rather not have your higher level components carry knowledge about those changing components.

Any element which runtime depends on the base.bst will now be able to execute programs provided by the imported base/alpine.bst runtime.

2.2.4. elements/hello.bst

kind: manual
description: |

  Building manually

# Depend on the base system
depends:
- base.bst

# Stage the files/src directory for building
sources:
  - kind: local
    path: files/src

# Now configure the commands to run
config:

  build-commands:
  - make PREFIX="%{prefix}"

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

Finally we have the element which executes commands. Looking at the manual element’s documentation, we can see that the element configuration exposes four command lists:

• configure-commands

  Commands which are run in preparation of a build. This is where you would normally call any configure stage build tools to configure the build how you like and generate some files needed for the build.

• build-commands

  Commands to run the build, usually a build system will invoke the compiler for you here.

• install-commands

  Commands to install the build results.

  Commands to install the build results into the target system, these should install files somewhere under %{install-root}.

• strip-commands

  Commands to doctor the build results after the install.

  Typically this involves stripping binaries of debugging symbols or stripping timestamps from build results to ensure reproducibility.

Tip

All other build elements implement exactly the same command lists too, except that they provide default commands specific to invoke the build systems they support.

The manual element however is the most basic and does not provide any default commands, so we have instructed it to use make to build and install our program.

2.3. Using the project

2.3.1. Build the hello.bst element

To build the project, run bst build in the following way:

user@host:~/running-commands$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources

BuildStream Version 1.93.5
    Session Start: Wednesday, 19-08-2020 at 14:52:38
    Project:       running-commands (/home/user/running-commands)
    Targets:       hello.bst

User Configuration
    Configuration File:      /home/user/.config/buildstream.conf
    Cache Directory:         /home/user/.cache/buildstream
    Log Files:               /home/user/.cache/buildstream/logs
    Source Mirrors:          /home/user/.cache/buildstream/sources
    Build Area:              /home/user/.cache/buildstream/build
    Strict Build Plan:       Yes
    Maximum Fetch Tasks:     10
    Maximum Build Tasks:     4
    Maximum Push Tasks:      4
    Maximum Network Retries: 2

Project: running-commands

    Element Plugins
        manual: core plugin
        stack:  core plugin
        import: core plugin

    Source Plugins
        local: core plugin
        tar:   core plugin

Pipeline
   buildable 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 base/alpine.bst 
     waiting e2d966e322d011d1eeafd8e77fc7257e72cce29a813b67c66b087358e785873e base.bst 
     waiting 0d8e771194a9b434f43d1cd44355b2e992136fd0941ca5366405ad8f9d4e3e05 hello.bst 
===============================================================================
[--:--:--][179c6ae9][   build:base/alpine.bst               ] START   running-commands/base-alpine/179c6ae9-build.415.log
[--:--:--][179c6ae9][   build:base/alpine.bst               ] START   Staging sources
[00:00:00][179c6ae9][   build:base/alpine.bst               ] SUCCESS Staging sources
[--:--:--][179c6ae9][   build:base/alpine.bst               ] START   Caching artifact
[00:00:00][179c6ae9][   build:base/alpine.bst               ] SUCCESS Caching artifact
[00:00:00][179c6ae9][   build:base/alpine.bst               ] SUCCESS running-commands/base-alpine/179c6ae9-build.415.log
[--:--:--][e2d966e3][   build:base.bst                      ] START   running-commands/base/e2d966e3-build.421.log
[--:--:--][e2d966e3][   build:base.bst                      ] START   Caching artifact
[00:00:00][e2d966e3][   build:base.bst                      ] SUCCESS Caching artifact
[00:00:00][e2d966e3][   build:base.bst                      ] SUCCESS running-commands/base/e2d966e3-build.421.log
[--:--:--][0d8e7711][   build:hello.bst                     ] START   running-commands/hello/0d8e7711-build.425.log
[--:--:--][0d8e7711][   build:hello.bst                     ] START   Staging dependencies
[00:00:00][0d8e7711][   build:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][0d8e7711][   build:hello.bst                     ] START   Staging sources
[00:00:00][0d8e7711][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][0d8e7711][   build:hello.bst                     ] START   Running commands

    make PREFIX="/usr"
    make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install

[00:00:00][0d8e7711][   build:hello.bst                     ] SUCCESS Running commands
[--:--:--][0d8e7711][   build:hello.bst                     ] START   Caching artifact
[00:00:00][0d8e7711][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:00][0d8e7711][   build:hello.bst                     ] SUCCESS running-commands/hello/0d8e7711-build.425.log
[00:00:00][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
    Total:       3
    Session:     3
    Fetch Queue: processed 0, skipped 3, failed 0 
    Build Queue: processed 3, skipped 0, failed 0

Now we’ve built our hello world program, using make and the C compiler provided by the Alpine Linux image.

In the first chapter we observed that the inputs and output of an element are directory trees. In this example, the directory tree generated by base/alpine.bst is consumed by hello.bst due to the implicit runtime dependency introduced by base.bst.

Tip

All of the dependencies which are required to run for the sake of a build, are staged at the root of the build sandbox. These comprise the runtime environment in which the depending element will run commands.

The result is that the make program and C compiler provided by base/alpine.bst were already in $PATH and ready to run when the commands were needed by hello.bst.

Now observe that all of the elements in the loaded pipeline are cached, the element is built:

user@host:~/running-commands$ bst show hello.bst

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
      cached 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 base/alpine.bst 
      cached e2d966e322d011d1eeafd8e77fc7257e72cce29a813b67c66b087358e785873e base.bst 
      cached 0d8e771194a9b434f43d1cd44355b2e992136fd0941ca5366405ad8f9d4e3e05 hello.bst 

2.3.2. Run the hello world program

Now that we’ve built everything, we can indulge ourselves in running the hello world program using bst shell:

user@host:~/running-commands$ bst shell hello.bst -- hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
[--:--:--][        ][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][        ][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][        ][    main:hello.bst                     ] STATUS  Running command

    hello

2020-08-19T14:52:40.620+0000 [492:140313127266368] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:52:40.620+0000 [492:140313127266368] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello World

Here, bst shell created a runtime environment for running the hello.bst element. This was done by staging all of the dependencies of hello.bst including the hello.bst output itself into a directory. Once a directory with all of the dependencies was staged and ready, we ran the hello command from within the build sandbox environment.

Tip

When specifying a command for bst shell to run, we always specify -- first. This is a commonly understood shell syntax to indicate that the remaining arguments are to be treated literally.

Specifying -- is optional and disambiguates BuildStream’s arguments and options from those of the program being run by bst shell.

2.4. Summary

In this chapter we’ve explored how to use the manual element, which forms the basis of all build elements.

We’ve also observed how the directory tree from the output artifact of one element is later staged at the root of the sandbox, as input for use by any build elements which depend on that element.

Tip

The way that elements consume their dependency input can vary across the different kinds of elements. This chapter describes how it works for build elements implementations, which are the most commonly used element type.