1. Composition¶
In this chapter we will explore how to create compositions of multiple
input filesystem trees, using the compose element.
Note
This example is distributed with BuildStream in the doc/examples/composition subdirectory.
1.1. Overview¶
Composing a directory tree based on a set of build dependencies is often one of the important steps you might perform in order to create a single artifact which can be checked out and deployed.
In order to use the compose element, it is important
to first understand the concept of split rules, which
we will cover in this chapter.
1.1.1. Introducing split rules¶
The split rules of an element declaration denote which sets of files in the given element’s resulting artifact belong to which domain name.
The domains can then be used in various ways, using plugins which understand split rule domains.
BuildStream’s default project configuration contains a sensible set of default split rule domains for the purpose of artifact splitting, they can be overridden in your project.conf, and finally on a per element basis in the public data of your element declarations.
Note
Projects are free to add additional split rule domains on top of the default domains provided by the default project configuration.
There is nothing wrong with defining split rule domains which overlap,
possibly capturing some of the same files also captured by another
domain, however you should be aware of this when later using your
split rules with a plugin which processes them, like the
compose element described in this chapter.
1.1.1.1. Example of split rule declaration¶
In an element, you might need to define or extend the split-rules
in order to capture files in custom locations in a logical domain.
Here is an example of how you might use the
list append directive
to append an additional rule to your split-rules list in order to
capture additional data files which your application or library might
want to include in the runtime domain:
# Add our .dat files to the runtime domain
public:
bst:
split-rules:
runtime:
(>):
- |
%{datadir}/foo/*.dat
Split rules are absolute paths which denote files within an artifact’s root
directory. The globbing patterns supported in split rules are defined in the
reference documentation here.
Important
Note that because of variable expansion, split rules can often be
resolved differently for elements which have overridden path
related variables, like %{prefix}.
This usually means that you do not need to explicitly extend or override split rules on a specific element unless your element installs files to special case locations.
1.2. Project structure¶
In this example we expand on the chapter about integration commands, so we will only discuss the files which are added or changed from that example.
1.2.1. elements/base/alpine.bst¶
kind: import
description: |
Alpine Linux base runtime
sources:
- kind: tar
url: alpine:integration-tests-base.v1.x86_64.tar.xz
ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
public:
bst:
#
# Run ldconfig in the libdir before running anything
#
integration-commands:
- ldconfig "%{libdir}"
#
# Extend the runtime split-rule domain for this element,
# such that we capture the runtime linker.
#
# There are various other things provided by this runtime
# such as tooling in /bin and an installation of python
# and perl, but we'll overlook these for the sake of
# this example.
#
split-rules:
runtime:
(>):
- "/lib/ld*.so*"
Here we have modified the base runtime, so as to specify that for this element, we want to also include the runtime linker into the runtime domain.
1.2.2. elements/runtime-only.bst¶
kind: compose
# Dependencies of a compose element cannot be transient,
# we can only build-depend on the inputs of a composition.
#
build-depends:
- hello.bst
config:
# Only include files from the runtime domain
#
include:
- runtime
# Don't include any files which do not match any existing
# split rule domains.
#
include-orphans: False
# Run integration commands before composition
#
integrate: True
As we can see, this compose element has
been configured to only include files from the runtime domain.
1.3. Using the project¶
Now that we’ve presented how split rules
work and shown how to use them in the context of this example, lets
use the compose element we’ve created and
observe the results.
1.3.1. Building the project¶
user@host:~/composition$ bst build runtime-only.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:52:51
Project: composition (/home/user/composition)
Targets: runtime-only.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: composition
Element Plugins
compose: core plugin
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
buildable 8468acd37cea031d15158fac387c4a783a9e42b275f1b91fb73e38baa3e49df4 base/alpine.bst
waiting 5ea28a5a98babd7d7f8510bcdda8ad09d44b2a0d77e89832c8a9cf6e6ef7523b base.bst
waiting fede43c61a86711b6b8e534f5f1669437e54268d9487658fe1fdd77e07538ff0 libhello.bst
waiting cf78ae7b8cc4a8876a16e894de3c3b1c73a1e2b62a7250db94dd3c48d02fe611 hello.bst
waiting f910812e55a1007cb79d9e085c90861eccc89257e4a132723ed1d59cfc93743c runtime-only.bst
===============================================================================
[--:--:--][8468acd3][ build:base/alpine.bst ] START composition/base-alpine/8468acd3-build.560.log
[--:--:--][8468acd3][ build:base/alpine.bst ] START Staging sources
[00:00:00][8468acd3][ build:base/alpine.bst ] SUCCESS Staging sources
[--:--:--][8468acd3][ build:base/alpine.bst ] START Caching artifact
[00:00:00][8468acd3][ build:base/alpine.bst ] SUCCESS Caching artifact
[00:00:00][8468acd3][ build:base/alpine.bst ] SUCCESS composition/base-alpine/8468acd3-build.560.log
[--:--:--][5ea28a5a][ build:base.bst ] START composition/base/5ea28a5a-build.566.log
[--:--:--][5ea28a5a][ build:base.bst ] START Caching artifact
[00:00:00][5ea28a5a][ build:base.bst ] SUCCESS Caching artifact
[00:00:00][5ea28a5a][ build:base.bst ] SUCCESS composition/base/5ea28a5a-build.566.log
[--:--:--][fede43c6][ build:libhello.bst ] START composition/libhello/fede43c6-build.570.log
[--:--:--][fede43c6][ build:libhello.bst ] START Staging dependencies
[00:00:00][fede43c6][ build:libhello.bst ] SUCCESS Staging dependencies
[--:--:--][fede43c6][ build:libhello.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:01][fede43c6][ build:libhello.bst ] SUCCESS Running commands
[--:--:--][fede43c6][ build:libhello.bst ] START Staging sources
[00:00:00][fede43c6][ build:libhello.bst ] SUCCESS Staging sources
[--:--:--][fede43c6][ build:libhello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][fede43c6][ build:libhello.bst ] SUCCESS Running commands
[--:--:--][fede43c6][ build:libhello.bst ] START Caching artifact
[00:00:00][fede43c6][ build:libhello.bst ] SUCCESS Caching artifact
[00:00:01][fede43c6][ build:libhello.bst ] SUCCESS composition/libhello/fede43c6-build.570.log
[--:--:--][cf78ae7b][ build:hello.bst ] START composition/hello/cf78ae7b-build.620.log
[--:--:--][cf78ae7b][ build:hello.bst ] START Staging dependencies
[00:00:00][cf78ae7b][ build:hello.bst ] SUCCESS Staging dependencies
[--:--:--][cf78ae7b][ build:hello.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:01][cf78ae7b][ build:hello.bst ] SUCCESS Running commands
[--:--:--][cf78ae7b][ build:hello.bst ] START Staging sources
[00:00:00][cf78ae7b][ build:hello.bst ] SUCCESS Staging sources
[--:--:--][cf78ae7b][ build:hello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][cf78ae7b][ build:hello.bst ] SUCCESS Running commands
[--:--:--][cf78ae7b][ build:hello.bst ] START Caching artifact
[00:00:00][cf78ae7b][ build:hello.bst ] SUCCESS Caching artifact
[00:00:01][cf78ae7b][ build:hello.bst ] SUCCESS composition/hello/cf78ae7b-build.620.log
[--:--:--][f910812e][ build:runtime-only.bst ] START composition/runtime-only/f910812e-build.667.log
[--:--:--][f910812e][ build:runtime-only.bst ] START Staging dependencies
[00:00:00][f910812e][ build:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][f910812e][ build:runtime-only.bst ] START Computing split
[00:00:00][f910812e][ build:runtime-only.bst ] SUCCESS Computing split
[--:--:--][f910812e][ build:runtime-only.bst ] START Integrating sandbox
[--:--:--][f910812e][ build:runtime-only.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:01][f910812e][ build:runtime-only.bst ] SUCCESS Running commands
[--:--:--][f910812e][ build:runtime-only.bst ] INFO Integration modified 0, added 6 and removed 0 files
[00:00:01][f910812e][ build:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][f910812e][ build:runtime-only.bst ] START Creating composition
Including files from domains: runtime
Excluding orphaned files
[--:--:--][f910812e][ build:runtime-only.bst ] INFO Composing 293 files
[00:00:00][f910812e][ build:runtime-only.bst ] SUCCESS Creating composition
[--:--:--][f910812e][ build:runtime-only.bst ] START Caching artifact
[00:00:00][f910812e][ build:runtime-only.bst ] SUCCESS Caching artifact
[00:00:01][f910812e][ build:runtime-only.bst ] SUCCESS composition/runtime-only/f910812e-build.667.log
[00:00:05][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 5
Session: 5
Fetch Queue: processed 0, skipped 5, failed 0
Build Queue: processed 5, skipped 0, failed 0
As you can see in the output, this composition has only a few hundred
files, but the complete alpine.bst runtime has several thousand
files.
1.3.2. List the content¶
At the risk of this being a long list, let’s list the contents of this artifact
user@host:~/composition$ bst artifact list-contents runtime-only.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
runtime-only.bst:
dev
dev/full
dev/null
dev/random
dev/shm
dev/urandom
dev/zero
lib
lib/ld-musl-x86_64.so.1
usr
usr/bin
usr/bin/2to3
usr/bin/2to3-3.6
usr/bin/[
usr/bin/[[
usr/bin/aclocal
usr/bin/aclocal-1.15
usr/bin/addr2line
usr/bin/ar
usr/bin/as
usr/bin/autoconf
usr/bin/autoheader
usr/bin/autom4te
usr/bin/automake
usr/bin/automake-1.15
usr/bin/autoreconf
usr/bin/autoscan
usr/bin/autoupdate
usr/bin/awk
usr/bin/basename
usr/bin/beep
usr/bin/blkdiscard
usr/bin/bunzip2
usr/bin/bzcat
usr/bin/bzip2
usr/bin/c++filt
usr/bin/c_rehash
usr/bin/cal
usr/bin/cc
usr/bin/ccmake
usr/bin/chvt
usr/bin/cksum
usr/bin/clear
usr/bin/cmake
usr/bin/cmp
usr/bin/comm
usr/bin/cpack
usr/bin/cpio
usr/bin/crontab
usr/bin/cryptpw
usr/bin/ctest
usr/bin/cut
usr/bin/dc
usr/bin/deallocvt
usr/bin/diff
usr/bin/dirname
usr/bin/dos2unix
usr/bin/du
usr/bin/dumpleases
usr/bin/dwp
usr/bin/easy_install-3.6
usr/bin/eject
usr/bin/elfedit
usr/bin/env
usr/bin/expand
usr/bin/expr
usr/bin/factor
usr/bin/fallocate
usr/bin/find
usr/bin/flock
usr/bin/fold
usr/bin/free
usr/bin/fuser
usr/bin/gdbm_dump
usr/bin/gdbm_load
usr/bin/gdbmtool
usr/bin/getconf
usr/bin/getent
usr/bin/gprof
usr/bin/groups
usr/bin/hd
usr/bin/head
usr/bin/hello
usr/bin/hexdump
usr/bin/hostid
usr/bin/iconv
usr/bin/id
usr/bin/ifnames
usr/bin/install
usr/bin/ipcrm
usr/bin/ipcs
usr/bin/killall
usr/bin/ld
usr/bin/ld.bfd
usr/bin/ldd
usr/bin/less
usr/bin/logger
usr/bin/lsof
usr/bin/lsusb
usr/bin/lzcat
usr/bin/lzma
usr/bin/lzopcat
usr/bin/m4
usr/bin/make
usr/bin/md5sum
usr/bin/mesg
usr/bin/microcom
usr/bin/mkfifo
usr/bin/mkpasswd
usr/bin/nc
usr/bin/nl
usr/bin/nm
usr/bin/nmeter
usr/bin/nohup
usr/bin/nproc
usr/bin/nsenter
usr/bin/nslookup
usr/bin/objcopy
usr/bin/objdump
usr/bin/od
usr/bin/openvt
usr/bin/passwd
usr/bin/paste
usr/bin/patch
usr/bin/perl
usr/bin/perl5.26.1
usr/bin/perldoc
usr/bin/pgrep
usr/bin/pip3
usr/bin/pip3.6
usr/bin/pkill
usr/bin/pmap
usr/bin/pod2html
usr/bin/pod2man
usr/bin/pod2text
usr/bin/pod2usage
usr/bin/podchecker
usr/bin/podselect
usr/bin/printf
usr/bin/pscan
usr/bin/pstree
usr/bin/pwdx
usr/bin/pydoc3
usr/bin/pydoc3.6
usr/bin/python3
usr/bin/python3.6
usr/bin/python3.6m
usr/bin/pyvenv
usr/bin/pyvenv-3.6
usr/bin/ranlib
usr/bin/readelf
usr/bin/readlink
usr/bin/realpath
usr/bin/renice
usr/bin/reset
usr/bin/resize
usr/bin/scanelf
usr/bin/seq
usr/bin/setkeycodes
usr/bin/setsid
usr/bin/sha1sum
usr/bin/sha256sum
usr/bin/sha3sum
usr/bin/sha512sum
usr/bin/showkey
usr/bin/shred
usr/bin/shuf
usr/bin/size
usr/bin/smemcap
usr/bin/sort
usr/bin/split
usr/bin/ssl_client
usr/bin/strings
usr/bin/strip
usr/bin/sum
usr/bin/tac
usr/bin/tail
usr/bin/tcc
usr/bin/tee
usr/bin/test
usr/bin/time
usr/bin/timeout
usr/bin/top
usr/bin/tr
usr/bin/traceroute
usr/bin/traceroute6
usr/bin/truncate
usr/bin/tty
usr/bin/ttysize
usr/bin/udhcpc6
usr/bin/unexpand
usr/bin/uniq
usr/bin/unix2dos
usr/bin/unlink
usr/bin/unlzma
usr/bin/unlzop
usr/bin/unshare
usr/bin/unxz
usr/bin/unzip
usr/bin/uptime
usr/bin/uudecode
usr/bin/uuencode
usr/bin/vi
usr/bin/vlock
usr/bin/volname
usr/bin/wc
usr/bin/wget
usr/bin/which
usr/bin/whoami
usr/bin/whois
usr/bin/xargs
usr/bin/xmlwf
usr/bin/xxd
usr/bin/xzcat
usr/bin/yes
usr/lib
usr/lib/libarchive.so.13
usr/lib/libarchive.so.13.3.2
usr/lib/libbfd-2.28.so
usr/lib/libbz2.so.1
usr/lib/libbz2.so.1.0.6
usr/lib/libc.so
usr/lib/libcrypto.so.42
usr/lib/libcrypto.so.42.0.0
usr/lib/libcurl.so.4
usr/lib/libcurl.so.4.5.0
usr/lib/libexpat.so.1
usr/lib/libexpat.so.1.6.7
usr/lib/libffi.so.6
usr/lib/libffi.so.6.0.4
usr/lib/libformw.so.6
usr/lib/libformw.so.6.0
usr/lib/libgcc_s.so.1
usr/lib/libgdbm.so.4
usr/lib/libgdbm.so.4.0.0
usr/lib/libgdbm_compat.so.4
usr/lib/libgdbm_compat.so.4.0.0
usr/lib/libhello.so
usr/lib/liblz4.so.1
usr/lib/liblz4.so.1.8.0
usr/lib/liblzma.so.5
usr/lib/liblzma.so.5.2.3
usr/lib/libmenuw.so.6
usr/lib/libmenuw.so.6.0
usr/lib/libncursesw.so.6
usr/lib/libncursesw.so.6.0
usr/lib/libopcodes-2.28.so
usr/lib/libpanelw.so.6
usr/lib/libpanelw.so.6.0
usr/lib/libpython3.6m.so.1.0
usr/lib/libpython3.so
usr/lib/libreadline.so.7
usr/lib/libreadline.so.7.0
usr/lib/librhash.so.0
usr/lib/libsqlite3.so.0
usr/lib/libsqlite3.so.0.8.6
usr/lib/libssh2.so.1
usr/lib/libssh2.so.1.0.1
usr/lib/libssl.so.44
usr/lib/libssl.so.44.0.1
usr/lib/libstdc++.so.6
usr/lib/libstdc++.so.6.0.22
usr/lib/libuv.so.1
usr/lib/libuv.so.1.0.0
usr/sbin
usr/sbin/add-shell
usr/sbin/addgroup
usr/sbin/adduser
usr/sbin/arping
usr/sbin/brctl
usr/sbin/chpasswd
usr/sbin/chroot
usr/sbin/crond
usr/sbin/delgroup
usr/sbin/deluser
usr/sbin/ether-wake
usr/sbin/fbset
usr/sbin/fdformat
usr/sbin/killall5
usr/sbin/loadfont
usr/sbin/lspci
usr/sbin/nanddump
usr/sbin/nandwrite
usr/sbin/nbd-client
usr/sbin/ntpd
usr/sbin/partprobe
usr/sbin/powertop
usr/sbin/rdate
usr/sbin/rdev
usr/sbin/readahead
usr/sbin/readprofile
usr/sbin/remove-shell
usr/sbin/rfkill
usr/sbin/sendmail
usr/sbin/setfont
usr/sbin/setlogcons
usr/sbin/update-ca-certificates
Some things to observe here:
The list does include the
/usr/bin/helloprogram and also the/usr/lib/libhello.soshared library.These paths are both captured by the default split rules for the runtime domain.
The list does not include the
/usr/include/libhello.hheader file which was used to compile/usr/bin/hello.The header file is not captured by the runtime domain by default. It is however captured by the devel domain.
The runtime linker
/lib/ld-musl-x86_64.so.1, as this was explicitly added to the runtime domain for thebase/alpine.bstelement which provides this file.
Tip
The reader at this time might want to list the content of
other elements built from this project, such as the
hello.bst element by itself, or the base/alpine.bst
element.
1.3.3. Run the program¶
Finally, lets just run the program we built.
user@host:~/composition$ bst shell runtime-only.bst -- hello audience
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:runtime-only.bst ] START Staging dependencies
[00:00:00][ ][ main:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:runtime-only.bst ] START Integrating sandbox
[00:00:00][ ][ main:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:runtime-only.bst ] STATUS Running command
hello audience
2020-08-19T14:52:57.783+0000 [733:140652674984000] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:52:57.784+0000 [733:140652674984000] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Hello audience
Here we can see that we at least have the required files to run
our hello world program, however we would not have if we were
missing the runtime linker which we added in base/alpine.bst.
1.4. Summary¶
In this chapter we’ve gotten familiar with split rules
annotations, and we’ve learned enough about the compose
element such that we can start creating our own compositions using
split domains.
We’ve also used the list append directive and we are now observing the contents of artifacts using bst artifact list-contents.