2. Strict mode¶
In this section, we will cover the usage of strict vs non-strict build plans in conjunction with workspaces, and how this can help to improve your edit/compile/test cycles.
Note
This example is distributed with BuildStream in the doc/examples/strict-mode subdirectory.
2.1. Overview¶
When working with BuildStream to create integrations, it is typical that you have a lot of components to build, and you frequently need to modify a component at various levels of the stack. When developing one or more applications, you might want to open a workspace and fix a bug in an application, or you might need to open a workspace on a low level shared library to fix the behavior of one or more misbehaving applications.
By default, BuildStream will always choose to be deterministic in order to produce the most correct build results as possible. As such, modifying a low level library will result in rebuilding all of it’s reverse dependencies, but this can be very time consuming and inconvenient for your edit/compile/test cycles.
This is when enabling non-strict build plans can be helpful.
To illustrate the facets of how this works, this example will present a project consisting of an application which is linked both statically and dynamically linked to a common library.
2.2. Project structure¶
This project is mostly based on the integration commands example, as such we will ignore large parts of this project and only focus on the elements which are of specific interest.
To illustrate the relationship of these two applications and the library, let’s briefly take a look at the underlying Makefiles which are used in this project, starting with the library and followed by both Makefiles used to build the application.
2.2.1. files/libhello/Makefile¶
# Sample makefile for hello library
#
.PHONY: all install
all: libhello.so libhello.a
install: all
install -d ${DESTDIR}${PREFIX}/lib
install -d ${DESTDIR}${PREFIX}/include
install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.a ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
%.o: %.c %.h
$(CC) -c $< -o $@ -Wall
libhello.a: libhello.o
$(AR) rcs $@ $^
libhello.so: libhello.o
$(CC) -shared -o $@ $<
2.2.2. files/hello/Makefile.dynamic¶
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install: all
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $< -lhello
2.2.3. files/hello/Makefile.static¶
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install: all
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $< /usr/lib/libhello.a
As we can see, we have a library that is distributed both as the dynamic
library libhello.so and also as the static archive libhello.a.
Now let’s take a look at the two separate elements which build the application, first the dynamically linked version and then the static one.
2.2.4. elements/hello-dynamic.bst¶
kind: manual
description: |
The dynamically linked hello application
# Depend on the hello library
depends:
- libhello.bst
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make -f Makefile.dynamic PREFIX="%{prefix}"
install-commands:
- make -f Makefile.dynamic -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Nothing very special to observe about this hello program, just a
manual element quite similar to the one we’ve
already seen in the running commands
example.
2.2.5. elements/hello-static.bst¶
kind: manual
description: |
The statically linked hello application
# Depend on the hello library with the strict option
#
depends:
- filename: libhello.bst
strict: true
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make -f Makefile.static PREFIX="%{prefix}"
install-commands:
- make -f Makefile.static -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Almost the same as the dynamic element, except here we have declared the
dependency to the libhello.bst element differently: this time we have enabled
the strict option in the dependency declaration.
The side effect of setting this option is that hello-static.bst will be
rebuilt any time that libhello.bst has changed, even when
non-strict build plans have been enabled.
Tip
Some element plugins are designed to consume the content of their
dependencies entirely, and output an artifact without any transient
runtime dependencies, an example of this is the compose
element.
In cases such as compose, it is not necessary to
explicitly annotate their dependencies as strict.
It is only helpful to set the strict attribute on a
dependency declaration in the case that the
specific dependency relationship causes data to be consumed verbatim,
as is the case with static linking.
2.3. Using the project¶
For the sake of brevity, let’s assume that you’ve already built all of the
elements of this project, and that you want to make some changes to the
libhello.bst element, and test how it might effect the hello program.
2.3.1. Everything is already built¶
user@host:~/strict-mode$ bst show hello-static.bst hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached 7b81cb08196aa99db72aaad24c7d53fea9bf381ffc99c0ba8408be7dcd0e31b5 base/alpine.bst
cached bb236c789793a2a3d8f666113b06d1484d96728801b704239556a5a432fcfde2 base.bst
cached 44fc0cae34345d6aea9d09ab67b9e27aae25cca4c4594cfa586cd39a73eed336 libhello.bst
cached b719dd20f6f75668cb233abd9831333a1969396be340ac8c90ce862f0b74f6ac hello-static.bst
cached ebe74811da86630ebb031093ef05ff8f2c84bdcc8e9fe644fda09b3e210bb844 hello-dynamic.bst
2.3.2. Open a workspace and modify libhello.c¶
Now let’s open up a workspace on the hello library
user@host:~/strict-mode$ bst workspace open --directory workspace_libhello libhello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
[--:--:--][ ][ main:core activity ] INFO Creating workspace for element libhello.bst
[--:--:--][ ][ main:libhello.bst ] START Staging sources to /home/user/strict-mode/workspace_libhello
[--:--:--][ ][ main:libhello.bst ] START Staging local files into CAS
[00:00:00][ ][ main:libhello.bst ] SUCCESS Staging local files into CAS
[00:00:00][ ][ main:libhello.bst ] SUCCESS Staging sources to /home/user/strict-mode/workspace_libhello
[--:--:--][ ][ main:core activity ] INFO Created a workspace for element: libhello.bst
And go ahead and make a modification like this:
--- libhello.c
+++ libhello.c
@@ -5,5 +5,5 @@
void hello(const char *person)
{
- printf("Hello %s\n", person);
+ printf("Good morning %s\n", person);
}
2.3.3. Observing hello-dynamic.bst¶
Let’s take a look at the bst show output for
the dynamically linked hello-dynamic.bst element.
user@host:~/strict-mode$ bst show hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached 7b81cb08196aa99db72aaad24c7d53fea9bf381ffc99c0ba8408be7dcd0e31b5 base/alpine.bst
cached bb236c789793a2a3d8f666113b06d1484d96728801b704239556a5a432fcfde2 base.bst
buildable 698246731b034344d933d80a01c88a15ba0bec2b08827f552afd4c3d5ac341d8 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
waiting 4ed6373c1ce58a589ede6015be721b0e2372a28f40cc155a716c8b6d1a6ca2a5 hello-dynamic.bst
As one might expect, the libhello.bst element is ready to be built
after having been modified, and the hello-dynamic.bst element is
waiting for libhello.bst to be built before it can build.
Now let’s take a look at the same elements if we pass the --no-strict
option to bst:
user@host:~/strict-mode$ bst --no-strict show hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached 7b81cb08196aa99db72aaad24c7d53fea9bf381ffc99c0ba8408be7dcd0e31b5 base/alpine.bst
cached bb236c789793a2a3d8f666113b06d1484d96728801b704239556a5a432fcfde2 base.bst
buildable 698246731b034344d933d80a01c88a15ba0bec2b08827f552afd4c3d5ac341d8 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
cached ebe74811da86630ebb031093ef05ff8f2c84bdcc8e9fe644fda09b3e210bb844 hello-dynamic.bst
Note that this time, the libhello.bst still needs to be built,
but the hello-dymamic.bst element is showing up as cached.
Tip
The bst show output will show some cache keys dimmed out in the case that they are not entirely deterministic.
Here we can see that hello-dynamic.bst is dimmed out because
it will not be rebuilt against the changed libhello.bst element,
and it also has a different cache key because of this.
2.3.4. Observing hello-static.bst¶
Now let’s observe the hello-static.bst element with strict mode
disabled:
user@host:~/strict-mode$ bst --no-strict show hello-static.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached 7b81cb08196aa99db72aaad24c7d53fea9bf381ffc99c0ba8408be7dcd0e31b5 base/alpine.bst
cached bb236c789793a2a3d8f666113b06d1484d96728801b704239556a5a432fcfde2 base.bst
buildable 698246731b034344d933d80a01c88a15ba0bec2b08827f552afd4c3d5ac341d8 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
waiting 92f8d883ccd10ae069ab4903e277977c92d62e99373c19b6b089e67bd8a44f00 hello-static.bst
Note that in this case the hello-strict.bst is going to be
rebuilt even in strict mode. This is because we annotated the
declaration of the libhello.bst dependency with the strict
attribute.
We did this because hello-strict.bst consumes the input of
libhello.bst verbatim, by way of statically linking to it, instead
of merely being affected by the content of libhello.bst at runtime,
as would be the case of static linking.
2.3.5. Building and running hello-dynamic.bst¶
Now let’s build hello-dynamic.bst with strict mode disabled.
user@host:~/strict-mode$ bst --no-strict build hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:53:31
Project: strict-mode (/home/user/strict-mode)
Targets: hello-dynamic.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: No
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: strict-mode
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
workspace: core plugin
tar: core plugin
Pipeline
cached 7b81cb08196aa99db72aaad24c7d53fea9bf381ffc99c0ba8408be7dcd0e31b5 base/alpine.bst
cached bb236c789793a2a3d8f666113b06d1484d96728801b704239556a5a432fcfde2 base.bst
buildable 698246731b034344d933d80a01c88a15ba0bec2b08827f552afd4c3d5ac341d8 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
cached ebe74811da86630ebb031093ef05ff8f2c84bdcc8e9fe644fda09b3e210bb844 hello-dynamic.bst
===============================================================================
[--:--:--][69824673][ fetch:libhello.bst ] START strict-mode/libhello/69824673-fetch.1401.log
[00:00:00][69824673][ fetch:libhello.bst ] SUCCESS strict-mode/libhello/69824673-fetch.1401.log
[--:--:--][69824673][ build:libhello.bst ] START strict-mode/libhello/69824673-build.1405.log
[--:--:--][69824673][ build:libhello.bst ] START Staging dependencies
[00:00:00][69824673][ build:libhello.bst ] SUCCESS Staging dependencies
[--:--:--][69824673][ build:libhello.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:01][69824673][ build:libhello.bst ] SUCCESS Running commands
[--:--:--][69824673][ build:libhello.bst ] START Staging sources
[00:00:00][69824673][ build:libhello.bst ] SUCCESS Staging sources
[--:--:--][69824673][ build:libhello.bst ] START Running commands
make PREFIX="/usr"
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][69824673][ build:libhello.bst ] SUCCESS Running commands
[--:--:--][69824673][ build:libhello.bst ] START Caching artifact
[00:00:00][69824673][ build:libhello.bst ] SUCCESS Caching artifact
[00:00:01][69824673][ build:libhello.bst ] SUCCESS strict-mode/libhello/69824673-build.1405.log
[00:00:02][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 4
Session: 1
Fetch Queue: processed 1, skipped 0, failed 0
Build Queue: processed 1, skipped 0, failed 0
Note that the bst build command completed without
having to build hello-dynamic.bst at all.
And now we can also run hello-dynamic.bst
user@host:~/strict-mode$ bst --no-strict shell hello-dynamic.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello-dynamic.bst ] START Staging dependencies
[00:00:00][ ][ main:hello-dynamic.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello-dynamic.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello-dynamic.bst ] START Running commands
ldconfig "/usr/lib"
2020-08-19T14:53:33.838+0000 [1479:140709478131776] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:33.839+0000 [1479:140709478131776] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
+ sh -e -c ldconfig "/usr/lib"
[00:00:01][ ][ main:hello-dynamic.bst ] SUCCESS Running commands
[00:00:01][ ][ main:hello-dynamic.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello-dynamic.bst ] STATUS Running command
hello
2020-08-19T14:53:35.309+0000 [1500:140179629955136] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:35.309+0000 [1500:140179629955136] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Good morning stranger
When running hello-dynamic.bst with no-strict mode, we are
actually reusing the old build of hello-dynamic.bst staged against
the new build of the modified libhello.bst element.
2.3.6. Building and running hello-static.bst¶
Finally, if we build hello-static.bst with strict mode disabled,
we can see that it will be rebuilt regardless of strict mode being
enabled.
user@host:~/strict-mode$ bst --no-strict build hello-static.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:53:35
Project: strict-mode (/home/user/strict-mode)
Targets: hello-static.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: No
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: strict-mode
Element Plugins
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
workspace: core plugin
tar: core plugin
Pipeline
cached 7b81cb08196aa99db72aaad24c7d53fea9bf381ffc99c0ba8408be7dcd0e31b5 base/alpine.bst
cached bb236c789793a2a3d8f666113b06d1484d96728801b704239556a5a432fcfde2 base.bst
cached 698246731b034344d933d80a01c88a15ba0bec2b08827f552afd4c3d5ac341d8 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
buildable 92f8d883ccd10ae069ab4903e277977c92d62e99373c19b6b089e67bd8a44f00 hello-static.bst
===============================================================================
[--:--:--][92f8d883][ build:hello-static.bst ] START strict-mode/hello-static/92f8d883-build.1535.log
[--:--:--][92f8d883][ build:hello-static.bst ] START Staging dependencies
[00:00:00][92f8d883][ build:hello-static.bst ] SUCCESS Staging dependencies
[--:--:--][92f8d883][ build:hello-static.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:01][92f8d883][ build:hello-static.bst ] SUCCESS Running commands
[--:--:--][92f8d883][ build:hello-static.bst ] START Staging sources
[00:00:00][92f8d883][ build:hello-static.bst ] SUCCESS Staging sources
[--:--:--][92f8d883][ build:hello-static.bst ] START Running commands
make -f Makefile.static PREFIX="/usr"
make -f Makefile.static -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][92f8d883][ build:hello-static.bst ] SUCCESS Running commands
[--:--:--][92f8d883][ build:hello-static.bst ] START Caching artifact
[00:00:00][92f8d883][ build:hello-static.bst ] SUCCESS Caching artifact
[00:00:01][92f8d883][ build:hello-static.bst ] SUCCESS strict-mode/hello-static/92f8d883-build.1535.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 4
Session: 1
Fetch Queue: processed 0, skipped 1, failed 0
Build Queue: processed 1, skipped 0, failed 0
This is of course because we declared its dependency on libhello.bst
as a strict dependency.
And by the same virtue, we can see that when we run the example it has properly relinked against the changed static archive, and has the updated text in the greeting:
user@host:~/strict-mode$ bst --no-strict shell hello-static.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello-static.bst ] START Staging dependencies
[00:00:00][ ][ main:hello-static.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello-static.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello-static.bst ] START Running commands
ldconfig "/usr/lib"
2020-08-19T14:53:38.044+0000 [1604:139730387032128] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:38.044+0000 [1604:139730387032128] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
+ sh -e -c ldconfig "/usr/lib"
[00:00:01][ ][ main:hello-static.bst ] SUCCESS Running commands
[00:00:01][ ][ main:hello-static.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello-static.bst ] STATUS Running command
hello
2020-08-19T14:53:39.393+0000 [1625:139643180886080] [buildboxcommon_runner.cpp:547] [WARNING] The --use-localcas option will be deprecated. LocalCAS support is now enabled by default.
2020-08-19T14:53:39.394+0000 [1625:139643180886080] [buildboxcommon_client.cpp:87] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default
Good morning stranger
2.4. Summary¶
In this chapter we’ve explored how to use non-strict build plans in order to avoid rebuilding reverse dependencies of a lower level element you might be working with in a workspace, consequently improving your edit/compile/test experience.
We’ve also explained how to ensure your project still works properly
with non-strict build plans when some elements perform static linking
(or other operations which consume data from their dependencies
verbatim), by annotating dependency declarations
as strict.