3. Overlapping files¶
In this chapter, we will discuss what happens when files from multiple element artifacts conflict with eachother, and what strategies we can use to resolve these situations.
Note
This example is distributed with BuildStream in the doc/examples/overlaps subdirectory.
3.1. Overview¶
This project builds on the previous chapter on composition, and as such we’ll only go over what has changed from there, which is not much.
3.2. Project structure¶
In this example we’ve just extended the libhello.bst and the hello.bst
elements such that they both install an additional file: %{docdir}/hello.txt.
We’ve updated the following Makefiles:
3.2.1. files/libhello/Makefile¶
# Sample makefile for hello library
#
.PHONY: all install
all: libhello.so
install:
install -d ${DESTDIR}${PREFIX}/lib
install -d ${DESTDIR}${PREFIX}/include
install -d ${DESTDIR}${PREFIX}/share/doc
install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
%.o: %.c %.h
$(CC) -c $< -o $@ -Wall
libhello.so: libhello.o
$(CC) -shared -o $@ $<
3.2.2. files/hello/Makefile¶
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -d ${DESTDIR}${PREFIX}/share/doc
install -m 755 hello ${DESTDIR}${PREFIX}/bin
install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
hello: hello.c
$(CC) $< -o $@ -Wall -lhello
As you can see, this now presents a conflict of sorts, where multiple elements in the pipeline want to install the same file.
3.3. Using the project¶
In this chapter, we’re only going to present the warning and then discuss how to mitigate this situation.
See what happens when we try to build the runtime-only.bst
compose element:
user@host:~/overlaps$ bst build runtime-only.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.5
Session Start: Wednesday, 19-08-2020 at 14:55:30
Project: overlaps (/home/user/overlaps)
Targets: runtime-only.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Project: overlaps
Element Plugins
compose: core plugin
manual: core plugin
stack: core plugin
import: core plugin
Source Plugins
local: core plugin
tar: core plugin
Pipeline
cached 8468acd37cea031d15158fac387c4a783a9e42b275f1b91fb73e38baa3e49df4 base/alpine.bst
cached d7a8127c809ec9c47fabd2c51928b27e6b61abfe35a0e64ddfd4e072c6de101b base.bst
cached f32fffc5d0aef5d0cd2bb2e6459d9fe7bc234d5c4751db6aebbe2709c292f6c6 libhello.bst
cached b0e11fa99a24ba51811edc3d39e26be2a2563a5d465117930dd3eefe7e5fcc99 hello.bst
buildable 380526eb857eb3c8b8e5646f8a7b2653cb8e8d9bf771ed1297caa2d5242d89d6 runtime-only.bst
===============================================================================
[--:--:--][380526eb][ build:runtime-only.bst ] START overlaps/runtime-only/380526eb-build.4803.log
[--:--:--][380526eb][ build:runtime-only.bst ] START Staging dependencies
[--:--:--][380526eb][ build:runtime-only.bst ] WARNING [overlaps]: Non-whitelisted overlaps detected
Staged files overwrite existing files in staging area:
/usr/share/doc/hello.txt: hello.bst is not permitted to overlap other elements, order hello.bst above libhello.bst
[00:00:00][380526eb][ build:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][380526eb][ build:runtime-only.bst ] START Computing split
[00:00:00][380526eb][ build:runtime-only.bst ] SUCCESS Computing split
[--:--:--][380526eb][ build:runtime-only.bst ] START Integrating sandbox
[--:--:--][380526eb][ build:runtime-only.bst ] START Running commands
ldconfig "/usr/lib"
[00:00:01][380526eb][ build:runtime-only.bst ] SUCCESS Running commands
[--:--:--][380526eb][ build:runtime-only.bst ] INFO Integration modified 0, added 6 and removed 0 files
[00:00:01][380526eb][ build:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][380526eb][ build:runtime-only.bst ] START Creating composition
Including files from domains: runtime
Excluding orphaned files
[--:--:--][380526eb][ build:runtime-only.bst ] INFO Composing 293 files
[00:00:00][380526eb][ build:runtime-only.bst ] SUCCESS Creating composition
[--:--:--][380526eb][ build:runtime-only.bst ] START Caching artifact
[00:00:00][380526eb][ build:runtime-only.bst ] SUCCESS Caching artifact
[00:00:01][380526eb][ build:runtime-only.bst ] SUCCESS overlaps/runtime-only/380526eb-build.4803.log
[00:00:02][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 5
Session: 1
Fetch Queue: processed 0, skipped 1, failed 0
Build Queue: processed 1, skipped 0, failed 0
Notice the warning message about the conflicting file, it is there to inform the user about which files are overlapping, and also which elements are being staged in which order.
Note also that BuildStream does not discover the overlap until the moment that you build a reverse dependency which will require staging of both artifacts.
Tip
The overlaps warning discussed here can be configured to be
a fatal warning. This is useful
in the case that you want to be strict about avoiding overlapping
files in your project.
3.4. Mitigating overlapping files¶
Since we recently discussed filtering of artifacts,
we should note that it is of course possible to handle this case by
having hello.bst depend on a filtered version of libhello.bst with the
offending file excluded.
However, working with filter elements just for the
sake of handling a conflicting artifact would be quite inconvenient, so we
have other means.
3.4.1. Whitelisting overlapping files¶
BuildStream allows explicitly ignoring such errors by adding the files
to an overlap whitelist, you could achieve
this in the given example by adding the following to the hello.bst element:
public:
bst:
overlap-whitelist:
- |
%{docdir}/hello.txt
Note
Note that glob patterns are also
supported in the whitelist.
3.4.2. Artifact munging¶
Another way around this situation is the munge the artifacts at install time such that there is no conflict.
This is the easiest approach in the case that you might want to keep the
underlying %{docdir}/hello.txt from libhello.bst and discard the
same file from hello.bst.
In this case, we might modify the hello.bst file so that it’s install
command contain an rm statement, as such:
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
- |
# Rid ourselves of the unwanted file at install time
rm -f %{install-root}%{docdir}/hello.txt
This would cause later builds of runtime-only.bst to no longer
conflict on the given file.
3.5. Summary¶
In this chapter we’ve presented a situation where an artifact can conflict with another artifact by way of providing the same files.
We’ve presented the overlap whitelist public data which is the typical solution for silencing the error when the outcome is desired, and also presented a strategy to deal with cases where you want to keep files from the overlapped artifact instead.