DownloadableFileSource - Abstract class for sources downloaded from a URI

This DownloadableFileSource class is a convenience class on can derive for implementing sources that get downloaded from a URI.

It provides utilities around handling mirrors, tracking and fetching the source.

Any derived classes must write their own stage() and get_unique_key() implementation.

class DownloadableFileSource(context: Context, project: Project, meta: buildstream._loader.metasource.MetaSource, variables: buildstream._variables.Variables, *, alias_override: Optional[Tuple[str, str]] = None, unique_id: Optional[int] = None)

Bases: buildstream.source.Source

COMMON_CONFIG_KEYS = ['kind', 'directory', 'url', 'ref', 'etag']
configure(node)

Configure the Plugin from loaded configuration data

Parameters

node – The loaded configuration dictionary

Raises

Plugin implementors should implement this method to read configuration data and store it.

The MappingNode.validate_keys() method should be used to ensure that the user has not specified keys in node which are unsupported by the plugin.

preflight()

Preflight Check

Raises

This method is run after Plugin.configure() and after the pipeline is fully constructed.

Implementors should simply raise SourceError or ElementError with an informative message in the case that the host environment is unsuitable for operation.

Plugins which require host tools (only sources usually) should obtain them with utils.get_host_tool() which will raise an error automatically informing the user that a host tool is needed.

get_unique_key()

Return something which uniquely identifies the plugin input

Returns

A string, list or dictionary which uniquely identifies the input

This is used to construct unique cache keys for elements and sources, sources should return something which uniquely identifies the payload, such as an sha256 sum of a tarball content.

Elements and Sources should implement this by collecting any configurations which could possibly affect the output and return a dictionary of these settings.

For Sources, this is guaranteed to only be called if Source.is_resolved() has returned True which is to say that the Source is expected to have an exact ref indicating exactly what source is going to be staged.

is_cached() → bool

Get whether the source has a local copy of its data.

This method is guaranteed to only be called whenever Source.is_resolved() returns True.

Returns: whether the source is cached locally or not.

load_ref(node)

Loads the ref for this Source from the specified node.

Parameters

node – The YAML node to load the ref from

Note

The ref for the Source is expected to be read at Plugin.configure() time, this will only be used for loading refs from alternative locations than in the element.bst file where the given Source object has been declared.

get_ref()

Fetch the internal ref, however it is represented

Returns

The internal source reference, or None

Return type

(simple object)

Note

The reference is the user provided (or track resolved) value the plugin uses to represent a specific input, like a commit in a VCS or a tarball’s checksum. Usually the reference is a string, but the plugin may choose to represent it with a tuple or such.

Implementations must return a None value in the case that the ref was not loaded. E.g. a (None, None) tuple is not acceptable.

set_ref(ref, node)

Applies the internal ref, however it is represented

Parameters

  • ref (simple object) – The internal source reference to set, or None

  • node – The same dictionary which was previously passed to Plugin.configure()

See Source.get_ref() for a discussion on the ref parameter.

Note

Implementors must support the special None value here to allow clearing any existing ref.

track()

Resolve a new ref from the plugin’s track option

Parameters

previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when BST_REQUIRES_PREVIOUS_SOURCES_TRACK is set to True.

Returns

A new internal source reference, or None

Return type

(simple object)

If the backend in question supports resolving references from a symbolic tracking branch or tag, then this should be implemented to perform this task on behalf of bst source track commands.

This usually requires fetching new content from a remote origin to see if a new ref has appeared for your branch or tag. If the backend store allows one to query for a new ref from a symbolic tracking data without downloading then that is desirable.

See Source.get_ref() for a discussion on the ref parameter.

fetch()

Fetch remote sources and mirror them locally, ensuring at least that the specific reference is cached locally.

Parameters

previous_sources_dir (str) – directory where previous sources are staged. Note that this keyword argument is available only when BST_REQUIRES_PREVIOUS_SOURCES_FETCH is set to True.

Raises

SourceError –

Implementors should raise SourceError if the there is some network error or if the source reference could not be matched.