# Caching Built Files

On multi-developer software projects, you can sometimes speed up every developer's builds a lot by allowing them to share a cache of the derived files that they build. After all, it is relatively rare that any in-progress change affects more than a few derived files, most will be unchanged. Using a cache can also help an individual developer: for example if you wish to start work on a new feature in a clean tree, those build artifacts which could be reused can be retrieved from the cache to populate the tree and save a lot of initial build time. &SCons; makes this easy and reliable.

## Specifying the Derived-File Cache Directory

To enable caching of derived files, use the &f-link-CacheDir; function in any &SConscript; file:

env = Environment() env.Program('hello.c') CacheDir('cache')
hello.c
CacheDir('/usr/local/build_cache')

The cache directory you specify must have read and write access for all developers who will be accessing the cached files (if `--cache-readonly` is used, only read access is required). It should also be in some central location that all builds will be able to access. In environments where developers are using separate systems (like individual workstations) for builds, this directory would typically be on a shared or NFS-mounted file system. While &SCons; will create the specified cache directory as needed, in this multi user scenario it is usually best to create it ahead of time so the access rights can be set up correctly.

Here's what happens: When a build has a &CacheDir; specified, every time a file is built, it is stored in that cache directory indexed by its &buildsig;. On subsequent builds, before an action is invoked to build a file, the &buildsig; is computed and &SCons; checks the derived-file cache directory to see if a file with the exact same &buildsig; already exists.

A few inside details: &SCons; tracks two main kinds of cryptographic hashes: a &contentsig;, which is a hash of the contents of a file participating in the build (depepdencies as well as targets); and a &buildsig;, which is a hash of the elements needed to build a target, such as the command line, the contents of the sources, and possibly information about tools used in the build. The hash function produces a unique signature from its inputs, no other set of inputs can produce that same signature. The &buildsig; from building a target is used as the filename of the target file in the derived-file cache - that way lookups are efficient, just compute a &buildsig; and see if a file exists with that as the name. The use of the &buildsig; provides protection from concflicts: if two developers have different setups, so they would produce built objects that are not identical, then because the difference in tools will show up in the &buildsig;, which is used as the name of the cache entry, they will end up being stored as separate entries.

If so, the derived file will not be built locally, but will be copied into the local build directory from the derived-file cache directory, like this:

scons -Q
scons -Q -c
scons -Q

Note that the &CacheDir; feature requires that the &buildsig; be calculated, even if you configure &SCons; to use timestamps to decide if files are up to date (see the chapter for information about the &f-link-Decider; function), since the &buildsig; is used to determine if a target file exists in the cache. Consequently, using &CacheDir; may reduce or negate any performance improvements from using timestamps for up-to-date decisions.

## Keeping Build Output Consistent

One potential drawback to using a derived-file cache is that the output printed by &SCons; can be inconsistent from invocation to invocation, because any given file may be rebuilt one time and retrieved from the derived-file cache the next time. This can make analyzing build output more difficult, especially for automated scripts that expect consistent output each time.

If, however, you use the `--cache-show` option, &SCons; will print the command line that it would have executed to build the file, even when it is retrieving the file from the derived-file cache. This keeps the build output consistent across builds:

scons -Q
scons -Q -c
scons -Q --cache-show

The trade-off, of course, is that you no longer know whether or not &SCons; has retrieved a derived file from cache or has rebuilt it locally.

## Not Using the Derived-File Cache for Specific Files

You may want to disable caching for certain specific files in your configuration. For example, if you only want to put executable files in a central cache, but not the intermediate object files, you can use the &f-link-NoCache; function to specify that the object files should not be cached:

env = Environment() obj = env.Object('hello.c') env.Program('hello.c') CacheDir('cache') NoCache('hello.o')
hello.c

Then when you run &scons; after cleaning the built targets, it will recompile the object file locally (since it doesn't exist in the derived-file cache directory), but still realize that the derived-file cache directory contains an up-to-date executable program that can be retrieved instead of re-linking:

    % scons -Q
    cc -o hello.o -c hello.c
    cc -o hello hello.o
    % scons -Q -c
    Removed hello.o
    Removed hello
    % scons -Q
    cc -o hello.o -c hello.c
    Retrieved `hello' from cache

## Disabling the Derived-File Cache

Retrieving an already-built file from the derived-file cache is usually a significant time-savings over rebuilding the file, but how much of a savings (or even whether it saves time at all) can depend a great deal on your system or network configuration. For example, retrieving cached files from a busy server over a busy network might end up being slower than rebuilding the files locally.

In these cases, you can specify the `--cache-disable` command-line option to tell &SCons; to not retrieve already-built files from the derived-file cache directory:

scons -Q
scons -Q -c
scons -Q
scons -Q -c
scons -Q --cache-disable

## Populating a Derived-File Cache With Already-Built Files

Sometimes, you may have one or more derived files already built in your local build tree that you wish to make available to other people doing builds. For example, you may find it more effective to perform integration builds with the cache disabled (per the previous section) and only populate the derived-file cache directory with the built files after the integration build has completed successfully. This way, the cache will only get filled up with derived files that are part of a complete, successful build not with files that might be later overwritten while you debug integration problems.

In this case, you can use the the `--cache-force` option to tell &SCons; to put all derived files in the cache, even if the files already exist in your local tree from having been built by a previous invocation:

scons -Q --cache-disable
scons -Q -c
scons -Q --cache-disable
scons -Q --cache-force
scons -Q

Notice how the above sample run demonstrates that the `--cache-disable` option avoids putting the built `hello.o` and `hello` files in the cache, but after using the `--cache-force` option, the files have been put in the cache for the next invocation to retrieve.

## Minimizing Cache Contention: the `--random` Option

If you allow multiple builds to update the derived-file cache directory simultaneously, two builds that occur at the same time can sometimes start "racing" with one another to build the same files in the same order. If, for example, you are linking multiple files into an executable program:

Program('prog', ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])
f1.c
f2.c
f3.c
f4.c
f5.c
f6.c

&SCons; will normally build the input object files on which the program depends in their normal, sorted order:

scons -Q

But if two such builds take place simultaneously, they may each look in the cache at nearly the same time and both decide that `f1.o` must be rebuilt and pushed into the derived-file cache directory, then both decide that `f2.o` must be rebuilt (and pushed into the derived-file cache directory), then both decide that `f3.o` must be rebuilt... This won't cause any actual build problems--both builds will succeed, generate correct output files, and populate the cache--but it does represent wasted effort.

To alleviate such contention for the cache, you can use the `--random` command-line option to tell &SCons; to build dependencies in a random order:

      % scons -Q --random
      cc -o f3.o -c f3.c
      cc -o f1.o -c f1.c
      cc -o f5.o -c f5.c
      cc -o f2.o -c f2.c
      cc -o f4.o -c f4.c
      cc -o prog f1.o f2.o f3.o f4.o f5.o

Multiple builds using the `--random` option will usually build their dependencies in different, random orders, which minimizes the chances for a lot of contention for same-named files in the derived-file cache directory. Multiple simultaneous builds might still race to try to build the same target file on occasion, but long sequences of inefficient contention should be rare.

Note, of course, the `--random` option will cause the output that &SCons; prints to be inconsistent from invocation to invocation, which may be an issue when trying to compare output from different build runs.

If you want to make sure dependencies will be built in a random order without having to specify the `--random` on very command line, you can use the &f-link-SetOption; function to set the `random` option within any &SConscript; file:

SetOption('random', 1) Program('prog', ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])
f1.c
f2.c
f3.c
f4.c
f5.c
f6.c

## Using a Custom CacheDir Class

&SCons;' internal `CacheDir` class can be extended to support customization around the details of caching behaviors, for example using compressed cache files, encrypted cache files, gathering statistics and data, or many other aspects.

To create your own custom cache class, your custom class must be a subclass of the `SCons.CacheDir.CacheDir` class. You can then pass your custom class to the &f-link-CacheDir; method or set the &consvar; &cv-link-CACHEDIR_CLASS; to the class before configuring the cache in that environment. SCons will internally invoke and use your custom class when performing cache operations. The below example shows a simple use case of overriding the `copy_from_cache` method to record the total number of bytes pulled from the cache.

import SCons import os

class CustomCacheDir(SCons.CacheDir.CacheDir): total_retrieved = 0

@classmethod def copy_from_cache(cls, env, src, dst): # record total bytes pulled from cache cls.total_retrieved += os.stat(src).st_size super().copy_from_cache(env, src, dst)

env = Environment() env.CacheDir('scons-cache', CustomCacheDir) # ...