FreeBSD Ports sccache integration

Posted on 2022-02-08

Back to index

💬 Introduction

Sccache is a ccache-like compiler caching tool. It can be used as a wrapper for rustc and other compilers which caches their output and reuses the cached results when called again. This can significantly speed up builds.

sccache-overlay in action: sccache statistics after repeated sysutils/hexyl builds
sccache-overlay in action: sccache statistics after repeated sysutils/hexyl builds

The Rust toolchain natively supports wrappers via the RUSTC_WRAPPER environment variable. RUSTC_WRAPPER is used to wrap rustc as well when cargo (or rather the cc crate) calls C or C++ compilers.

Unfortunately the ports framework did not yet support Sccache. tcberner@ tried to add it to the ports framework in D31243 but left the inherent bootstrapping problem open.

🌍https://reviews.freebsd.org/D31243

I describe the current solutions to these problems below.

↺ Provisioning Sccache

The provisioning problem can be solved by providing the binary via an overlay. Poudriere (poudriere-devel) supports overlays via the -O option. Poudriere will provision the overlays to build jails into /overlays. Non-Poudriere builds can support it via the OVERLAYS variable.

However since the binary is compiled for the host it won't just work as is in random Poudriere jails which might be a non-native one or for older FreeBSD versions than the host's. We could provide an sccache binary for each architecture and version combination but this quickly becomes too complicated. All we want is to be able to run the native binary inside the jails. This is easy enough to solve if we could provide a static binary that the host kernel can run.

Build Sccache statically

It does not seem possible at the moment (or I was unable to figure it out) to build Rust binaries that link with libc statically on FreeBSD. The "solution" I came up with is to build a dynamic binary and provide all required libraries and a copy of the host's ld-elf.so as well.

Since we cannot use the jail's native ld-elf.so for this. We build the binary with

LDFLAGS+=       -Wl,-rpath=\$$ORIGIN/../lib \
                -Wl,-dynamic-linker,/tmp/sccache-overlay/ld-elf.so.1

The binary's libraries are placed inside ../lib based on the binary location (using $ORIGIN). $ORIGIN is not supported for -dynamic-linker. It must be an absolute path. Since /tmp is writable inside Poudriere we can put the linker inside /tmp/sccache-overlay:

…
sccache-start:
…
        @${LN} -Fs ${_SCCACHE_LIBS} /tmp/sccache-overlay

FreeBSD's ldd has some nice options to get a list of all required libraries:

$ ldd -f '%p\n' /usr/local/share/sccache/overlay/bin/sccache
/lib/libthr.so.3
/lib/libgcc_s.so.1
/lib/libc.so.7

/libexec/ld-elf.so.1 needs to be added to the list too.

❓ Where do we cache the compilation results to?

Networking is largely disabled during Poudriere builds so we cannot use Sccache's Redis or similar support to cache to the network. This limits us to local storage.

Poudriere has Ccache support and uses nullfs(5) to mount its cache directory inside Poudriere jails. Since changing Poudriere for this is out of the question, we simply piggy back on the ccache directory and put sccache's directory inside ${CCACHE_DIR}/sccache as per tcberner@'s original patch.

⇒ ports-mgmt/sccache-overlay

I tied it all together in ports-mgmt/sccache-overlay.

🌍ports-mgmt/sccache-overlay in Git

Users can easily set it up with

pkg install sccache-overlay

and by following a couple of some simple instructions.

Local setup

For local non-Poudriere builds add this to /etc/make.conf:

SCCACHE_DIR=    ${HOME}/.sccache
OVERLAYS+=      /usr/local/share/sccache/overlay

Poudriere setup

At the moment overlays support is only available in poudriere-devel.

For Poudriere builds the setup is slightly more complicated.

First we need to enable Poudriere's ccache support. This involves setting CCACHE_DIR in poudriere.conf. It is described elsewhere in detail. The sccache overlay sets up SCCACHE_DIR to write into CCACHE_DIR.

Next we need to make the overlay available to Poudriere.

$ poudriere ports -c -p sccache -m null -M /usr/local/share/sccache/overlay

Builds using sccache can then be started with:

$ poudriere bulk -O sccache ...

This document is also available on Gemini: gemini://tobik.me/2022/02/08/sccache.gmi