Setting Up a Raspberry Pi Cluster for Distributed Compile Caching with distcc and ccache

This site's Ceph cluster guide covers turning a stack of Raspberry Pis into distributed storage. This project points a Pi cluster at a different kind of distributed workload: compiling code faster by spreading the actual compilation work across every Pi on the shelf, paired with a compile cache that skips rebuilding anything that hasn't changed. It's a genuinely useful setup for anyone building Klipper firmware for multiple printer boards, cross-compiling for ESP32/embedded targets repeatedly, or just tired of waiting on a single Pi (or even a single desktop) to grind through a large C/C++ build from scratch every time.

Two Different Problems, Two Different Tools

distcc solves the "this build has many files and only one CPU is working on them" problem by farming out individual compilation units to other machines on the network that have distcc's daemon running, while the final link step still happens on your main machine. ccache solves a completely different problem: it caches compiled object files keyed by source content and compiler flags, so recompiling a file you haven't actually changed returns the cached result in milliseconds instead of re-running the compiler. These compose well together — ccache can call out to distcc for anything that isn't a cache hit, giving you both "skip work that doesn't need redoing" and "parallelize whatever work is actually needed" in the same build.

Setting Up distcc Across the Cluster

Install distcc on every Pi in the cluster (sudo apt install distcc), and on each worker node, start the distcc daemon restricted to accept connections only from your trusted local network — distcc's daemon has no built-in authentication, so this is a hard requirement, not a nicety:

sudo systemctl enable --now distcc # Edit /etc/default/distcc on each worker: STARTDISTCC="true" ALLOWEDNETS="192.168.1.0/24" LISTENER="0.0.0.0"

On the machine driving the build (which can itself be one of the Pis, or a more powerful desktop/laptop coordinating the cluster), set the DISTCC_HOSTS environment variable listing every worker, including localhost with a job-count limit matching its own core count, then invoke make with a job count well above your total core count across the cluster (distcc handles scheduling, so slight oversubscription keeps workers from idling while waiting on file transfers):

export DISTCC_HOSTS="localhost/4 192.168.1.101/4 192.168.1.102/4 192.168.1.103/4" make -j16 CC="distcc gcc" CXX="distcc g++" Cluster SizeTypical Use CaseRealistic Speedup vs Single Pi 2-3 Pi 4/5 nodesKlipper builds for a handful of printer boards2-3x on larger builds, less on small ones (network/startup overhead dominates) 4-6 Pi nodesRepeated ESP-IDF or embedded cross-compiles3-5x, with diminishing returns as file-transfer overhead grows Mixed Pi + desktopDesktop drives the build, Pis absorb overflow compile jobsMeaningful for desktop builds that are CPU-bound on many small files

Layering ccache on Top

Install ccache on the driving machine and configure it to call distcc as its "real compiler" rather than calling the compiler directly, which gives you cache hits for unchanged files and distributed compilation for everything else in a single pass:

sudo apt install ccache export CCACHE_PREFIX="distcc" ccache -M 5G # set a cache size appropriate to your available storage make -j16 CC="ccache gcc" CXX="ccache g++"

Point ccache's cache directory at reasonably fast storage — a USB SSD rather than a microSD card if you're doing this regularly, since ccache's performance advantage depends on cache lookups being fast, and a cache living on a slow SD card can partially erode the benefit on very large codebases with many cache entries.

Where This Actually Pays Off

Firmware builds for 3D printer control boards (Klipper's make menuconfig + build cycle, repeated across several different printer configurations), ESP-IDF builds for ESP32 projects iterated on frequently, and any home-lab project involving repeated from-scratch compiles of a moderately large codebase are the sweet spot. A single quick firmware tweak and rebuild on one board won't show a dramatic difference — the overhead of coordinating the cluster has to be smaller than the time saved, which generally means this setup earns its complexity on larger, more frequent build workloads rather than one-off compiles.

Practical Notes

Keep all nodes on the same architecture (ARM Pi nodes compiling for ARM targets, or all cross-compiling to the same embedded target) — distcc distributes compilation assuming the compiler toolchain produces compatible output across nodes, and mixing architectures without matching cross-compilers set up correctly on every node is a common source of confusing, hard-to-diagnose build failures. Firewall the distcc port (3632 by default) at your router if any node has a public-facing connection, since an exposed, unauthenticated distcc daemon is a known way for an attacker to run arbitrary compiled code on your hardware.

Once running, the cluster quietly turns what used to be "go get coffee while this compiles" into a build that finishes before you've sat back down — a genuinely practical use for a handful of otherwise idle Pi boards sitting on a shelf.