Building a Ceph Distributed Storage Cluster with Raspberry Pi

This site's Raspberry Pi NAS guide covers ZFS and mergerfs on a single Pi — solid choices for one box with a handful of drives. Ceph is a different animal entirely: a distributed storage system designed from the ground up to spread data redundantly across many independent nodes, so that losing any single machine (or even several) doesn't lose data or take storage offline. It's the same technology backing large-scale OpenStack and Proxmox clusters, scaled down here to run across a handful of Raspberry Pis, each with modest attached storage. This is a genuinely different problem than single-box NAS redundancy, and a genuinely different, more advanced skill to have in a homelab — worth it if you're already running (or want to learn) real distributed systems concepts, overkill if you just want reliable file storage for a few TB of data.

Why Ceph Instead of Just RAID or ZFS on One Pi

ZFS and mergerfs, as covered elsewhere on this site, protect against a drive failure within a single Raspberry Pi. They do not protect against the Pi itself dying — a failed SD card corrupting the boot OS, a fried USB controller, a power supply that takes out attached drives, or simply needing to take the box offline for maintenance. Ceph's core design goal is exactly this: data is replicated (or erasure-coded) across multiple independent hosts, so the cluster keeps serving reads and writes even while an entire node is down, and self-heals by re-replicating data once the node returns or is replaced. The tradeoff is real complexity — Ceph has more moving parts, more RAM and network bandwidth appetite, and a steeper learning curve than a single ZFS pool.

Ceph's Core Components

ComponentRole MON (Monitor)Maintains the cluster map and quorum — needs an odd number (3 or 5) for consensus; if a majority of MONs can't reach each other, the cluster stops accepting writes to protect consistency OSD (Object Storage Daemon)One process per physical disk, actually storing and serving data — more OSDs across more nodes generally means better performance and resilience MGR (Manager)Handles cluster metrics, the dashboard, and orchestration modules; typically runs alongside a MON MDS (Metadata Server)Only needed for CephFS (the POSIX filesystem interface) — not required if you only use RBD (block storage) or RGW (S3-compatible object storage)

Hardware Requirements — Be Realistic

Ceph on Raspberry Pi hardware is a real thing people run successfully, but it is not free lunch — this is the one self-hosted workload on this site where undersized hardware doesn't just run slowly, it can genuinely fail to form a stable cluster.

At least 3 nodes minimum, ideally 4-5, for meaningful redundancy and quorum stability. A 2-node Ceph cluster has no real quorum and isn't a supported production pattern.
Pi 4 (4GB) minimum, Pi 5 (8GB) strongly preferred — each OSD process budgets roughly 1-4GB of RAM depending on configuration, and running a MON plus multiple OSDs on an 4GB Pi 4 leaves uncomfortably little headroom.
Wired Gigabit Ethernet is mandatory, not optional — Ceph's inter-node replication traffic is constant and substantial; running this over WiFi is a well-documented path to a cluster that never reaches a healthy state.
Boot and OSD storage on USB SSDs, not SD cards — Ceph's write patterns (constant metadata journaling) will wear out and eventually corrupt an SD card in a way that's a known failure mode in every Pi Ceph writeup; this is non-negotiable for anything meant to stay up.
A dedicated switch for the cluster network is worth considering once you're past 3 nodes — Ceph benefits from low, consistent latency between nodes more than from raw throughput.

Installing Ceph: cephadm vs Rook

There are two realistic paths to a working Ceph cluster on a Pi fleet:

cephadm — Ceph's own official deployment tool, using Podman or Docker containers per daemon, orchestrated over SSH from a bootstrap node. This is the more "native" Ceph experience and matches what most official documentation assumes.
Rook — a Kubernetes operator that deploys and manages Ceph inside a K3s or K8s cluster. If you've already built the Raspberry Pi K3s cluster covered elsewhere on this site, Rook lets Ceph piggyback on that existing Kubernetes control plane rather than standing up a second orchestration layer, which is often the more sensible choice if Kubernetes is already part of your homelab.

For a first cluster with no existing Kubernetes deployment, cephadm is the more direct path: bootstrap the first MON/MGR on one node, then join additional hosts and add OSDs from the Ceph dashboard or CLI as physical drives are attached to each Pi.

Bootstrapping the Cluster

Install a container runtime (Podman, Ceph's preferred choice) on every node, and ensure passwordless SSH from the bootstrap node to every other node — cephadm orchestrates the whole fleet over SSH.
On the first node, run cephadm bootstrap --mon-ip <first-node-ip> — this stands up the initial MON, MGR, and the Ceph dashboard, and prints a generated admin password.
Distribute the cluster's public SSH key (cephadm generates one) to every additional node's authorized_keys, then use ceph orch host add <hostname> <ip> to bring each one into the cluster.
Once nodes are joined, list available raw block devices with ceph orch device ls and add them as OSDs with ceph orch daemon add osd <host>:<device> — each USB SSD attached to each Pi becomes one OSD.
Deploy additional MONs on at least two more nodes (ceph orch apply mon --placement=...) so the cluster has proper quorum before you trust it with real data.

Choosing How You'll Actually Use the Storage

InterfaceWhat it gives youGood for RBD (RADOS Block Device)Virtual block devices, like a network-attached raw diskVM disk images, backing store for a K3s cluster's persistent volumes CephFSA POSIX-compliant shared filesystem, mountable on multiple clients simultaneouslyGeneral-purpose shared file storage, replacing an NFS or Samba share with something self-healing RGW (RADOS Gateway)S3-compatible object storage APIBackup targets, application storage that expects an S3 endpoint (Immich, Restic, and similar tools covered elsewhere on this site can often target it directly)

Monitoring Cluster Health

The single most important habit with Ceph is checking ceph status (or the dashboard) regularly rather than assuming a quiet cluster is a healthy one — Ceph is designed to keep serving data even in a degraded state, which means a node can be down and actively re-replicating data for hours without anything visibly breaking on the client side. A healthy cluster reports HEALTH_OK; HEALTH_WARN or HEALTH_ERR states (a down OSD, insufficient replicas, a full disk) need attention before they compound. Set the replication factor (size) to at least 3 for anything you actually care about losing — the default — since a size-2 pool can leave data unrecoverable if two nodes fail close together.

When This Isn't the Right Tool

Be honest about the workload before building this. Ceph on a handful of Raspberry Pis will never deliver the raw throughput of a single beefy NAS with real drives and a 10Gbps NIC — the value proposition here is resilience and the learning experience of running real distributed storage, not speed. For a home media server or backup target where "one box goes down and I lose access for an hour while I fix it" is an acceptable outcome, the ZFS/mergerfs single-Pi NAS approach covered elsewhere on this site is simpler, cheaper, and easier to reason about. Reach for Ceph specifically when you want storage that survives a node failure without any manual intervention, or when you're deliberately building homelab experience with the same distributed storage technology used in real production clusters.

A Raspberry Pi Ceph cluster is as much an exercise in distributed-systems literacy as it is a practical storage solution — the concepts of quorum, replication, and self-healing translate directly to the same technology running in datacenters, just at a scale a handful of Pis and USB SSDs can actually demonstrate. Start with three nodes, wired Gigabit, USB SSDs instead of SD cards, and a genuine willingness to watch ceph status while you learn what "degraded" and "recovering" actually look like in practice.

Related Guides

Loading_