Ceph Distributed Storage for Colocation: Architecture, CRUSH Maps, and Production Sizing

Ceph is a distributed storage system designed for fault tolerance, self-management, and scalability from terabytes to exabytes. Unlike traditional SAN storage with a proprietary controller, Ceph distributes intelligence across all nodes using the CRUSH algorithm (Controlled Replication Under Scalable Hashing), eliminating single points of failure and allowing linear capacity and performance scaling by adding nodes.

For colocation environments running OpenStack, Proxmox, or standalone workloads, Ceph is the standard choice for persistent, highly available storage without the cost of proprietary SAN hardware.

Ceph Component Architecture

• MON (Monitor) — maintains the cluster map (OSD map, CRUSH map, MDS map). Must be deployed in odd numbers (minimum 3) for quorum. Low CPU and memory requirements; dedicated small VMs or nodes are appropriate.
• OSD (Object Storage Daemon) — one OSD process per physical disk (or per partition in BlueStore). OSDs handle data replication, recovery, and rebalancing. These are the data-carrying components.
• MGR (Manager) — provides the dashboard, metrics collection (via Prometheus exporter), and orchestration. Minimum 2 for HA. Active-standby model.
• MDS (Metadata Server) — required only if using CephFS (the POSIX file system interface). Not needed for RBD (block) or RGW (object) use cases.
• RGW (RADOS Gateway) — S3-compatible and Swift-compatible object storage API layer. Stateless; scale horizontally behind a load balancer.

BlueStore: The Modern OSD Backend

BlueStore (introduced in Ceph Luminous, now the default) bypasses the filesystem entirely — OSDs write directly to raw block devices using RocksDB for metadata. Key advantages over the legacy FileStore backend:

• No double-write penalty — FileStore wrote to a journal and then to the OSD filesystem; BlueStore writes once
• Built-in compression and checksum on every object
• Native support for NVMe drives with dramatically lower write amplification
• WAL (Write-Ahead Log) and RocksDB metadata can be placed on a separate fast device (NVMe) while bulk data resides on cheaper SAS HDDs

CRUSH Map Design

The CRUSH algorithm determines where data is placed across OSDs based on a hierarchical map of your infrastructure. By designing the CRUSH map to reflect your physical topology, you ensure that replicas of each data object are placed on failure-independent hardware.

A typical three-tier CRUSH hierarchy for a colocated cluster:

root default
  datacenter pune-dc1
    rack rack-01
      host ceph-node-01  (12 × 4TB SAS HDDs = 12 OSDs)
      host ceph-node-02
      host ceph-node-03
    rack rack-02
      host ceph-node-04
      host ceph-node-05
      host ceph-node-06

With a chooseleaf rule at the host level and replication factor 3 (the default), Ceph guarantees that the three copies of each object reside on three different hosts. If you change the chooseleaf to rack, the three replicas will span three different racks — surviving a complete rack failure.

Networking Design

Ceph uses two distinct network paths:

• Public network — client-to-OSD traffic (RBD I/O, RGW requests, CephFS I/O). Must be reachable by all clients.
• Cluster network — OSD-to-OSD replication and recovery traffic. Should be isolated from the public network; typically a dedicated 10/25 GbE backbone.

Minimum network recommendation for production: 10 GbE public, 25 GbE cluster. For all-NVMe or high-density SSD clusters: 25 GbE public, 100 GbE cluster. Ceph is extremely sensitive to network latency and packet loss on the cluster network — any drops will trigger OSD flapping and recovery cycles.

Production Sizing Guidelines

Erasure Coding vs Replication

The default replication (RF3) stores 3 full copies of every object. Erasure coding (EC) stores k data chunks + m parity chunks, requiring k+m OSDs minimum but achieving better raw storage efficiency at the cost of higher CPU overhead and slower random I/O.

• EC 4+2 (6 OSDs minimum): 1.5× raw-to-usable ratio vs 3× for RF3. ~40% raw storage savings. Write performance 15–25% lower. Good for large objects (S3 objects, cold backup data).
• EC 8+3: 1.375× ratio. 11-node minimum. Best for high-volume archival. Not suitable for random I/O (RBD/block workloads).

Recommendation: use RF3 for block storage (RBD), EC 4+2 or 8+3 for object storage (RGW) when capacity efficiency is important.

"Ceph's operational model requires deeper Linux and networking expertise than proprietary SAN, but the payoff is linear scalability, no vendor lock-in, and storage costs that are 60–80% lower than comparable enterprise SAN."

Sripadatech designs and operates Ceph clusters in colocation environments. Contact us to discuss your storage architecture.