Right Pool, Right Workload: How Every Database We Ran Was Quietly Paying a 17× Storage Tax

A Warning That Wasn't What It Looked Like

It started with a persistent slow-operations warning on the SSD storage tier. The cluster was fully healthy — every storage daemon up, every placement group active and clean, no data at risk — but commit and apply tail latency was spiking into the hundreds and occasionally thousands of milliseconds. Classified correctly, this was not an incident. It was a design risk to fix soon. Digging in turned a latency symptom into an architecture lesson.

The Root Cause: Erasure Coding Under Database fsyncs

Every production database's data lived on a 3+1 erasure-coded pool. Erasure coding is wonderful for large sequential objects and cold capacity — it gives you redundancy at a fraction of replication's storage cost. It is close to the worst possible choice for a database write pattern.

The reason is the partial-stripe read-modify-write. A database's durability comes from small synchronous writes — the write-ahead log, the append-only file, the transaction log — fsync'd constantly. On a 3+1 erasure-coded pool, a small write that touches only part of a stripe forces the system to *read* the rest of the stripe, recompute parity, and write it back. Every fsync pays that tax. The symptom presents as storage latency; the cause is a workload-to-pool mismatch.

Quantify It Before You Act

Rather than argue from theory, we benchmarked it. A 4K random-write test across pools, on the same cluster:

End to end, the new replicated-NVMe pool was about 17× faster than the erasure-coded pool the databases were on. Isolating the erasure-coding penalty alone — replicated versus EC on *identical* SSD media — accounts for roughly 12× of that. The lesson crystallises into a rule: erasure coding is the wrong tool for small synchronous random writes. Use replication for databases; save erasure coding for bulk and cold data.

Build the Right Tier, De-Risk the Cutover

The fix was a *new* replicated-NVMe pool (three-way replication, its own placement rule and StorageClass) rather than reusing a pre-existing erasure-coded NVMe pool that had been sitting idle — additive and fully reversible. Two things made the migration safe rather than scary:

A cephx footgun worth knowing. The storage CSI user has *per-pool* capabilities. Add a new pool and forget to extend that capability string, and volume mapping fails with a permission error even though pool creation succeeded. Two sub-traps cost real debugging time: the capability string must stay on a single line (a wrapped paste embeds a newline that silently corrupts capability evaluation), and authentication tickets cache capabilities for about an hour, so you must restart the CSI pods or the fix looks like it didn't work.

Canary, then everything, with rollback retained. A single low-stakes service was migrated first and verified end to end before anything critical was touched. Then all seven stateful workloads were moved and individually verified — a relational database recovered its write-ahead log cleanly, three message-queue volumes came back healthy, and the largest, a 473.6 GB vector-embedding database, was copied byte-exact. That last one was driven by hand rather than the automated mover, because the mover's 30-minute wait would have prematurely cut over to a copy that was only partway through a ~75-minute transfer — a silent truncation of live data avoided only by watching it. Every old volume was kept on a retain policy for rollback. Zero data loss; every database write now bypasses the read-modify-write penalty.

The Cautionary Sidebar

There is a coda. The same erasure-coded geometry that throttled the databases bit back operationally a few weeks later. The EC pools used a k=3, m=1 profile across exactly four hosts — meaning the minimum-size equals k, with zero spare host. A textbook-careful, fully-flagged graceful sleep of a *single* storage host dropped the pools to minimum-size under live client I/O; the reconstruction load starved the daemons' heartbeats, a daemon was marked down, placement groups fell below minimum size, I/O blocked, a kernel hang-task watchdog fired, and three storage hosts rebooted. The cluster self-healed to full health with no data loss — but the lesson is structural: setting maintenance flags is necessary and not sufficient when your erasure-coding width leaves no margin. The real fix is a fifth host, not a flag.

Right pool, right workload — and enough margin that one host going away is a non-event.