Separating storage and compute without the cloud: what it takes

Classic warehouses tied storage and compute to the same node. More compute meant more servers, and more servers meant more disks — even when you did not need the space. Separation solves that: data lives in object storage, compute nodes are stateless and can come and go.

Whether this works on your own hardware is not a question of technology but of three building blocks.

Block one: S3-compatible storage

The S3 interface has become the de facto standard, and several systems speak it on your own hardware: Ceph with the RADOS Gateway, MinIO, NetApp StorageGRID, Dell ECS. For a lakehouse three properties matter more than benchmark throughput:

• Atomic key replacement. Without it a snapshot swap is not safe — in that case the catalog takes over that job.
• Read-after-write consistency. Write a file, read it immediately, and it must be there. All the systems named meet this today.
• Behavior with many small objects. A lakehouse's metadata load consists of thousands of small files, not a few large ones.

That last point is almost always overlooked in selection processes. Storage that shines on large sequential files can fall off sharply on metadata access — and metadata access is what dominates in operation.

Block two: an orchestrator that takes nodes away

Adding compute nodes is easy. The trick is getting rid of them again. On Kubernetes that means warehouses run as their own resources, an operator scales them by queue depth, and a node is torn down once it has had nothing to do for a while.

apiVersion: lavalake.io/v1
kind: Warehouse
metadata:
  name: bi-xl
spec:
  size: xl
  minReplicas: 0          # 0, not 1 — otherwise there is no auto-suspend
  maxReplicas: 8
  autoSuspendAfter: 10m
  scaleOn:
    queueDepth: 4
  isolation: dedicated    # do not mix BI load with ETL

Setting minReplicas to 1 gives you no auto-suspend but a permanently running floor. It is the most common reason the expected savings do not materialize.

Block three: caching, because the network does not go away

Separated storage means every query reads over the network. A local NVMe cache on the compute node is the difference between usable and painful. What matters is that the cache is keyed on the snapshot and not on the file name — otherwise it serves stale data after a write.

Where elasticity ends

And here is the part missing from product comparisons: on your own hardware, elasticity is bounded above. With eight nodes you can scale to eight. In the cloud you can scale to eighty — and you will pay for it.

That leads to an honest boundary. If you have sharply varying, rare peaks and almost nothing in between, the cloud serves you better. If you have a high, steady baseline — which is the norm for reporting and controlling — you are paying in the cloud for elasticity you do not use.

The interesting number is therefore not peak load but the ratio of peak to median. Below roughly three, provisioned capacity is usually cheaper. Above ten it flips. That calculation can be answered directly from an existing warehouse's query history — and it is rarely asked.