Build cache eviction: how the size limit works

A self-hosted build cache cannot grow forever. FosterStack Cache has a size cap, and when it is reached it removes the entries used least recently. This page shows exactly what that looks like, from a real 0.2.1 server with a deliberately tiny cap.

Setting the cap

Set FSCACHE_MAX_BYTES, in bytes. The default, 0, means no limit. Real servers use gigabytes; see server sizing for values you can copy. For this test I used 3500 bytes, so that a few small entries fill it:

FSCACHE_MAX_BYTES=3500 ./fscache

What happened

I stored three 1,000-byte entries, read one of them, then stored two more. After each step I looked at /statusz, the status page, and tried to read every key back.

Step	Stored afterwards	Evicted so far	Which keys could be read
Store k1, k2, k3	3,000 bytes, 3 entries	0	all three
Read k1	unchanged	0	k1 is now the most recently used
Store k4 (would go over 3,500)	3,000 bytes, 3 entries	1	k1, k3, k4. k2 is gone.
Store k5	3,000 bytes, 3 entries	2	k3, k4, k5. k1 is gone.

The stored size never went above the cap. Each time something new did not fit, the entry that had gone longest without being used was removed.

Reading counts as using. In the last step k1 went too, even though I had read it earlier. That is because my own checks after storing k4 read k1, k3 and k4 in that order, which made k1 the oldest again. A cache hit keeps an entry alive.

An entry bigger than the whole cap

I also tried storing 4,000 bytes against the 3,500 cap. The server refused it with a 413 and stored nothing, and nothing else was evicted to make room. The other limit, FSCACHE_MAX_BODY_BYTES, is separate and defaults to 1 GiB per entry. See the status codes page.

Watching it on a real cache

The same numbers are in /statusz as store_bytes, max_bytes, store_entries and evicted_entries, and in the Prometheus metrics (fscache_evicted_entries_total and fscache_store_bytes). A rising eviction count together with a hit rate that never climbs means the cap is too small for your build. See monitoring.

Try it yourself

FSCACHE_ADDR=127.0.0.1:8080 FSCACHE_DATA_DIR=./data FSCACHE_MAX_BYTES=3500 ./fscache &
until curl -s localhost:8080/healthz >/dev/null; do sleep 0.2; done   # wait until it is up

put() { head -c 1000 /dev/zero | curl -s -o /dev/null -w "PUT $1 %{http_code}\n" -X PUT --data-binary @- localhost:8080/$1; }
get() { curl -s -o /dev/null -w "$1=%{http_code} " localhost:8080/$1; }
put k1; put k2; put k3
get k1; echo
put k4
for k in k1 k2 k3 k4; do get $k; done; echo
curl -s localhost:8080/statusz | grep -E 'store_bytes|evicted'

# the output of the middle line in my run:
# k1=200 k2=404 k3=200 k4=200

Server sizing FosterStack home