↓Skip to main content
  1. Posts/

GC Pauses and Latency: Where the Time Actually Hides

·2553 words·12 mins·
Go Kubernetes Containers Garbage-Collection Latency
Dave Amit
Author
Dave Amit
Principal Architect with nearly two decades designing distributed systems and cloud-native platforms. I bring deep systems expertise, hands-on AI-assisted engineering workflows, and a track record of shipping things that actually scale. Currently writing Go and Rust, running on Kubernetes, and exploring what happens when strong architectural judgment meets modern AI tooling.
Table of Contents
Go in Containers - This article is part of a series.
Part 3: This Article

Here’s a Go program’s entire stop-the-world budget for churning through 2 GB of garbage on my laptop — about 43 collection cycles, every pause accounted for, straight from runtime/metrics (/sched/pauses/total/gc:seconds):

gc cycles: 43
pauses p50=0.0205ms p90=0.0287ms p99=0.197ms max=0.197ms

Twenty microseconds median (7 to 20 across five runs), and the worst pause in any of the five runs stayed under 200 microseconds. That’s what Go’s two stop-the-world phases cost on a healthy box in go1.27.3, and it’s worth staring at, because the folklore about “millisecond GC pauses” is off by a factor of fifty at the median — in Go’s favor.

Now the same collector, same machine, inside a container with --cpus=0.5:

GOMAXPROCS=2 NumCPU=6
gc cycles: 458
pauses p50=0.0573ms p90=0.131ms p99=83.9ms max=83.9ms

The median is still microseconds. The tail is about fifteen hundred times the median. One number in this trace is doing all the screaming, and this post is about where it comes from, how to see it, and what actually fixes it — with every number from real runs, same harness style as part 2.

Test environment. go1.27.3 on an Apple M4 Pro (12 cores, 24 GB, macOS 27.0.1) for the “my laptop” and benchmark runs. The container runs use static linux/arm64 binaries on alpine in a podman 6.1.4 VM (6 CPUs, 2 GiB, cgroup v2), so they show how quota throttling behaves in a Linux VM, not how Mac hardware behaves. I first measured this on go1.26.4 in July and re-ran everything on go1.27.3 in November; every number below is from the re-run, and where the two disagreed, the re-run wins.

The anatomy of a pause, with real names
#

Go’s GC stops the world exactly twice per cycle, and the runtime documentation names the phases precisely: “stop-the-world (STW) sweep termination, concurrent mark and scan, and STW mark termination.” (You’ll meet these names again in gctrace output, so learn them as the runtime spells them — the first pause flips the write barrier on, the second closes out the mark.)

Both phases are designed to be tiny, and the healthy-box histogram above shows the design working. But an STW pause has two components: the work, and the wait to be scheduled to do the work. The GC guide’s latency section lists five ways GC costs you latency — brief STW pauses are only the first, and on a healthy machine none of the five hurt.

In a container, the second component stops being negligible. The CFS quota is an average: --cpus=0.5 means 50ms of CPU time per 100ms period (kernel cgroup-v2 docs, cpu.max). Burn the budget early and the kernel freezes every thread in the cgroup until the next period. If that freeze lands while the runtime is inside an STW window — goroutines stopped, world halted, cleanup half-done — the pause is no longer “the work.” It’s the work plus the remainder of the CFS period.

The multiplication, on tape
#

The harness: busy-loop goroutines burning CPU (one per P) plus a 1 MiB/iteration allocation churn, 20 seconds, in a --cpus=0.5 container. Two configurations, one variable, three runs each, plus a control run twice.

Run A — Go 1.25+ defaults. The runtime reads the cgroup quota (part 1’s story) and picks its floor:

GOMAXPROCS=2 NumCPU=6
gc cycles: 458
pauses p50=0.0573ms p90=0.131ms p99=83.9ms max=83.9ms

Across the three runs: median 57µs every time, p99 84ms every time, worst pause 84ms in one run and 101ms in the other two.

Run B — GOMAXPROCS=4 forced, simulating the pre-1.25 world or any of the opt-outs part 1 catalogued:

GOMAXPROCS=4 NumCPU=6
gc cycles: 555
pauses p50=0.115ms p90=0.262ms p99=67.1ms max=101ms

Across the three runs: median 115µs every time, worst pause 101ms every time.

Those 84 and 101 ms values are bucket bounds of the runtime’s pause histogram, and they sit right around one 100 ms CFS period. That fits the mechanism: if the quota runs dry while the runtime is inside a stop-the-world window, the pause absorbs the rest of the kernel freeze. I didn’t trace individual pauses, so treat that as the explanation that fits, not a photograph. What the counters do confirm is that the cgroup was throttled in nearly every period: nr_throttled was 195 to 201 out of 200 to 201 periods in all six runs. (The spinners guarantee that, so this shows the throttling was constant, not that it caused any particular pause.)

The control — same binary, same spinners, no quota:

GOMAXPROCS=6 NumCPU=6
gc cycles: 6234
pauses p50=0.0819ms p90=0.115ms p99=0.164ms max=0.393ms
nr_throttled 0

Remove the quota and the worst pause drops to under 0.4 ms (0.393 and 0.328 in the two runs), with nr_throttled at zero. One caveat: with no quota the runtime sees six CPUs, so this run has six Ps and six spinners against Run A’s two. So the claim is “remove the quota and the tail disappears, even with more Ps”, not a like-for-like P count.

Run A does not say the 1.25 defaults save you. In these runs the medians didn’t track the number of Ps (57µs at 2 Ps, 115µs at 4, 82µs in the 6-P no-quota control), and the worst pause landed in the same 84 to 101 ms range for both throttled configurations. The p50 tells you your GC is healthy. The p99 here tells you about your kernel. When p99/p50 is three orders of magnitude, you usually don’t have a GC problem — you have a scheduling problem wearing GC’s clothes.

Where the latency hides when it’s not hiding in pauses
#

The GC guide’s full list of latency sources, and this is the part most posts skip: STW pauses are item one of five. The others — the GC taking 25% of CPU during mark, user goroutines drafted into mark assists when allocation outpaces the collector, write-barrier overhead, and root-scanning delays for individual goroutines — cost you latency between the pauses, and under a CPU quota they cost you double: every CPU-second the GC spends is a CPU-second closer to the throttle.

That’s the invoice for part 2’s central trade, so let’s pay it explicitly. The GOMEMLIMIT rescue turned 7 collections into 22; the overshoot run collected 247 times in 0.36 seconds. Each of those extra cycles is ~25% CPU during its mark phase plus two STW windows plus assist pressure on your hottest allocators — all of it billed against the same quota your request handlers need. GC frequency is latency you pay somewhere else, and inside a container “somewhere else” is the CFS meter. When the meter runs out mid-pause, you get the 84 to 101 ms tail.

Assists deserve one more sentence, because they’re the sneakiest of the five: when allocation outruns marking, the runtime makes the allocating goroutine do GC work at the allocation site. Your P99 spike may be a request goroutine paying a GC tax mid-request — invisible in pause histograms, visible only in an execution trace as MARK ASSIST slices.

Green Tea: the new arithmetic
#

Part 2 promised this. Go 1.26 made Green Tea the default garbage collector (release notes: “we expect somewhere between a 10–40% reduction in garbage collection overhead in real-world programs”), and you can still A/B it against the old collector with GOEXPERIMENT=nogreenteagc at build time. That escape hatch was slated to go in 1.27, and go1.27.3 still builds with it, so treat it as temporary.

The workload where it shines is exactly the one this series has been beating on: pointer-dense heaps. One million *int64 held live, ten forced cycles, both collectors, run natively on the laptop (no container), five runs each:

--- go1.27.3 default (Green Tea):
mode=ptrs n=1000000  wall/cycle=1.30ms  gc-cpu/cycle=4.93ms
  pauses: count=20 max<=0.0205ms
--- GOEXPERIMENT=nogreenteagc:
mode=ptrs n=1000000  wall/cycle=5.80ms  gc-cpu/cycle=5.81ms
  pauses: count=20 max<=0.0205ms

Read the lines that matter. Wall time per cycle: 5.0–5.9ms → 1.2–1.3ms across the five runs, roughly four times faster. CPU per cycle only drops by about 15 percent (4.58–4.96ms against 5.34–6.24ms), so most of the wall-time gain is the same work finishing sooner; Green Tea’s design scans memory in contiguous blocks, and I didn’t isolate which part of that produces the speedup. STW pauses: unchanged — about 20 microseconds or less under both collectors.

One honest wrinkle: inside the podman VM, the classic collector’s numbers were all over the place (4.5–5.1ms in two runs, 17–28ms in six), while Green Tea stayed at 1.3–1.5ms. Native macOS was stable, so that’s what I quote. I haven’t established why the VM varied.

That’s the honest headline: Green Tea changes the throughput arithmetic, not the pause arithmetic. Your pauses were already microseconds; they still are. What shrinks is the concurrent mark phase — the 25%-CPU window, the assist pressure, the quota burn. In a container, that means fewer CPU-seconds on the CFS meter per cycle, which, by my inference (I didn’t measure it), means fewer chances for the kernel to freeze you mid-anything. The multiplication effect gets less fuel, and part 2’s frequency tax gets 10–40% cheaper without touching a knob.

And pointer density is still your biggest lever
#

Same benchmark, Green Tea, one variable — a million int64 versus a million *int64:

mode=ints n=1000000  wall/cycle=0.00ms  gc-cpu/cycle=0.20ms
mode=ptrs n=1000000  wall/cycle=1.30ms  gc-cpu/cycle=4.93ms

About twenty-four times the GC CPU for the same million values, because a slice of a million integers is one object with no pointers — the GC guide is explicit that pointer-free memory is cheap to skip — while a million pointers is a million objects to chase. (Folklore says “1ms vs 100ms scan”; the measured gap on this box is 0.2ms vs about 4.8ms per cycle. Directionally the same lesson, honestly smaller numbers.) If your heap is a pointer thicket — []*Thing, map[string]*Entry, linked structures — flattening it buys more than any GOGC value.

The rest of the allocation-rate toolbox is what it’s always been: sync.Pool for high-churn buffers, value slices over pointer slices, fewer tiny objects per request. Less garbage → fewer cycles → fewer STW windows for the kernel to land a freeze on → less assist pressure. The Green Tea and pointer numbers above measure the CPU side of that chain; the cycle-count and window-landing links I did not measure.

The tuning levers, ranked honestly
#

1. Give the GC less work (allocation rate, pointer density). The only lever that improves every row of the cost table at once. Measured above: about 24× on GC CPU.

2. Give the runtime honest CPU accounting. Part 1’s whole story, now with a latency price tag: forced GOMAXPROCS=4 against a 0.5-CPU quota doubled my median pause (57µs to 115µs), and the worst pause landed in the same 84 to 101 ms range either way. Audit the opt-outs (runtime.GOMAXPROCS(NumCPU) in init code, base-image env vars).

3. Trade memory for frequency — with part 2’s caveat attached. Raising GOGC reduces cycle frequency, and fewer cycles means fewer throttle-exposed windows. But once GOMEMLIMIT’s ceiling is below GOGC’s proportional target, GOGC is dead weight — part 2’s metronome run (every cycle triggering at 388 MB, GOGC’s ~524 MiB wish ignored) is what that looks like. Near the limit, the limit is the pacer; raise GOGC there and nothing changes but your confidence.

4. The Kubernetes lever, split into its actual parts. (a) If throttling lands during GC windows, the quota is too tight for the runtime’s appetite — raise the limit or cut the appetite (levers 1–3). (b) request ≈ limit buys placement predictability — burst capacity above your request depends on node neighbors, so a big gap makes pause tails node-dependent and irreproducible. (c) The “remove CPU limits entirely, keep requests” school trades hard ceilings for scheduler fairness; it genuinely eliminates CFS-period freezes (the no-quota control above: worst pause under 0.4 ms, nr_throttled zero), at the cost of noisy-neighbor exposure. Whichever you choose, watch nr_throttled in the container’s cpu.stat (or container_cpu_cfs_throttled_seconds_total in cAdvisor/Prometheus) — if it’s climbing, your pauses may be being amplified, whatever your dashboard’s average CPU says.

Measuring it yourself
#

  • /sched/pauses/total/gc:seconds (runtime/metrics, Go 1.22+; the older /gc/pauses:seconds still exists but is deprecated — they’re the same histogram, I verified both report identical values). Track p50 and p99: p50 is your GC, p99 is your kernel.
  • GODEBUG=gctrace=1 — the clock columns show wall time per phase; the phase names match the ones above.
  • go tool trace — the only per-goroutine view: MARK ASSIST slices on request goroutines, and STW windows visibly stretching when the scheduler can’t run anything.
  • cat /sys/fs/cgroup/cpu.stat inside the container — nr_throttled and throttled_usec are the kernel’s own confession, and correlating their growth with pause-histogram spikes is the diagnosis. (Part 4 turns this into a full workflow.)

The harness for every number in this post is ~150 lines total — a churn loop (with a 50 MB live ballast for realistic cycle spacing) plus busy-loop spinners for the container runs, and a two-mode slice benchmark for the pointer/Green Tea A/Bs — built with CGO_ENABLED=0 GOOS=linux, run under podman run --cpus=0.5 with plain -e GOMAXPROCS=... as the only variable. Rebuild it from the descriptions above in ten minutes; the shape matters, not the exact numbers — your box will print different ones, and even on mine the throttled tail bounces run to run (p99 between 67ms and 101ms across the six throttled runs, worst pause 84ms or 101ms; the medians stay in the tens to low hundreds of microseconds).

Opinionated Takeaway
#

  • Stop repeating “millisecond pauses.” Healthy go1.27.3 pauses are tens of microseconds — about 20µs median, and never past 200µs in five runs of about 43 cycles on this box. If you’re seeing milliseconds, something is stretching them, and it’s usually not the collector.
  • The tail is usually the kernel, not the GC. In my runs, 84 to 101ms sat right at one CFS period, and removing the quota made it go away (one Linux VM, and the control had more Ps, so check it on your own box). Diagnose p99 pause spikes against cpu.stat’s nr_throttled before touching a single GC knob.
  • Green Tea is a throughput gift, not a latency gift — roughly 4× less wall time per cycle on a pointer-heavy heap (about 15% less CPU), pauses unchanged. Take the 10–40% quota relief and say thank you, but don’t expect your histogram to move.
  • Pointer density is the biggest lever you own — about 24× GC CPU between []int64 and []*int64, measured. Flatten before you tune.
  • GOGC near GOMEMLIMIT is a no-op — part 2’s metronome proved it; don’t budget latency wins from a knob the pacer is ignoring.
  • GOMAXPROCS honesty is a latency feature. In my runs the medians didn’t follow the P count, and only removing the quota removed the tail — and an opt-out somewhere in your dependency tree silently undoes the default.

The GC isn’t a language problem, and inside a container it isn’t even really a memory problem. It’s a resource-interaction problem: the collector spends CPU, the kernel meters CPU, and your P99 lives in the gap between their accounting. You now have the tools to read both ledgers. Next: the debugging workflow that puts them together when production is on fire.


Next in the Go in Containers series: The Container Debugging Workflow — pprof, ephemeral containers, and getting signals out of a scratch image at 3 a.m.


Go in Containers series: Intro — A Silent Crisis · 1. GOMAXPROCS Demystified · 2. GC Under Pressure · 3. GC Pauses and Latency · 4. The Container Debugging Workflow

Go in Containers - This article is part of a series.
Part 3: This Article

Related

Garbage Collection Under Pressure: Memory Limits in Containers
·2246 words·11 mins
Go Kubernetes Containers Garbage-Collection Memory
GOMAXPROCS Demystified
·1953 words·10 mins
Go Kubernetes Containers Scheduler Performance
GitHub Issues as an AI Agent Task Manager
·2087 words·10 mins
Ai Go Automation Github Devops
Stop the Plumbing Hell: Building `bs`
·486 words·3 mins
Go Devops Build-Systems Bs Engineering
Go's Container Problem: A Silent Crisis
·706 words·4 mins
Go Kubernetes Runtime SRE