Go Concurrency Isn't a Language Problem — It's an Ownership Problem
Every Go service I have audited in the last two years had the same shape: goroutines spawned with a bare go keyword, tracked in a sync.WaitGroup declared three...
Every Go service I have audited in the last two years had the same shape: goroutines spawned with a bare go keyword, tracked in a sync.WaitGroup declared three functions away, and joined by a wg.Wait() that nothing guarantees will ever return. The code compiles. The tests pass. And at 3 AM, the RSS graph is a staircase while runtime.NumGoroutine() climbs past six figures.
Go concurrency is not a language problem. The primitives are correct. The problem is ownership: nothing in the type system, the compiler, or go vet forces a goroutine to declare who is responsible for ending it. Bare go is an unowned resource handle.
This is an argument for making ownership explicit at the call site — so the reader of a diff can see, without leaving the function, that every goroutine has a bounded lifetime.
The Default Pattern Is Brokered By Convention, Not By Compiler
A goroutine runs until its function returns or the process exits. There is no goroutine.Kill(). The runtime does not reclaim a goroutine blocked on a channel, a mutex, or a network read that has no deadline.
That is the entire failure model. A goroutine blocked on a receive from a channel whose sender deferred its close behind an early-return is not a slow goroutine. It is a permanent allocation — 2 KB of stack minimum, plus whatever buffer, connection, or pointer graph it captured.
func (s *Service) refresh(ctx context.Context) {
updates := make(chan State)
go func() {
for st := range updates {
s.apply(st)
}
}()
for {
select {
case st := <-s.incoming:
updates <- st
case <-ctx.Done():
return
}
}
}This looks defensible. It is a leak. When ctx fires, the outer loop returns and updates is never closed. The consumer goroutine parks on range updates forever. Restart refresh on every reconnect and you accumulate one orphan per attempt.
Contrast the same intent with an owned lifecycle:
func (s *Service) refresh(ctx context.Context) error {
updates := make(chan State, 16)
var wg sync.WaitGroup
ctx, cancel := context.WithCancel(ctx)
defer cancel()
wg.Go(func() {
defer close(updates)
for {
select {
case st, ok := <-s.incoming:
if !ok {
return
}
select {
case updates <- st:
case <-ctx.Done():
return
}
case <-ctx.Done():
return
}
}
})
wg.Go(func() {
for st := range updates {
if ctx.Err() != nil {
return
}
s.apply(st)
}
})
wg.Wait()
if err := ctx.Err(); err != nil {
return fmt.Errorf("refresh loop: %w", err)
}
return nil
}Three differences matter. The sender closes the channel, so range terminates. The buffered channel means a shutdown does not require the consumer to be scheduled to unblock the producer. And wg.Go is atomic — the counter increment and the goroutine start cannot be separated by a scheduler pause, so there is no window where Wait returns before the work begins.
Go 1.25 added that WaitGroup.Go method. The ergonomic win is trivial. The correctness win is not: a mismatched Add/Done pair is now impossible to express. If it compiles, the counting is sound.
Structured Concurrency Is The Only Contract A Reviewer Can Check
The property that makes the second version reviewable is that it is structured: the block does not return before every goroutine it started has returned. You can verify that by reading one function. The first version required you to hold the lifetime of a shell goroutine in your head across a file boundary.
For any fan-out where the unit can fail, errgroup is the owned form. It does three things a hand-rolled WaitGroup does not: it propagates the first error, it cancels siblings on that error, and it gives you a context whose lifetime is bounded by the group.
func (r *Resolver) FetchAll(ctx context.Context, ids []string) (map[string]Record, error) {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(16)
mu := sync.Mutex{}
out := make(map[string]Record, len(ids))
for _, id := range ids {
id := id
g.Go(func() error {
rec, err := r.fetch(ctx, id)
if err != nil {
return fmt.Errorf("fetch %s: %w", id, err)
}
mu.Lock()
out[id] = rec
mu.Unlock()
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return out, nil
}Two details are load-bearing. SetLimit(16) blocks the loop once sixteen goroutines are live, which converts an unbounded spawn into backpressure on the producer itself — no semaphore channel, no worker pool plumbing. And the limit number is not aesthetic: it should equal whatever downstream actually constrains you, typically the connection pool size. Picking a round number because it looks tidy is how you turn a healthy service into a self-inflicted 503 generator.
The cost of errgroup is negligible; the cost of not having structured ownership is intermittent leaks, hidden errors, and post-mortems.
That claim is checkable. A bare WaitGroup versus errgroup microbenchmark measures goroutine allocation, not group overhead — the delta is tens of nanoseconds per group on an M-class core, and it disappears entirely the moment doWork touches a socket. If your benchmark shows a 30% gap between the two, you are benchmarking goroutine creation.
Where errgroup is genuinely the wrong tool:
- Long-lived daemons.
Waitblocks until every goroutine returns; a daemon never returns. Use an explicitStart/Stoppair with aCancelFuncand a privateWaitGroup. - Log-and-continue pipelines. First-error semantics are pointless when each stage already handles its own failure. Use channels for data, a
WaitGroupfor the join. - All errors required. Validation over a batch wants every failure, not the first. Collect into a slice under a mutex or use
errors.Join. - Selective cancellation. If the parent needs to stop some children and not others, each child needs its own
context.WithCancel; the group's one-shot lifecycle does not fit.
Two Levers That Look Like Optimizations And Usually Are Not
The first is a worker pool. The second is a mutex. Both have real cost profiles, and both are routinely reached for before measurement says to.
Worker pool versus SetLimit
SetLimit spawns one goroutine per item and blocks the producer when the limit is hit. A worker pool spawns exactly N goroutines and feeds them through a channel.
For a thousand items doing network I/O, they are indistinguishable and SetLimit is four lines shorter. The pool wins when the item count is genuinely enormous and the work is microsecond-scale, because it amortizes goroutine startup instead of paying it per item. On a 10K-item CPU-bound hash benchmark, unbounded spawn measured 5.5x slower than an 8-worker pool; at 100 items the gap was already 3.3x.
But the honest version of that result is uncomfortable: in the same benchmark suite, fan-out/fan-in was slower than plain sequential execution for cheap CPU-bound work. Channel sends cost 300–400 ns against a mutex lock's ~20 ns, and the coordination overhead exceeded any parallelism gain.
// Coordinate only when the unit of work is expensive.
// I/O-bound: fan out. SHA256 of a 1KB buffer: do not.
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(runtime.GOMAXPROCS(0))
for _, item := range items {
item := item
g.Go(func() error { return process(ctx, item) })
}The mutex on the aggregation struct
Reaching for sync.Mutex because channels felt hard swings to the other extreme. Every goroutine serializes on the same lock to append its result. On multi-core hardware that is cache-line bouncing: cores fight over one line and invalidate each other's caches. It compiles, survives staging, and becomes a single-threaded bottleneck the moment concurrency crosses a few dozen.
That is correctness becoming a performance problem at scale. The fix is almost always structural, not a different lock: pre-allocate the result slice and give each goroutine its own index. No mutex, no channel, no contention.
results := make([]Result, len(targets))
g, ctx := errgroup.WithContext(ctx)
for i, t := range targets {
i, t := i, t
g.Go(func() error {
r, err := fetch(ctx, t)
if err != nil {
return err
}
results[i] = r
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}Each goroutine owns a distinct slice element, so there is no shared mutable state and no lock. This is the pattern to reach for before any mutex inside a fan-out.
The Race Detector Does Not Catch The Bug You Actually Shipped
Two failure classes get conflated, and the tooling gap between them is where production incidents live.
A data race is concurrent access to the same memory with at least one write. go test -race finds these, at 5–10x overhead — CI and staging only, never production. A race condition is an unpredictable interleaving that produces a wrong state without any unsynchronized access at all. The race detector is silent on it.
type Account struct {
balance atomic.Int64
}
// No data race. Race detector is clean.
// Still wrong: check-then-act is not atomic.
func (a *Account) Withdraw(amount int64) error {
if a.balance.Load() < amount {
return ErrInsufficientFunds
}
a.balance.Add(-amount)
return nil
}Under concurrency this drives the balance negative. Every individual operation is atomic — composition is not. The fix is a compare-and-swap loop or, when composite, a mutex around the whole precondition.
func (a *Account) Withdraw(amount int64) error {
for {
cur := a.balance.Load()
if cur < amount {
return ErrInsufficientFunds
}
if a.balance.CompareAndSwap(cur, cur-amount) {
return nil
}
}
}This is why "we run -race in CI" is not a concurrency strategy. It catches one of the two classes, and it catches it only on the interleaving the test happened to hit.
Put Leak Detection In The Test Harness
Leak detection belongs in tests, not in production code. A goroutine test that asserts on synchronization handles reads like synchronous code, and the leak check is a deferred assertion.
func TestShutdownJoinsWorkers(t *testing.T) {
defer goleak.VerifyNone(t,
goleak.IgnoreTopFunction("net/http.(*Server).Serve"),
goleak.IgnoreAnyFunction("database/sql.(*DB).connectionOpener"),
)
svc := NewService()
svc.Start()
svc.Stop() // if Stop forgets to join its goroutines, this test fails
}Three constraints to know before wiring this into a suite:
VerifyNoneis incompatible witht.Parallel(): it cannot distinguish a leaked goroutine from a sibling subtest still running. UseVerifyTestMain(m), which runs after everything completes, and accept that it tells you the package leaked, not which test.- The default ignore list already filters runtime, finalizer, and
os/signalgoroutines. Everything else counts. Third-party background goroutines —net/httpkeep-alive readers, gRPC, connection-pool reapers — will accumulateIgnoreTopFunctionentries that reference unexported internals and can silently break across dependency upgrades. - The check is a heuristic with bounded retries, not a proof. A goroutine mid-shutdown when the snapshot fires is retried, not failed.
Go 1.25 shipped testing/synctest, which runs an isolated bubble with a fake clock and no third-party dependency. It catches goroutines blocked on synchronization primitives but cannot see a CPU-bound loop with no exit. Go 1.26 added an experimental GC-based goroutineleak pprof profile, opt-in via GOEXPERIMENT=goroutineleakprofile, planned default-on in 1.27. It only reports goroutines blocked on primitives the GC has marked unreachable — a goroutine parked on a globally reachable channel stays invisible. Pattern review and goleak are still required.
In production, the cheap invariant is a baseline, not a threshold. Establish the goroutine count of a healthy idle instance — HTTP servers, connection pools, and gRPC clients all hold background goroutines by design — then alarm on sustained deviation, not on an absolute number.
Make Ownership The Thing You Change
None of this requires new primitives. Go's concurrency model is not missing features, and the runtime is not going to grow a goroutine.Kill().
The refactor is narrower than it sounds. Run grep -rn "wg.Add(1)" . across the service. Every hit is a candidate. If the goroutine cannot fail, wg.Go. If it can, it probably wanted errgroup the whole time. Where a group has a limit, set it from the real downstream constraint and say so in the constant's name. Where an aggregation struct has a mutex, ask whether index-based writes remove it entirely.
Then replace the concurrency test with a time.Sleep in it with a synctest.Test bubble and attach goleak to the package. A missing exit path becomes a red check in CI instead of a staircase on a dashboard six weeks later.