Why Is My Go Slice Growing Unexpectedly / Causing Memory Bloat?
Two different bugs share this search. Growth: append does not add one slot. When capacity is full it allocates a larger array, often double while the slice is small, so cap jumps. Bloat: a slice header that still points at a large array keeps that entire array alive. len can be 4 while cap is a gigabyte. The GC will not free what you still reference.
The failure
Unexpected zeros, then a fat cap:
package main
import "fmt"
func main() {
s := make([]int, 1000) // 1000 zeros, cap 1000
for i := 0; i < 1000; i++ {
s = append(s, i) // grows past 1000
}
fmt.Println(len(s), cap(s)) // 2000, and cap is even larger
}
You wanted 1000 values. You got 1000 zeros plus 1000 values, and a backing array bigger than that because append overshot.
The quieter leak, a small return that pins a huge buffer:
func firstLine(file []byte) []byte {
for i, b := range file {
if b == '\n' {
return file[:i] // len is small; cap is still len(file)
}
}
return file
}
As long as the caller holds that result, the whole file stays in memory. len lies about how much you pay.
Why this happens
Capacity grows in chunks. The spec does not promise a factor. The runtime typically doubles while the slice is small, then grows by a smaller fraction. That is why ten appends print cap 1, 2, 4, 8, 16. It is cheaper than copying on every insert. It is not "the slice is the size of my data."
A subslice does not shrink the array. file[:i] is a new header. Its pointer still names the original array. Capacity is "from this pointer to the end of that array." The GC sees one live pointer into a 50 MB allocation and keeps all 50 MB.
Delete by reslicing leaves ghosts. s = append(s[:i], s[i+1:]...) shortens len. The last slot of the array may still hold a pointer. That object cannot be collected until you overwrite or nil it, if the array lives on.
s := []*Obj{a, b, c}
s = s[:len(s)-1] // c is gone from len, still in the array if cap > len
The fix
Pre-size when you know the count. Use length 0 and a capacity, or set length and assign by index, do not make a full length and then append the real data.
out := make([]int, 0, len(in))
for _, v := range in {
out = append(out, v*2)
}
// or
out := make([]int, len(in))
for i, v := range in {
out[i] = v * 2
}
Copy a fragment out of a large buffer so the large array can die:
func firstLine(file []byte) []byte {
for i, b := range file {
if b == '\n' {
line := make([]byte, i)
copy(line, file[:i])
return line
}
}
out := make([]byte, len(file))
copy(out, file)
return out
}
After deleting pointer elements from a long-lived slice, clear the tail:
copy(s[i:], s[i+1:])
s[len(s)-1] = nil
s = s[:len(s)-1]
Check cap in pprof heap dumps when RSS looks insane relative to len. A tiny slice with a huge capacity is this bug.