Worker Pools in Go

Build a worker pool in Go with channels and a WaitGroup to process many jobs with a fixed number of goroutines, collect results and handle errors.

Starting one goroutine per job is fine for a hundred jobs. For a million jobs, or when each job calls an API with a rate limit, you want a fixed number of goroutines that take jobs from a queue. That is a worker pool.

The Basic Worker Pool

package main

import (
	"fmt"
	"sync"
	"time"
)

func process(n int) int {
	time.Sleep(10 * time.Millisecond) // pretend this is slow work
	return n * n
}

func main() {
	jobs := make(chan int)
	results := make(chan int)

	const workers = 3
	var wg sync.WaitGroup
	for w := 1; w <= workers; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for n := range jobs {
				results <- process(n)
			}
		}()
	}

	go func() {
		for i := 1; i <= 9; i++ {
			jobs <- i
		}
		close(jobs)
	}()

	go func() {
		wg.Wait()
		close(results)
	}()

	sum := 0
	for r := range results {
		sum += r
	}
	fmt.Println("sum of squares:", sum)
}

This prints sum of squares: 285, and takes about three rounds of work instead of nine, because three workers run at once.

The pieces fit together like this:

producer --jobs--> worker 1 \
                   worker 2  --results--> main
                   worker 3 /
  1. Workers loop with for n := range jobs. They stop when jobs is closed and empty.
  2. The producer sends all jobs and then closes jobs.
  3. A closer goroutine waits for all workers and then closes results.
  4. main ranges over results until it is closed.

Forgetting either close:

If jobs is never closed, workers wait forever and results is never closed. If results is never closed, the final loop waits forever. Both end in all goroutines are asleep - deadlock!.

Keeping Results in Order

Results arrive in whatever order workers finish. If order matters, send the job index along and write into a slice:

package main

import (
	"fmt"
	"strings"
	"sync"
)

type job struct {
	index int
	text  string
}

func main() {
	inputs := []string{"go", "rust", "tauri", "axum"}
	out := make([]string, len(inputs))

	jobs := make(chan job)
	var wg sync.WaitGroup
	for w := 0; w < 2; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for j := range jobs {
				out[j.index] = strings.ToUpper(j.text)
			}
		}()
	}
	for i, t := range inputs {
		jobs <- job{index: i, text: t}
	}
	close(jobs)
	wg.Wait()
	fmt.Println(out)
}

This always prints [GO RUST TAURI AXUM]. Each worker writes only to its job's slot, which is safe without a lock because no two jobs share an index.

Handling Errors From Jobs

Send errors back as part of the result so the collector can decide what to do:

package main

import (
	"fmt"
	"strconv"
	"sync"
)

type result struct {
	input string
	value int
	err   error
}

func main() {
	inputs := []string{"10", "x", "30", "4o"}
	jobs := make(chan string)
	results := make(chan result)

	var wg sync.WaitGroup
	for w := 0; w < 2; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for in := range jobs {
				n, err := strconv.Atoi(in)
				results <- result{input: in, value: n, err: err}
			}
		}()
	}
	go func() {
		for _, in := range inputs {
			jobs <- in
		}
		close(jobs)
	}()
	go func() {
		wg.Wait()
		close(results)
	}()

	total, failed := 0, 0
	for r := range results {
		if r.err != nil {
			failed++
			continue
		}
		total += r.value
	}
	fmt.Println("total:", total, "failed:", failed)
}

This prints total: 40 failed: 2. Every job reports back, and the collector counts failures without stopping the pool. If the first error should stop everything, errgroup with a context is a better fit.

Choosing the Number of Workers

  • For CPU-heavy work, runtime.NumCPU() workers is a good starting point.
  • For I/O-heavy work like HTTP calls or database queries, more workers help, but stay below what the remote service allows.
  • Measure with real data before tuning further.

Unbounded goroutines are not a pool:

for _, j := range jobs { go process(j) } with a million jobs starts a million goroutines at once and can overload memory or the service you call. A pool or a semaphore keeps the load fixed.

A correct pool:

Jobs channel closed by the producer, results channel closed after wg.Wait() in its own goroutine, and every job's error reported back. With those three pieces, the pool always finishes.

Summary

A worker pool uses a jobs channel, a fixed number of worker goroutines ranging over it, and a results channel closed after a WaitGroup finishes. Close jobs from the producer and results from a separate closer goroutine. Pass indexes when order matters, return errors inside results, and size the pool for CPU or I/O work.