worker-pool — Concurrent Worker Pool in Go

Fan-out/fan-in pattern for parallel task processing using goroutines and channels. Study project #5 in the Go Backend Developer roadmap.

---

For Recruiters

What and Why

Fifth project in the roadmap — first deep dive into Go's concurrency model. This is not just "launch a goroutine" — it's a production worker pool pattern: a fixed number of goroutines process a task queue in parallel, results are collected via channels, graceful shutdown via context.WithCancel.

Concurrency is the main reason companies choose Go. This pattern simulates a real-world task: N incoming jobs (e.g. HTTP requests to an external API), a pool of W workers, process everything in parallel, collect results, and shut down cleanly — no goroutine leaks, no deadlocks.

What This Project Demonstrates

Skill	Implementation
Goroutines	W workers launched via go worker(...), each in its own goroutine
Channels	jobs chan Job for distributing tasks, results chan Result for collecting
Fan-out / Fan-in	one producer → N workers → one collector
sync.WaitGroup	wait for all workers to finish before closing results
context.WithCancel	cancel all workers via a single signal
Graceful shutdown	os.Signal → cancel() → workers exit → program terminates
Pipeline pattern	generate → process → collect as a channel chain

Stack

• Language: Go 1.22+
• Dependencies: standard library only
• Platform: Linux / macOS / Windows

---

For Developers

Architecture Decisions

Why a fixed worker pool instead of a goroutine per task?

// Bad — uncontrolled goroutine growth under high load
for _, job := range jobs {
    go process(job) // 100k jobs = 100k goroutines = OOM
}

// Good — fixed pool, controlled resource usage
jobsCh := make(chan Job, bufferSize)
for i := 0; i < workerCount; i++ {
    go worker(ctx, jobsCh, resultsCh, &wg)
}

A goroutine costs ~2-8 KB of stack. 100k goroutines = up to 800 MB just for stacks. A pool of 10 workers handles the same 100k jobs with controlled memory usage.

Why sync.WaitGroup for workers instead of just closing the channel?

WaitGroup guarantees all workers have finished before we close results. Closing results while a worker is still writing to it causes a panic. The pattern is strict:

var wg sync.WaitGroup
for i := 0; i < N; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        worker(...)
    }()
}

// Separate goroutine: waits for all workers, then closes results
go func() {
    wg.Wait()
    close(results)
}()

Why context.WithCancel instead of a global done bool flag?

// Bad — race condition on global variable
var done bool
// worker reads done without synchronization

// Good — context propagation, Go standard
ctx, cancel := context.WithCancel(context.Background())
// worker checks ctx.Done() — thread-safe

context is the Go standard for cancellation. It propagates down the call stack, works with select, and is compatible with the standard library (http.Request, sql.DB). A global flag is an anti-pattern.

Why buffered channels?

jobs    := make(chan Job, workerCount*2) // buffer = 2x workers
results := make(chan Result, workerCount*2)

The buffer allows the producer to not block while workers are busy, and workers to not block while the collector hasn't read the previous result yet. Without a buffer — extra blocking and lower throughput.

Structure

worker-pool/
├── main.go      # entry point: channel creation, worker launch, result collection
├── worker.go    # func worker: reads from jobs, writes to results, listens to ctx.Done()
├── job.go       # Job and Result types, func generateJobs
├── go.mod
└── README.md

Install and Run

git clone https://github.com/Shipovmax/worker-pool
cd worker-pool
go run . -workers=5 -jobs=20

Usage

go run . [flags]

Flags:
  -workers int   number of workers (default: 3)
  -jobs    int   number of jobs (default: 10)

Examples

go run . -workers=3 -jobs=10
# Worker 2 | Job 1 | 142ms
# Worker 1 | Job 2 | 89ms
# Worker 3 | Job 3 | 203ms | ERROR: job 3 failed
# ...
# Done: 9 success, 1 errors, total 721ms

# Interrupt with Ctrl+C
go run . -workers=5 -jobs=100
# Worker 1 | Job 1 | 112ms
# ^C
# Done: 3 success, 0 errors, total 350ms

Error Handling

Worker receives a job → simulates work via time.Sleep
→ 20% chance of error (rand)
→ Result{Err: error} goes to collector
→ collector counts errors separately, does not stop processing

context cancelled (Ctrl+C)
→ all workers exit select via ctx.Done()
→ wg.Done() called for each worker
→ results channel closed
→ program exits without goroutine leaks