io.Reader and io.Writer in Go

Files, network connections, HTTP bodies, compressed streams and in-memory buffers all look different, but Go lets you treat them the same way. The reason is two tiny interfaces in the io package. Once they click, a lot of the standard library starts to feel simple.

The Two Interfaces

// norun
type Reader interface {
	Read(p []byte) (n int, err error)
}

type Writer interface {
	Write(p []byte) (n int, err error)
}

A Reader fills the slice you give it and says how many bytes it put there. When there is nothing left, it returns io.EOF. A Writer takes bytes and says how many it wrote.

*os.File, net.Conn, *bytes.Buffer, *strings.Reader, http.Request.Body and gzip.Reader all implement one or both. A function that accepts an io.Reader works with every one of them.

Write Functions That Accept io.Reader

Here is a word counter. Because it takes an io.Reader instead of a file name, it works with a string, a file or a network stream, and it is easy to test:

package main

import (
	"bufio"
	"fmt"
	"io"
	"strings"
)

func countWords(r io.Reader) (int, error) {
	s := bufio.NewScanner(r)
	s.Split(bufio.ScanWords)
	n := 0
	for s.Scan() {
		n++
	}
	return n, s.Err()
}

func main() {
	n, err := countWords(strings.NewReader("the quick brown fox"))
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Println("words:", n)
}

words: 4

In a real program you would call countWords(f) with an open file or countWords(r.Body) in an HTTP handler. The function does not care.

io.Copy Does the Loop for You

Copying from any reader to any writer is one call:

package main

import (
	"fmt"
	"io"
	"log"
	"os"
	"path/filepath"
	"strings"
)

func main() {
	src := strings.NewReader("report line 1\nreport line 2\n")

	path := filepath.Join(os.TempDir(), "report.txt")
	dst, err := os.Create(path)
	if err != nil {
		log.Fatal(err)
	}
	defer os.Remove(path)

	n, err := io.Copy(dst, src)
	if err != nil {
		log.Fatal(err)
	}
	if err := dst.Close(); err != nil {
		log.Fatal(err)
	}
	fmt.Println("copied", n, "bytes")
}

copied 28 bytes

io.Copy reads in chunks, so it can copy a 10 GB file using only a small buffer. Use it instead of io.ReadAll followed by Write whenever the data might be large.

Composing Readers and Writers

Small wrappers in io let you plug streams together.

package main

import (
	"bytes"
	"crypto/sha256"
	"fmt"
	"io"
	"os"
	"strings"
)

func main() {
	// Write to two places at once.
	var saved bytes.Buffer
	w := io.MultiWriter(os.Stdout, &saved)
	fmt.Fprintln(w, "hello from MultiWriter")
	fmt.Println("buffer has", saved.Len(), "bytes")

	// Hash data while it is being read.
	h := sha256.New()
	r := io.TeeReader(strings.NewReader("some upload"), h)
	data, _ := io.ReadAll(r)
	fmt.Printf("read %q, sha256 %x...\n", data, h.Sum(nil)[:4])

	// Never read more than 5 bytes.
	limited := io.LimitReader(strings.NewReader("0123456789"), 5)
	part, _ := io.ReadAll(limited)
	fmt.Printf("limited: %q\n", part)
}

hello from MultiWriter
buffer has 23 bytes
read "some upload", sha256 67af79fa...
limited: "01234"

• io.MultiWriter sends every write to all its writers, like the tee command.
• io.TeeReader passes everything it reads to a writer as well. Here it feeds a hash, so you get the checksum without reading the data twice.
• io.LimitReader stops after a number of bytes, which protects you from a client sending a huge body.

A hash from crypto/sha256 is itself an io.Writer, and so is a gzip writer, an HTTP response and a terminal. That is why these pieces fit together so easily.

Implementing io.Writer Yourself

Any type with a Write method is a writer. This one counts bytes passing through:

package main

import (
	"fmt"
	"io"
	"strings"
)

type countingWriter struct {
	w     io.Writer
	total int64
}

func (c *countingWriter) Write(p []byte) (int, error) {
	n, err := c.w.Write(p)
	c.total += int64(n)
	return n, err
}

func main() {
	cw := &countingWriter{w: io.Discard}
	io.Copy(cw, strings.NewReader(strings.Repeat("x", 1000)))
	fmt.Println("bytes seen:", cw.total)
}

bytes seen: 1000

io.Discard is a writer that throws everything away, which is handy when you only care about a side effect like the count.

Summary

io.Reader and io.Writer are one-method interfaces that files, buffers, network connections and HTTP bodies all implement. Write functions that accept them so they work with any source and are easy to test. Use io.Copy to move data without loading it all into memory, and combine streams with io.MultiWriter, io.TeeReader and io.LimitReader.