Buffered IO with bufio in Go

Read lines with bufio.Scanner, fix the token too long error, read user input, and speed up writes with bufio.Writer and Flush.

Reading a file line by line and printing lots of output quickly are two of the most common jobs in Go, and both go through bufio. This page covers bufio.Scanner for reading, the "token too long" error that surprises almost everyone, and bufio.Writer with the Flush call people forget.

Reading Lines with bufio.Scanner

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func main() {
	input := "alice,admin\nbob,editor\n\ncarol,viewer\n"

	s := bufio.NewScanner(strings.NewReader(input))
	line := 0
	for s.Scan() {
		line++
		text := s.Text()
		if text == "" {
			continue
		}
		fmt.Printf("%d: %s\n", line, text)
	}
	if err := s.Err(); err != nil {
		fmt.Println("read error:", err)
	}
}

1: alice,admin
2: bob,editor
4: carol,viewer

Scan moves to the next line and returns false at the end or on an error. Text returns the line without the newline, and it also strips a trailing \r, so Windows files work too. To read a file, pass the *os.File from os.Open instead of the strings.Reader.

Always check s.Err():

When Scan returns false, the loop ends quietly whether the file finished or something went wrong. Only s.Err() tells you which. Skipping it is how programs silently process half a file.

Fixing "bufio.Scanner: token too long"

By default a Scanner refuses lines longer than 64 KB. Minified JSON, log lines with big payloads, or CSV exports hit this limit fast:

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func main() {
	long := strings.Repeat("a", 100_000)

	s := bufio.NewScanner(strings.NewReader(long))
	for s.Scan() {
	}
	fmt.Println("default:", s.Err())

	s = bufio.NewScanner(strings.NewReader(long))
	s.Buffer(make([]byte, 0, 64*1024), 1024*1024) // allow lines up to 1 MB
	for s.Scan() {
		fmt.Println("with Buffer: line of", len(s.Text()), "bytes")
	}
	fmt.Println("error:", s.Err())
}

default: bufio.Scanner: token too long
with Buffer: line of 100000 bytes
error: <nil>

s.Buffer(buf, max) must be called before the first Scan. The first argument is the starting buffer and the second is the largest line you will accept. If lines can be any size, use bufio.Reader and ReadString('\n') instead, which has no limit.

Reading Words and User Input

Split changes what counts as one token. bufio.ScanWords splits on any whitespace:

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func main() {
	s := bufio.NewScanner(strings.NewReader("  go   is\n fun "))
	s.Split(bufio.ScanWords)
	for s.Scan() {
		fmt.Printf("[%s] ", s.Text())
	}
	fmt.Println()
}

[go] [is] [fun]

For reading what a user types in a terminal, wrap os.Stdin the same way:

// norun
reader := bufio.NewScanner(os.Stdin)
fmt.Print("Your name: ")
if reader.Scan() {
	fmt.Println("Hello,", reader.Text())
}

This is more reliable than fmt.Scanln, which stops at the first space and leaves the rest of the line for the next call.

Faster Output with bufio.Writer

Each fmt.Println to os.Stdout or to a file is a system call. Printing a million lines that way is slow. A bufio.Writer collects output in memory and writes it in large chunks:

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	w := bufio.NewWriter(os.Stdout)
	defer w.Flush()

	for i := 1; i <= 3; i++ {
		fmt.Fprintf(w, "line %d\n", i)
	}
}

line 1
line 2
line 3

This is the standard trick in competitive programming and in any tool that prints a lot.

Forgetting Flush loses output:

Data sits in the buffer until it is full or you call Flush. If the program exits first, the end of your output never appears, and the file looks cut off. Call Flush before closing the file, and check its error. Note that os.Exit and log.Fatal skip deferred calls, so a deferred Flush will not run in that case.

When writing a file, the order is: flush the buffer, then close the file.

// norun
w := bufio.NewWriter(f)
// ... writes ...
if err := w.Flush(); err != nil {
	return err
}
return f.Close()

Summary

Use bufio.Scanner to read lines or words and always check s.Err() after the loop. If you see "bufio.Scanner: token too long", raise the limit with s.Buffer before scanning, or switch to bufio.Reader. Wrap heavy output in bufio.Writer for speed, and always Flush it before the program or file closes.