25.3 json.Encoder and json.Decoder: Streaming JSON
Alright, let’s get our hands dirty with the real workhorses of the encoding/json package: json.Encoder and json.Decoder. You’ve probably met json.Marshal and json.Unmarshal—they’re fine for small, self-contained jobs. But when you’re dealing with streams of data, whether it’s from an HTTP response body, a file on disk, or a network socket, the Marshal/Unmarshal duo starts to feel like using a sledgehammer to crack a nut. They need the whole nut in your hand at once.
Encoder and Decoder are different. They’re designed to work directly with io.Writer and io.Reader streams. This is a big deal for two reasons: memory and responsiveness. Instead of slurping a 2GB JSON file into memory and then parsing it (hello, ioutil.ReadAll, you beautiful disaster), a Decoder can process it piece by piece. This is how you build scalable, efficient systems that don’t crumple under pressure.
Streaming Data with json.Decoder
Think of the json.Decoder as your personal JSON interpreter for a continuous feed of data. You give it any io.Reader—a file, a network connection, os.Stdin—and it will read and decode tokens from that stream sequentially.
The most powerful method in its arsenal is Decode(). You call it repeatedly until it returns an io.EOF error. This is perfect for JSON arrays, especially large ones, where you can process each element as it’s decoded.
package main
import (
"encoding/json"
"fmt"
"strings"
)
func main() {
// Let's simulate a long JSON array stream from a network connection
const stream = `
{"Name": "Alice", "Age": 25}
{"Name": "Bob", "Age": 22}
{"Name": "Claire", "Age": 29}
`
reader := strings.NewReader(stream)
dec := json.NewDecoder(reader)
type Person struct {
Name string
Age int
}
for {
var p Person
err := dec.Decode(&p) // Decode the next token into p
if err != nil {
break // We'll handle the error properly in a second
}
fmt.Printf("Processed: %+v\n", p)
}
}
But wait, the error handling here is sloppy. A break on any error? That’s a classic rookie mistake. The Decode method will return io.EOF when it’s done, which is a perfectly expected error. You must check for it specifically.
for {
var p Person
if err := dec.Decode(&p); err == io.EOF {
break // Happy path, we're done
} else if err != nil {
log.Fatal(err) // Something actually went wrong
}
fmt.Printf("Processed: %+v\n", p)
}
Taking Control with MoreDecode and Buffering
Here’s a pro tip: the decoder buffers its input. If there’s data left in the buffer after a Decode() call, you can find out about it with Buffered(). This is incredibly useful for peeking ahead or for situations where your stream contains multiple JSON values but also other stuff. You can also use More() to check if the next token is an array element (if you’re in an array) or the start of a new value. This gives you fine-grained control over the parsing process.
Streaming Data with json.Encoder
On the flip side, json.Encoder is your tool for writing JSON to a stream. You create it with an io.Writer and use its Encode() method to write values one after another. This is how you generate JSON output without having to build a gigantic data structure in memory first.
package main
import (
"encoding/json"
"os"
)
func main() {
people := []struct {
Name string
Age int
}{
{"Alice", 25},
{"Bob", 22},
{"Claire", 29},
}
enc := json.NewEncoder(os.Stdout) // Write JSON to standard output
enc.SetIndent("", " ") // Because unindented JSON is a crime against readability
for _, p := range people {
if err := enc.Encode(p); err != nil {
panic(err)
}
// Each call to Encode() writes a complete JSON value followed by a newline.
// This output is a valid JSON sequence, but it's NOT a single JSON array.
}
}
This will output:
{
"Name": "Alice",
"Age": 25
}
{
"Name": "Bob",
"Age": 22
}
{
"Name": "Claire",
"Age": 29
}
Notice the output? It’s a series of JSON objects, not a single JSON array. This is a crucial distinction. If you need a proper array, you have to write the brackets yourself. This is one of those “questionable choices” – it’s flexible but often not what people expect. You’re responsible for the structure.
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
fmt.Println("[") // Write the opening bracket of the array
for i, p := range people {
if err := enc.Encode(p); err != nil {
panic(err)
}
if i < len(people)-1 {
fmt.Print(",") // Write a comma after each element except the last
}
}
fmt.Println("]") // Write the closing bracket
The Slog Logger Integration
Now, why did we just cover this in a chapter about slog? Because slog uses json.Encoder under the hood when you create a slog.JSONHandler. When you log a statement, the handler doesn’t build a giant map and then marshal it. It creates a json.Encoder and streams the key-value pairs directly to your chosen output writer. This is efficient, memory-smart, and exactly the kind of design that makes slog a serious logging library for production systems. Understanding how the encoder works gives you insight into your logging pipeline’s performance and behavior.