Phone:

Hidden from the page source until you click: friction against scrapers, not a guarantee.

Email:

[email protected]

Category:

Go

Tags:
  • go
  • io-copy
  • error-handling
  • files
  • streaming
  • debugging

Go's io.Copy Can Write Gigabytes Before Reporting an Error

io.Copy has a pleasantly small signature:

func Copy(dst Writer, src Reader) (written int64, err error)

That simplicity can conceal an awkward fact. If it returns an error, the destination may already contain almost all the input. A failed 20 GB transfer might leave 19.9 GB written, visible and perhaps already consumed by something else.

This is not io.Copy behaving badly. Streams are incremental, and neither io.Reader nor io.Writer has a rollback operation. The trouble starts when code treats the call as though it were a transaction:

if _, err := io.Copy(dst, src); err != nil {
	return err
}

The error is handled, technically. The partially changed destination is not.

The byte count belongs to the error path

The documentation for io.Copy says it returns the number of bytes copied and the first error encountered. That count remains meaningful when err is non-nil:

n, err := io.Copy(dst, src)
if err != nil {
	return fmt.Errorf("copy failed after %d bytes: %w", n, err)
}

Logging n will not repair anything, but it changes a vague production report into useful evidence. Zero bytes suggests an immediate permissions, connection or format problem. Several gigabytes suggests a capacity limit, a remote disconnect, media failure or deadline reached during an otherwise functioning transfer.

It also tells the caller that cleanup is required. A destination file may need removing, a multipart upload may need aborting, and a protocol connection may no longer be reusable. Returning only err discards the most useful bit of context the function gave you.

Data and an error can arrive together

Actually, this bit is interesting: a Go reader is allowed to return both bytes and an error from the same call. The caller must process the bytes before acting on the error.

The ordinary loop inside io.Copy does exactly that. It reads a chunk, writes any bytes received, increments the total, and only then returns the read error. The standard library source is short enough to be worth reading.

A source can therefore say, in effect, "these 12,000 bytes are valid, and the stream failed immediately afterwards". Those bytes reach the destination. The final count includes them.

Write failures have similar partial-success semantics. A writer may accept some bytes and return an error. If it accepts fewer bytes without supplying an error, io.Copy turns that broken writer behaviour into io.ErrShortWrite.

The buffer is not a transaction boundary

io.Copy normally works in chunks, but the chunk size does not limit the amount written before failure. It merely limits how much data is being handled by one iteration. Ten thousand successful iterations are still ten thousand changes to the destination.

Nor does io.CopyBuffer make the operation safer. Its supplied buffer might not even be used. If the source implements io.WriterTo, io.Copy delegates to that first; otherwise it may use the destination's io.ReaderFrom. These fast paths are part of the documented contract.

This matters when wrapping readers or writers for metering, testing or fault injection. The actual copying loop may live in one of those specialised methods rather than in io.Copy. Optimising the buffer size is rarely the answer to partial output anyway. The answer is deciding what partial output means for this destination.

Files can be staged before they become visible

When replacing a file, write to a temporary file in the same directory, check every relevant error, then rename it into place:

package atomicfile

import (
	"fmt"
	"io"
	"io/fs"
	"os"
	"path/filepath"
)

func Write(path string, src io.Reader, perm fs.FileMode) error {
	dir := filepath.Dir(path)
	base := filepath.Base(path)

	tmp, err := os.CreateTemp(dir, "."+base+".tmp-*")
	if err != nil {
		return fmt.Errorf("create temporary file: %w", err)
	}
	name := tmp.Name()
	defer func() {
		_ = tmp.Close()
		_ = os.Remove(name)
	}()

	n, err := io.Copy(tmp, src)
	if err != nil {
		return fmt.Errorf("write temporary file after %d bytes: %w", n, err)
	}
	if err := tmp.Chmod(perm); err != nil {
		return fmt.Errorf("set permissions: %w", err)
	}
	if err := tmp.Sync(); err != nil {
		return fmt.Errorf("sync temporary file: %w", err)
	}
	if err := tmp.Close(); err != nil {
		return fmt.Errorf("close temporary file: %w", err)
	}
	if err := os.Rename(name, path); err != nil {
		return fmt.Errorf("replace destination: %w", err)
	}
	return nil
}

Creating the temporary file beside the destination is deliberate. A rename across filesystems generally cannot provide the same atomic replacement. Platform rules around replacing an existing file also differ, so portable applications should test the exact target systems they support.

The rename addresses visibility: readers see the old file or the new file, rather than a half-written mixture. Crash durability is a separate problem. Code requiring a durable commit may also need to synchronise the containing directory after the rename, with platform-specific handling. Atomic and durable are annoyingly different words for good reason.

Buffered writers add one more place to fail

If dst is a bufio.Writer, a successful copy only means the writer accepted the bytes. Some of them may still be in memory. The later flush can fail:

buf := bufio.NewWriter(dst)

n, err := io.Copy(buf, src)
if err != nil {
	return fmt.Errorf("copy failed after %d bytes: %w", n, err)
}
if err := buf.Flush(); err != nil {
	return fmt.Errorf("flush copied data: %w", err)
}

The same principle applies to Close. Some destinations report delayed write failures only when flushed or closed. A nil result from io.Copy is therefore not necessarily the final success signal for the whole operation.

Some destinations cannot be rolled back

An HTTP response, TCP connection, pipe or decompressor output is not a file you can quietly rename. Once bytes have gone downstream, they have gone. The design choices are narrower:

  • Buffer the complete result before sending it, if it is small and bounded.
  • Stage it in a temporary file or object, then publish it.
  • Use a protocol that supports an explicit abort or incomplete status.
  • Accept partial delivery and make the receiver validate a length, checksum or final record.

For HTTP handlers, the first response bytes may also commit the status and headers. Discovering an upstream read error afterwards does not let the handler replace the response with a tidy 500. Closing the connection or terminating a framed response may be the only honest signal left.

io.Copy promises streaming, not atomicity. Keep its byte count, check the errors that occur after it, and arrange rollback or staging before starting the copy. By the time the function tells you something went wrong, the destination may have had a very productive afternoon.

← Back to the blog