Blog / Rust

1203 words

  • rust
  • bufwriter
  • error-handling
  • file-io
  • drop
  • std-io

Rust's BufWriter Drops Errors: Why You Must Call flush() Yourself

Here is a small program that writes one line to a file and returns Ok(()) from main:

use std::fs::File;
use std::io::{BufWriter, Write};

fn main() -> std::io::Result<()> {
    let mut out = BufWriter::new(File::create("/dev/full")?);
    writeln!(out, "hello")?;
    Ok(())
}

On Linux, /dev/full is a device where every write fails with ENOSPC, "no space left on device". It exists precisely so you can test how your code behaves when the disk fills up. So you would expect this program to complain. It doesn't: every call in it returns Ok, and the process exits with status 0. The line went nowhere, and nothing told you.

Add out.flush()?; before the final Ok(()) and you get the error you wanted: No space left on device (os error 28), and a non-zero exit. (I'm describing what the standard library documents here rather than pasting a terminal session, so try it yourself; /dev/full makes it a ten-second experiment.)

Where the error goes

BufWriter collects small writes into an in-memory buffer (8 KiB by default) so that you make one write(2) syscall instead of hundreds. That is the whole point of it. The consequence is that writeln!(out, "hello") succeeded because it only copied five bytes and a newline into a Vec. The file was never touched.

The real write happens later, either when the buffer fills or when the writer is dropped. And Drop::drop returns nothing. It can't return an error, so the implementation does the only thing it can. Paraphrasing the standard library source:

impl<W: ?Sized + Write> Drop for BufWriter<W> {
    fn drop(&mut self) {
        if !self.panicked {
            // dtors should not panic, so we ignore a failed flush
            let _r = self.flush_buf();
        }
    }
}

let _r = is the bit that matters. The flush is attempted, the Result is bound to a variable nobody reads, and it goes away. The documentation for BufWriter says this outright: it is critical to call flush before the writer is dropped, because any errors that happen during the drop are ignored.

Which is a slightly awkward thing to find in a language whose reputation is built on not letting you ignore errors. The compiler can't help, because you didn't ignore anything. The ignoring happens inside a destructor, out of sight. (Related, and something I've written about before in a different guise: destructors are a poor place to do fallible work, because there's nobody to hand the failure to.)

The fix: flush, and return the result

The boring solution is the right one. Call flush() yourself, at a point where you can propagate the error:

use std::fs::File;
use std::io::{self, BufWriter, Write};

fn write_report(path: &str, lines: &[String]) -> io::Result<()> {
    let mut out = BufWriter::new(File::create(path)?);
    for line in lines {
        writeln!(out, "{line}")?;
    }
    out.flush()
}

Returning out.flush() as the tail expression means the caller sees the failure. After a successful flush the buffer is empty, so the drop that follows has nothing to write and can't fail in any interesting way. The docs make that point too: flushing first means drop won't even attempt file operations.

Note that the ? on writeln! is not enough on its own. Writes can fail mid-loop if the buffer fills and gets pushed out, so you do sometimes see errors from write itself. That is what makes this bug so intermittent in practice: a report bigger than 8 KiB fails loudly at some point in the loop, while a report smaller than 8 KiB sails through until the silent drop. A test with a big input passes the error path; production with a small one loses the data.

into_inner is the other honest exit

If you want the underlying writer back, into_inner() flushes and returns a Result:

let file = out.into_inner().map_err(|e| e.into_error())?;
file.sync_all()?;

The error type there is IntoInnerError<BufWriter<W>>, which is a bit of a mouthful, but actually this bit is interesting: it carries the BufWriter itself, with the unwritten bytes still in it. You can call e.error() to look at what went wrong and e.into_inner() to get the writer back and try again, say after freeing some disk space or reconnecting a socket. into_error(), used above, just throws the writer away and keeps the io::Error, which is what most callers want.

Pulling the file back out is also how you get to sync_all, which brings me to the next trap.

flush() is not fsync

BufWriter::flush does two things: it drains its own buffer into the inner writer, then calls the inner writer's flush. For a File, the inner flush is a no-op, because File has no user-space buffer of its own. So a successful flush() on a BufWriter<File> tells you that write(2) accepted the bytes. It says nothing about them reaching the disk; they may be sitting in the kernel's page cache. If a power cut must not lose the data, you need File::sync_all as well, and for replacing an existing file safely you want the write-to-temp-then-rename dance covered in the earlier post on std::fs not being atomic.

Two different failure modes, then, and two different calls:

  • flush() catches errors from moving your buffered bytes to the OS (full disk, closed pipe, dropped connection).
  • sync_all() catches errors from the OS moving them to storage.

Other ways the buffer never gets flushed

Drop is a best effort even when you ignore the error, and there are paths where it doesn't run at all:

  • std::process::exit does not run destructors for your locals, so a BufWriter still alive in main loses its buffer. Flush before you exit, or structure the code so the writer is dropped first.
  • A panic = "abort" build doesn't unwind, so nothing gets dropped on the way out.
  • Leaking the value (mem::forget, or a reference cycle in an Rc) means drop never happens.
  • If the inner writer panicked in the middle of a write, BufWriter remembers that and skips the flush in drop, rather than risk writing a half-sent buffer twice.

Standard output is a slightly different animal. Stdout is line-buffered internally and the runtime flushes it on a normal exit, which is why println! tends to just work. Wrap it in your own BufWriter for speed, though, and you have taken on the same duty as with a file. The failure you're most likely to meet there is a closed pipe: run your tool as mytool | head -1, and once head exits the writes start failing with BrokenPipe. Rust ignores SIGPIPE by default, so you get an error value to deal with, and a BufWriter dropped without a flush will swallow it.

Making it hard to forget

If the writer lives inside a struct, give the struct a consuming method instead of relying on callers to remember:

struct Report {
    out: BufWriter<File>,
}

impl Report {
    fn line(&mut self, s: &str) -> io::Result<()> {
        writeln!(self.out, "{s}")
    }

    /// Consumes the report, so it can't be written to afterwards.
    fn finish(mut self) -> io::Result<()> {
        self.out.flush()
    }
}

Taking self by value means the type system stops further writes after finish, and the name tells a reader that something fallible happens there. It doesn't stop someone dropping a Report without calling it, which is the same hole as before, just narrower. Rust has no way to force a call at the end of a value's life; the nearest you can get is a #[must_use] on a type or a debug assertion in your own Drop that shouts if finish was never called.

The same reasoning applies well beyond BufWriter. Anything that buffers and completes on drop, such as compressors and encoders that write a trailer, has the identical shape: the last bytes, and the last error, arrive at the point where nobody is listening. When a type in Rust has a method called finish, flush or into_inner, that is usually the library telling you where the error handling was meant to happen.

← Back to the blog