Blog / Go

779 words

  • go
  • net-http
  • static-files
  • fileserver
  • performance
  • http-caching

Serving 10,000 Static Pages from Go with Only net/http

Yes, a tiny Go program can serve 10,000 static pages with no framework, and the interesting part is how little of it you write. The standard library already does conditional requests, range requests, content types and index files. Your job is mostly to stop it doing two things you did not ask for.

The whole server

Here is a complete server. It serves a directory called public, refuses directory listings, adds a cache header and sets a header timeout.

package main

import (
	"io/fs"
	"log"
	"net/http"
	"os"
	"path"
	"time"
)

// noListFS hides directories that have no index.html,
// so http.FileServer cannot render a file listing.
type noListFS struct{ fs.FS }

func (n noListFS) Open(name string) (fs.File, error) {
	f, err := n.FS.Open(name)
	if err != nil {
		return nil, err
	}
	st, err := f.Stat()
	if err != nil {
		f.Close()
		return nil, err
	}
	if st.IsDir() {
		idx, err := n.FS.Open(path.Join(name, "index.html"))
		if err != nil {
			f.Close()
			return nil, fs.ErrNotExist
		}
		idx.Close()
	}
	return f, nil
}

func cache(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Cache-Control", "public, max-age=300")
		next.ServeHTTP(w, r)
	})
}

func main() {
	site := http.FileServerFS(noListFS{os.DirFS("public")})

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           cache(site),
		ReadHeaderTimeout: 5 * time.Second,
	}
	log.Fatal(srv.ListenAndServe())
}

That is the lot. No router, no template engine, no index of the 10,000 files held in memory.

Why 10,000 files is not a problem

The file count barely matters, because nothing is loaded at start-up. Each request opens one file, calls Stat, and streams it out. The "index" is your filesystem's directory structure, which was built to look names up quickly.

The operating system then does the caching. Files that are requested often stay in the page cache, so repeat requests are mostly memory reads. Go is not holding your site in a map; the kernel is, and it is better at that job.

  • Memory use is proportional to concurrent requests, not to page count.
  • Start-up time is the same for ten files or ten million.
  • Adding or editing a page needs no restart, because the next request simply opens the new file.

What the standard library quietly gives you

Under FileServer sits http.ServeContent, and it handles more than most people expect:

  • Last-Modified from the file's modification time, and 304 Not Modified for If-Modified-Since.
  • Range requests, so interrupted downloads and media seeking work.
  • A Content-Type from the file extension, falling back to sniffing the first bytes.
  • A redirect from /dir to /dir/, and serving index.html from that directory.

Quick detour: requesting /index.html directly gets a redirect to ./. That is deliberate, to stop two URLs for the same page. It surprises people the first time a curl -I shows a 301 for a file that plainly exists.

The two things you must switch off

The first is directory listings. By default, a directory without an index.html returns an HTML listing of every file in it. For a site with 10,000 generated pages that is rarely something you want published. The noListFS wrapper above turns it into a 404.

The second is anything the filesystem exposes that you did not intend. os.DirFS is a convenience, not a sandbox, and a symlink inside public can point outside it. If the content directory is not entirely yours, open it with os.OpenRoot and use the FS method on the result (Go 1.24 and later), which refuses to follow paths out of the root.

Caching decides your real throughput

The fastest request is the one never made. With Last-Modified and Cache-Control: max-age, browsers and any CDN in front will revalidate cheaply or skip the trip altogether. The five-minute value in the example is a placeholder; pick it based on how often pages change.

Fingerprinted assets (app.3f9a1c.css) are different. Their names change when the content does, so they can safely take max-age=31536000, immutable. Add that in the middleware with a suffix or directory check, not for the whole site.

Embedding is tempting, with a catch

You can swap os.DirFS("public") for an embed.FS and ship one binary. It works with the same FileServerFS call. The catch is that embedded files report a zero modification time, so you get no Last-Modified and no cheap revalidation.

If you embed, set an explicit ETag yourself, perhaps derived from a build identifier, or accept that clients refetch once the max-age runs out. For 10,000 pages that also means a very large binary, so on disk is usually the saner choice.

Measuring it yourself

Do not trust anyone's numbers for your hardware, including mine. Generate a realistic tree and hit it:

mkdir -p public
for i in $(seq 1 10000); do
  mkdir -p public/p/$i
  echo "<h1>Page $i</h1>" > public/p/$i/index.html
done
go run .
# elsewhere: any load tool, requesting random /p/N/ paths

Watch for two things while it runs: whether the first pass over cold files is slower than the second (page cache at work), and whether the file descriptor limit (ulimit -n) bites under high concurrency. Both are operating system behaviour, not Go.

Where it stops being enough

This setup does not compress anything. If you want gzip or brotli, either put a reverse proxy or CDN in front, or write precompressed .gz siblings and pick them by Accept-Encoding in your own handler. It also does not terminate TLS unless you give it certificates, and it has no rate limiting.

For a personal site or documentation tree, though, that is a fair trade: about 50 lines, one dependency (the standard library), and nothing to upgrade on a Tuesday night.

← Back to the blog