Articles / Go

753 words

  • go
  • idempotency
  • net-http
  • api-design
  • reliability
  • distributed-systems

Why Your API Retry Creates Two Orders: Idempotency Keys in Go

A client sends POST /orders, the server creates the order, and the response is lost on the way back. The client sees a timeout, retries, and now there are two orders. Nothing was buggy in the usual sense: both sides did exactly what they were told.

The client cannot tell "the request never arrived" from "it arrived and the reply vanished". Retrying is the only sensible option, so it will happen. The fix is to make the retry harmless, and the standard tool is an idempotency key.

The client names the attempt

The client generates a unique value (a random UUID is fine) once per logical operation and sends it as an Idempotency-Key header. Every retry of that operation reuses the same value. The server remembers what it did for each key and replays the stored response instead of doing the work again.

That is the whole concept. The details are where it goes wrong. A decent implementation has to handle four cases:

  • The key arrives again after the first request finished: replay the saved response.
  • The key arrives while the first request is still running: do not start a second copy.
  • The key is reused with a different request body: that is a client bug, so reject it.
  • Two users pick the same key: they must never see each other's responses.

A small middleware

This is an in-memory version, enough to show the shape. The store has one mutex and a map from key to entry. The claim step must be atomic: check and insert under the same lock, otherwise two simultaneous retries both think they are first.

package idem

import (
	"bytes"
	"crypto/sha256"
	"io"
	"net/http"
	"net/http/httptest"
	"sync"
)

type entry struct {
	fp     [32]byte
	done   bool
	status int
	header http.Header
	body   []byte
}

type Store struct {
	mu sync.Mutex
	m  map[string]*entry
}

func NewStore() *Store { return &Store{m: make(map[string]*entry)} }

type state int

const (
	fresh state = iota
	inFlight
	finished
	mismatch
)

// claim atomically registers key, or reports what already holds it.
func (s *Store) claim(key string, fp [32]byte) (*entry, state) {
	s.mu.Lock()
	defer s.mu.Unlock()
	if e, ok := s.m[key]; ok {
		switch {
		case e.fp != fp:
			return nil, mismatch
		case !e.done:
			return nil, inFlight
		}
		return e, finished
	}
	s.m[key] = &entry{fp: fp}
	return nil, fresh
}

func (s *Store) release(key string) {
	s.mu.Lock()
	delete(s.m, key)
	s.mu.Unlock()
}

func (s *Store) complete(key string, rec *httptest.ResponseRecorder) {
	s.mu.Lock()
	defer s.mu.Unlock()
	e := s.m[key]
	e.done, e.status = true, rec.Code
	e.header, e.body = rec.Header().Clone(), rec.Body.Bytes()
}

The middleware reads the body, fingerprints it together with the method and path, and scopes the key to the caller. I am assuming an earlier auth layer put a user ID somewhere you can get at; here it is a placeholder function.

func Middleware(s *Store, userID func(*http.Request) string, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		hk := r.Header.Get("Idempotency-Key")
		if hk == "" || r.Method != http.MethodPost {
			next.ServeHTTP(w, r)
			return
		}
		body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
		if err != nil {
			http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
			return
		}
		r.Body = io.NopCloser(bytes.NewReader(body))

		key := userID(r) + ":" + hk
		fp := sha256.Sum256(append([]byte(r.Method+" "+r.URL.Path+"\n"), body...))

		prev, st := s.claim(key, fp)
		switch st {
		case mismatch:
			http.Error(w, "key reused with a different request", http.StatusUnprocessableEntity)
			return
		case inFlight:
			http.Error(w, "original request still in progress", http.StatusConflict)
			return
		case finished:
			for k, v := range prev.header {
				w.Header()[k] = v
			}
			w.Header().Set("Idempotent-Replayed", "true")
			w.WriteHeader(prev.status)
			w.Write(prev.body)
			return
		}

		rec := httptest.NewRecorder()
		next.ServeHTTP(rec, r)
		if rec.Code >= 500 {
			s.release(key) // let the client retry a failure
		} else {
			s.complete(key, rec)
		}
		for k, v := range rec.Header() {
			w.Header()[k] = v
		}
		w.WriteHeader(rec.Code)
		w.Write(rec.Body.Bytes())
	})
}

Why each branch exists

The inFlight case is the one people forget. A client with an aggressive timeout can retry while the first request is still inside your handler. Without that branch you get exactly the double order you were trying to prevent, just with extra steps. A 409 tells the client to wait and try again.

The fingerprint check catches a different mistake: a client that reuses one key for two different orders. Replaying the first response for the second request would silently drop the second order. Better to fail loudly with a 422.

Scoping the key by user matters for security. If keys are global, user B can send user A's key and receive A's saved response, which may contain A's data. Treat the key as a lookup value, never as a secret.

Quick detour: why not just dedupe on the order contents?

You could hash the body and refuse duplicates, but two genuinely identical orders are legal. Someone really can buy two of the same thing a minute apart. Only the client knows whether a request is a retry or a new intention, which is why the client has to supply the key.

What this version cheats on

An in-memory map dies with the process and does not work across several instances. In production the claim becomes an INSERT into a table with a unique constraint on (user_id, key), and the database arbitrates races for you. Postgres's INSERT ... ON CONFLICT DO NOTHING and checking the affected row count is the usual way.

Better still, write the idempotency row and the order in one transaction. If they commit separately, a crash between them leaves you either with an order and no record (retry makes a duplicate) or a record and no order (retry replays a response for something that never happened).

Other gaps worth knowing about:

  • Nothing expires. Add a TTL (24 hours is a common choice) and a cleanup job.
  • A panic in the handler leaves the key stuck as in-flight. A defer that releases on panic, or an expiry on in-flight entries, fixes it.
  • Releasing the key on a 5xx is a choice. It assumes failed requests had no side effects, which is only true if the handler is transactional. Some APIs cache errors too.
  • Buffering the whole response in a recorder is fine for small JSON and wrong for streaming.

The retry will always come, because the network gives the client no better option. Making the server recognise it is a few dozen lines, and cheaper than refunding the second order.

← All articles