How std::atomic Actually Works: A Tour Down to the x86 Cache Line

Ref

cppreference: std::atomic

Intel® 64 and IA-32 Architectures Software Developer's Manual, Vol. 3A — "Multiple-Processor Management"

x86-TSO: A Rigorous and Usable Programmer's Model for x86 Multiprocessors

Agner Fog — Instruction Tables

A Primer on Memory Consistency and Cache Coherence, 2nd ed. (open access)

Intro

std::atomic is one of those things every C++ programmer has used and very few can explain down to the metal. "It makes the variable thread-safe" is the usual answer, and it is wrong in at least three interesting ways: it doesn't make sequences of operations safe, its loads and stores usually compile to the exact same instruction as a plain variable, and the thing that actually makes fetch_add work lives in the cache, not in your code.

This post walks through what std::atomic really guarantees and how each guarantee is implemented on x86-64 — from what the compiler is no longer allowed to do, down to what a lock prefix does to a cache line. Every assembly listing below is real output from GCC 13.3 at -O2, not a guess.

What "Atomic" Actually Promises

An operation is atomic if no other thread can ever observe it half-done. Other threads see either the state before it or the state after it — nothing in between.

That sounds abstract until you see what "half-done" looks like. Suppose a market-data thread keeps the latest quote in a struct and a strategy thread reads it:

struct Quote { double price; int64_t qty; };   // 16 bytes
Quote latest;                                   // plain, non-atomic

The writer stores price, then stores qty — two separate 8-byte writes. If the reader happens to land between them, it gets the new price paired with the old quantity: a quote that never existed. This is called tearing, and the same thing happens to a single 64-bit integer on a 32-bit CPU (written as two 32-bit halves), or to a value that straddles two cache lines.

So the first promise of std::atomic<T> is: every access to the object is all-or-nothing. There are really two kinds of access to think about:

• Plain reads and writes — load() and store().
• Read-modify-writes (RMW) — fetch_add, exchange, compare_exchange_* and the operators built on them (++, +=, |=…). Here the read, the computation and the write together form one indivisible step.

They are implemented completely differently, so we'll take them one at a time. But first, the part people usually miss.

A Plain Variable Is Not Enough

On x86, an aligned int load is a single mov, and the hardware already guarantees it can't tear. So why not just use a plain int for a flag that one thread writes and another reads?

Because the C++ standard says that a write racing with any other access to a non-atomic object is a data race, and a data race is undefined behaviour. The compiler is therefore entitled to assume that no other thread touches your plain variables, and it optimises accordingly. Here is the classic stop-flag loop:

bool plain_stop = false;
std::atomic<bool> stop{false};

void spin_plain()  { while (!plain_stop) {} }
void spin_atomic() { while (!stop.load(std::memory_order_acquire)) {} }

spin_plain:
    ret                              ; the whole loop is gone

spin_atomic:
.L22:
    movzx   eax, BYTE PTR stop[rip]  ; re-read on every iteration
    test    al, al
    je      .L22
    ret

The plain version doesn't even become an infinite loop — GCC deleted it outright. From the compiler's point of view nothing can change plain_stop inside the loop, so the loop either exits immediately or spins forever without side effects, and the latter is UB anyway. Either way, ret is a legal translation.

Beyond hoisting loads out of loops, a compiler working on plain variables may also keep values in registers instead of re-reading memory, merge several stores into one, split one store into two smaller ones (yes, GCC sometimes stores a 64-bit constant as two 32-bit immediates), or invent stores that weren't in the source. Each of these is perfectly fine in a single-threaded world, and every one of them breaks concurrent code.

So the job of std::atomic happens on three layers:

1. Compiler — every atomic access must actually happen, exactly once, as one full-width access, and must not be moved across other operations in ways its memory order forbids.
2. Hardware — the access has to be one the CPU itself guarantees is indivisible.
3. Ordering — the memory order you pick constrains how surrounding ordinary reads and writes become visible to other cores.

volatile covers only part of the first layer. It stops the compiler from dropping accesses, but guarantees nothing about tearing or ordering, and a race on a volatile object is still UB. It is for memory-mapped I/O, not for threads.

Loads and Stores: Atomic for Free

The Intel SDM promises that a naturally aligned read or write of 1, 2, 4 or 8 bytes is always carried out atomically. The reason is mechanical: an aligned access of that size can never cross a 64-byte cache line, and the core moves data in and out of L1 one line at a time, so another core can observe the line only before or after the access, never during it.

That means std::atomic<T> has very little to do for small types. It just needs to:

• guarantee alignment — std::atomic<int64_t> is always 8-byte aligned, even inside a packed struct where a plain int64_t might not be;
• emit one full-width instruction — never two halves;
• forbid the compiler games above.

Once those hold, the hardware does the rest. Look at what all the memory orders compile to:

std::atomic<int> a;
int  load_relaxed()       { return a.load(std::memory_order_relaxed); }
int  load_acquire()       { return a.load(std::memory_order_acquire); }
int  load_seqcst()        { return a.load(); }
void store_relaxed(int v) { a.store(v, std::memory_order_relaxed); }
void store_release(int v) { a.store(v, std::memory_order_release); }
void store_seqcst(int v)  { a.store(v); }

load_relaxed:   mov  eax, DWORD PTR a[rip]
load_acquire:   mov  eax, DWORD PTR a[rip]
load_seqcst:    mov  eax, DWORD PTR a[rip]
store_relaxed:  mov  DWORD PTR a[rip], edi
store_release:  mov  DWORD PTR a[rip], edi
store_seqcst:   xchg edi, DWORD PTR a[rip]

Five of six are a plain mov, byte-for-byte identical to what a non-atomic int would produce. The atomicity of a load or store on x86 is almost entirely a contract with the compiler. The one outlier, the seq_cst store, is about ordering rather than atomicity, and we'll get to it.

When the type is too wide for one instruction, things change. On 32-bit x86 an atomic<int64_t> uses an SSE movq or cmpxchg8b. For 16 bytes, GCC calls into libatomic (__atomic_load_16), which picks an implementation at runtime: lock cmpxchg16b on older CPUs (a compare-and-swap abused as a load, so every read takes the expensive path), or a plain aligned vmovdqa on AVX-capable CPUs, where Intel and AMD now document 16-byte aligned SSE accesses as atomic. Anything bigger falls back to a lock hidden inside the library. It's still atomic, just no longer lock-free — check is_lock_free() before you put a big struct in a std::atomic on a hot path.

The Gap Between a Load and a Store

Here is the trap: atomic load + atomic store is not an atomic read-modify-write.

std::atomic<int> a;
void load_then_store() { a.store(a.load() + 1); }
void assign_plus()     { a = a + 1; }          // same thing!
void inc()             { a.fetch_add(1); }     // what you actually wanted

load_then_store:
    mov   eax, DWORD PTR a[rip]   ; atomic load
    add   eax, 1                  ; plain register arithmetic
    xchg  eax, DWORD PTR a[rip]   ; atomic (seq_cst) store
assign_plus:
    mov   eax, DWORD PTR a[rip]
    add   eax, 1
    xchg  eax, DWORD PTR a[rip]
inc:
    lock add DWORD PTR a[rip], 1

Notice that a = a + 1 and a += 1 look the same and are completely different. The first calls load(), does arithmetic on an ordinary int, then calls store() — and even though the store is a fully fenced xchg, nothing stops another core from slipping in between the mov and the xchg:

This is not a theoretical race. Four threads each doing a million increments on a two-core box:

expected    4000000
load+store  2092176   (about half the increments vanished)
fetch_add   4000000
CAS loop    4000000

Each individual operation was atomic. The combination was not. Making a read-modify-write indivisible needs help from the hardware, and on x86 that help is a single prefix byte.

Inside the lock Prefix

lock (0xF0) is an instruction prefix that can be attached to read-modify-write instructions with a memory destination — add, sub, and, or, xor, inc, dec, xadd, cmpxchg, bts and friends. xchg with a memory operand is locked implicitly, which is why the seq_cst store above used it. The prefix tells the core: make this whole read-compute-write indivisible with respect to every other core in the system.

The name is historical. Before the Pentium Pro, lock really did assert a LOCK# pin and freeze the shared bus, stopping every other processor's memory traffic. Modern cores do something far more surgical: they lock one cache line, using the cache-coherence protocol they already have.

Quick refresher on coherence. Each cached line is in one of a few states — the MESI family:

• Modified — only I have it, and it's dirty;
• Exclusive — only I have it, still clean;
• Shared — several cores have read-only copies;
• Invalid.

A core may write a line only when it holds it in M or E. To get there, it sends a Read For Ownership (RFO) that invalidates everyone else's copy. (Intel uses MESIF and AMD uses MOESI, but the extra states don't change the story.)

With that in place, here is roughly what happens for lock add [x], 1:

1. Drain the store buffer. Every earlier store has to be committed to L1 first. This is why every locked instruction is also a full memory barrier.
2. RFO. Acquire line x in Exclusive or Modified state; every other copy is invalidated.
3. Pin the line. While it's pinned, the core delays its answers to any snoop that asks for this line — another core trying to read it or take it away simply waits.
4. Read, add, write — all inside the local L1.
5. Unpin and answer the delayed snoops. The next core gets the line with the finished value in it.

Between steps 3 and 5 nobody else can read or write the line, so from the outside the whole operation happens at a single instant. That is all atomicity is.

Where it gets expensive

• Uncontended, a locked RMW costs roughly 15–25 cycles on recent Intel cores (see Agner Fog's tables), versus about 1 for a plain add. Most of that is the store-buffer drain, and the fact that nothing around it can overlap.
• Contended, the line has to travel. Every core that wants to fetch_add the same counter must pull the line into its own L1 first, and the line ping-pongs across the chip — tens of nanoseconds within a socket, more across sockets.
• Split locks. If the operand straddles two cache lines there is no single line to pin, and the CPU falls back to the old behaviour of locking the whole interconnect. Every core in the machine stalls, and it costs on the order of a thousand cycles. Linux (5.7+) can detect this via split_lock_detect and will warn or throttle the offending process. That's one more reason std::atomic insists on alignment.

fetch_add, exchange and the CAS Loop

x86 has dedicated locked instructions for the common RMWs, and the compiler uses them directly:

; a.fetch_add(1), result unused
    lock add  DWORD PTR a[rip], 1
; return a.fetch_add(1)
    mov       eax, 1
    lock xadd DWORD PTR a[rip], eax      ; eax ← old value
; return a.exchange(v)
    mov       eax, edi
    xchg      eax, DWORD PTR a[rip]      ; implicitly locked

Each of these is one instruction that always succeeds. There is no retry. It doesn't matter what the old value was — the core holds the line and applies the operation.

Compare-and-swap is the general-purpose tool. cmpxchg compares eax with memory; if they're equal it writes the new value, otherwise it loads the current value into eax. That second half is exactly why compare_exchange_* updates expected on failure. You can build any RMW from it:

int cas_inc() {
    int old = a.load(std::memory_order_relaxed);
    while (!a.compare_exchange_weak(old, old + 1)) {}
    return old;
}

    mov   eax, DWORD PTR a[rip]
.L15:
    lea   edx, 1[rax]                  ; desired = old + 1
    lock  cmpxchg DWORD PTR a[rip], edx
    jne   .L15                         ; lost the race: eax already holds the new old

Correct, but not equivalent to fetch_add. If another core changes the value between the first load and the cmpxchg, the CAS fails and we go round again — reading, re-acquiring the line, trying again. Under contention lock xadd is wait-free, since every attempt succeeds, while a CAS loop is merely lock-free: the system makes progress, but an unlucky thread can keep losing. On the same two-core box, the timings were about 14 ns per fetch_add versus about 28 ns per CAS-loop increment. (That box is a 2-vCPU VM, so read it for the ratio, not the absolute numbers.) The gap grows with core count.

So the rule is: if a dedicated RMW exists, use it; reach for CAS only when it doesn't. Real cases for CAS include atomic max/min (fetch_max only arrives in C++26), saturating or conditional updates ("decrement only if positive"), updating several fields packed into one 64-bit word, and floating point. C++20 gave us atomic<double>::fetch_add, but x86 has no lock xadd for floats, so GCC quietly compiles it into a lock cmpxchg loop.

weak vs strong

compare_exchange_weak may fail spuriously — returning false even though the value matched. That comes from LL/SC architectures (older ARM, POWER, RISC-V), where CAS is built from a load-linked/store-conditional pair and the store can fail for unrelated reasons such as an interrupt or the line being evicted. strong hides that with an internal retry loop.

On x86 there is nothing to hide. Both compile to the identical lock cmpxchg:

cas_strong:                         cas_weak:
    mov  eax, edi                       mov  eax, edi
    lock cmpxchg [a], esi               lock cmpxchg [a], esi
    sete al                             sete al

Still, choose by meaning so the code stays correct on ARM: use weak inside a retry loop, and strong for a one-shot decision such as "only one thread gets to initialise this".

Memory Order, the x86 Edition

Atomicity says a single access is indivisible. Memory order says how the accesses around it become visible to other cores — the third layer from earlier. On x86 this turns out to be much simpler than the C++ model suggests, because the hardware memory model, x86-TSO (Total Store Order), is already strong:

• loads are not reordered with other loads;
• stores are not reordered with other stores;
• loads are not reordered with later stores;
• the only reordering: a store may be delayed past a later load to a different address.

That one exception comes from the store buffer. A store doesn't go straight to L1; it waits in a per-core queue so the pipeline doesn't stall on cache misses. Loads, meanwhile, go straight to the cache (checking the core's own buffer first). So a core can effectively perform its load before its earlier store becomes globally visible:

This explains the assembly from earlier:

• acquire load / release store cost nothing on x86. TSO already provides everything they promise, so they compile to plain mov, and the memory order only stops the compiler from reordering.
• A seq_cst store becomes xchg (older compilers emit mov + mfence). Sequential consistency must forbid the store→load reordering above, so the store buffer has to be drained, and a locked instruction does exactly that.
• seq_cst loads are still plain mov — the cost is paid on the store side.
• Every locked RMW is a full barrier. On x86, fetch_add(1, std::memory_order_relaxed) emits the same lock add as the seq_cst version. Relaxing an RMW only gives the compiler more freedom; the CPU does the same work.

None of this licenses writing x86-only orderings in portable code. On ARM, acquire and release are real instructions (LDAR/STLR), and a missing order there is a real bug. But it does tell you where the cost on x86 actually lives: seq_cst stores and RMWs.

Putting It Together: An SPSC Ring Buffer

Everything above adds up to the most useful pattern in low-latency C++: if a variable has exactly one writer, a load followed by a store is a safe read-modify-write, because nobody else can slip into the gap. A single-producer / single-consumer queue is built entirely on that:

template <class T, size_t N>               // N a power of two
class SpscQueue {
    alignas(64) std::atomic<size_t> head_{0};   // written only by the consumer
    alignas(64) std::atomic<size_t> tail_{0};   // written only by the producer
    alignas(64) T buf_[N];
public:
    bool push(const T& v) {                    // producer thread
        size_t t = tail_.load(std::memory_order_relaxed);         // own variable
        if (t - head_.load(std::memory_order_acquire) == N) return false;  // full
        buf_[t & (N - 1)] = v;
        tail_.store(t + 1, std::memory_order_release);            // publish
        return true;
    }
    bool pop(T& out) {                         // consumer thread
        size_t h = head_.load(std::memory_order_relaxed);         // own variable
        if (h == tail_.load(std::memory_order_acquire)) return false;      // empty
        out = buf_[h & (N - 1)];
        head_.store(h + 1, std::memory_order_release);            // free the slot
        return true;
    }
};

On x86 this compiles to zero locked instructions — every atomic access is a plain mov. The release store of tail_ guarantees the slot write is visible before the new tail is, and the acquire load on the other side guarantees the consumer doesn't read the slot early. Together they replace a mutex entirely.

The alignas(64) matters just as much. If head_ and tail_ shared a cache line, the producer's writes would keep invalidating the consumer's copy and vice versa. That's the same line ping-pong as a contended lock, without any lock in the code. This is false sharing. (std::hardware_destructive_interference_size is the portable spelling of 64, although some compilers warn about using it in headers.) Production queues go one step further and have each side cache the other side's index locally, so the shared line is only touched when the cached copy says the queue looks full or empty.

Cheat Sheet

Wrapping Up

Strip away the API, and std::atomic on x86 is three promises kept by three different parties:

• The compiler promises every access really happens, once, at full width, in an order your memory order allows. That's all a relaxed load or store costs — nothing at runtime.
• The hardware promises aligned accesses up to 8 bytes can't tear, because they never leave a single cache line.
• The lock prefix turns read-modify-write into one indivisible step by pinning a cache line through the coherence protocol, and it drains the store buffer on the way, which is where both its cost and its barrier semantics come from.

Once you see it that way, performance advice falls out naturally: design for a single writer so loads and stores are enough, prefer a dedicated RMW over a CAS loop, keep hot atomics on their own cache lines, and remember that on x86 the only memory order you ever really pay for is seq_cst on a store.