Memory Allocation in Multithreaded C++: malloc, Memory Pools, and Ring Buffers

It all starts with a classic question: in a system where thread A continuously allocates memory and constructs objects, while thread B continuously consumes and destroys them, how should you design the threading model for this scenario? You can probably answer right away: a simple SPSC ring buffer solves it. But the trade-offs behind that answer are worth a closer look.

Ref

• glibc Wiki
• Mimalloc: Free List Sharding in Action

Just malloc It, Just free It

Without swapping the native malloc (on Linux, usually glibc's ptmalloc2) for an allocator like mimalloc, calling malloc repeatedly on the hot path is generally expensive. The main costs are lock contention, metadata management on the slow path, and the occasional memory request that falls through to a system call. In high-frequency scenarios — say, sending a trading event — if every event triggers a malloc, some processing, and a free, your latency numbers across the board will likely look disappointing compared to strategies like memory pooling. That said, malloc has its virtues: it's always simple, always thread-safe, with all the internal machinery wrapped up for you. If you just want to write C++ as a "faster, strongly typed Python," this approach is perfectly fine.

Shared Memory Pool

Let's jump straight to a shared memory pool with a recycling mechanism (without recycling, it's really no different from a plain vector). A shared pool has a few paths for handling concurrency under multithreaded allocation. First, locking: the simplest option, and as everyone knows, slow — skip. Second, relying on atomic CAS operations and memory ordering — but CAS can't save you here either. (From here on, I'll assume you're already familiar with SPSC buffers.) Unlike an SPSC buffer, which guarantees one thread only writes the head and the other only writes the tail, a shared pool's free list is fundamentally an intrusive linked-list stack with a single shared variable: the head pointer. Thread A reads and writes head, and so does thread B. Cache line padding can't rescue you here — they're contending over the same variable, so the cache line ping-pong is unavoidable. And the fiercer the contention, the higher the CAS failure rate, further amplifying latency spikes. As a side note, this kind of lock-free stack also has the classic ABA problem to deal with — the engineering complexity is far higher than it looks.

struct Node { Node* next; /* payload... */ };

std::atomic<Node*> head;

Node* acquire() { // Called by A, B
    Node* old = head.load();
    while (old &&
           !head.compare_exchange_weak(old, old->next));
    return old;
}

void release(Node* n) { // Called by A, B
    n->next = head.load();
    while (!head.compare_exchange_weak(n->next, n))
        ;
}

TLS (Thread-Local Storage) Memory Pool

A TLS pool solves the "shared" problem: instead of one pool of memory, every thread gets its own. When a thread needs to allocate, it takes only from its own pool and touches only its own head pointer — no more cache line ping-pong. It also scales beautifully: adding more threads doesn't change the architecture at all.

Functionally, though, TLS is actually weaker than a shared pool: TLS assumes objects don't flow across threads — whatever a thread allocates, that same thread frees. For our producer A and consumer B, imagine B returning freed objects to B's own TLS pool: A's pool only drains, constantly requesting fresh resources from upstream; B's pool only fills, accumulating without bound. The result: memory gets hoarded to death in B while A keeps hitting the slow path. This kind of unbounded memory growth caused by "allocation and deallocation happening on different threads" is what the Hoard paper calls "blowup," and the mimalloc paper specifically addresses it in its discussion of cross-thread frees.

TLS Pool + Remote-Free List

So the next step reframes the problem as: "how do objects freed by other threads get returned to the original allocating thread's pool?" The common, general-purpose solution is to attach a remote-free list to each thread's pool, designed as MPSC: other threads use CAS to push freed nodes onto the allocating thread's list, and when A's local pool runs dry, it detaches the entire chain in one shot (a single XCHG) and recycles it in bulk. mimalloc's thread-free list follows exactly this idea; tcmalloc's transfer cache, while structurally different (it's a bulk-transfer layer between the thread cache and the central cache), shares the same principle of amortizing cross-thread contention through batched transfers. The cost is that the free path still involves cross-thread atomic operations — but the contention is amortized, rather than ping-ponging once per resource like the shared pool. This design is also more flexible: the thread topology it supports isn't limited to this article's SPSC setup — it allows resources allocated by one thread to be freed by any other thread.

struct ThreadPool {
    Node* local_free = nullptr;      // local
    std::atomic<Node*> remote_free{nullptr};  // remote
};

// Called by other threads
void remote_release(ThreadPool& owner, Node* n) {
    n->next = owner.remote_free.load();
    while (!owner.remote_free.compare_exchange_weak(n->next, n))
        ;
}

// Called by original thread: Free the entire list
Node* ThreadPool_refill(ThreadPool& p) {
    p.local_free = p.remote_free.exchange(nullptr);
    return p.local_free;
}

SPSC Ring Buffer

The SPSC buffer is the easiest to write and the best-performing option here — as its name says, it's built specifically for the SPSC scenario. Under the hood it's just a fixed-length contiguous buffer (a C array or vector) plus head and tail pointers represented as indices: the producer only writes the tail, the consumer only writes the head, and the gap between them is kept smaller than the buffer size. Of course, many popular implementations also support resizing, and their concurrency designs are broadly similar (for instance, MoodyCamel's size_approx() can only return an "approximate" size). For allocation, the producer can construct objects in place at the tail; the consumer calls pop to take out a copy, advances the head pointer, and on the next round that slot simply gets overwritten with new data. Just watch the buffer size — in practice, you can load-test against real production traffic and size the capacity accordingly.

MPSC, MPMC

These variants exist to handle different scenarios. In practice, though, MPMC offers the worst bang for the buck — many problems can be solved by composing multiple MPSC/SPSC queues instead (intrusive MPSC queues are common building blocks for this). Be extra careful when using SPSC: each end must have exactly one thread touching it. A common pattern is assembling multiple SPSC/MPSC queues into a unified cache, where different externally exposed interfaces route to different internal SPSC queues.

mimalloc vs ptmalloc2

Back to skipping the memory pool and just calling malloc: under this article's thread topology, mimalloc usually performs better. The topic of malloc strategies deserves a blog post of its own — here I'll only point out how the two differ in multithreaded allocation. ptmalloc2 spreads contention across arenas: each thread grabs an arena via trylock, but inside an arena everything is locked — even the optimized paths take a lock. And a cross-thread free requires locking the arena that owns that memory (i.e., some other thread's arena). mimalloc, by contrast, is built around TLS: the local allocation path has no locks — not even atomic operations — and cross-thread frees go through its thread-free list mechanism. That's why it performs so well under this thread topology.

In the end

It's been a long time since I last hand-rolled a malloc—