Skip to content

Day97 - Memory Barriers

Goal

Today I learned why modern CPUs and compilers may reorder memory accesses, and how Linux memory barriers guarantee the required ordering between CPUs, devices, and DMA engines.

Unlike previous topics that focused on memory allocation or virtual memory mapping, this topic focuses on memory ordering rather than memory address translation.

What I Learned

Compiler Reordering

Modern compilers may reorder instructions during optimization as long as the observable program behavior remains unchanged.

For example:

data = 100;
ready = 1;

may be reordered internally if no dependency exists.

A compiler barrier prevents the compiler from moving memory accesses across the barrier.

CPU Reordering

Even if instructions are emitted in program order, modern CPUs may execute or retire memory accesses out of order for better performance.

Examples include:

  • Store Buffer
  • Out-of-Order Execution
  • Speculative Execution

As a result, the order observed by another CPU or a device may differ from the original program order.

Memory Barriers

Memory barriers introduce ordering constraints that prevent specific memory accesses from being reordered.

Linux provides different types of barriers depending on the required ordering semantics.

Conceptually:

Program Order
        │
        ▼
Memory Barrier
        │
        ▼
Visible Memory Order

Linux Memory Barrier APIs

Linux provides several classes of memory barrier APIs.

Compiler barrier:

  • barrier()

CPU barriers:

  • mb()
  • rmb()
  • wmb()

SMP barriers:

  • smp_mb()
  • smp_rmb()
  • smp_wmb()

Acquire / Release:

  • smp_store_release()
  • smp_load_acquire()

Each API provides different ordering guarantees depending on the synchronization scenario.

MMIO Ordering

Memory ordering is also required when accessing device registers.

A common driver pattern is:

Write device configuration

↓

Ensure configuration is globally visible

↓

Write START register

Without a write barrier, the device may observe the START command before the configuration has become visible.

DMA Ordering

DMA descriptors are prepared in normal memory before notifying the device.

Typical driver sequence:

Fill DMA descriptor

↓

wmb()

↓

Write DMA doorbell register

The write barrier guarantees that the descriptor becomes visible before the DMA engine receives the notification.

Memory Ordering Simulator

To better understand ordering behavior, I implemented a simple Memory Ordering Simulator.

Instead of emulating CPU pipelines or hardware caches, the simulator visualizes:

  • Program Order
  • Visible Memory Order
  • Ordering Constraints introduced by Memory Barriers

The simulator intentionally demonstrates deterministic ordering behavior to illustrate how barriers affect observable memory access order.

Lab Summary

Implemented four lab cases:

  1. Store Reordering Without Barrier
  2. Store Reordering With Release Barrier
  3. DMA Doorbell Reordering Without Write Barrier
  4. DMA Doorbell Ordering With Write Barrier

The simulator demonstrates how Linux memory barriers preserve the required ordering between shared memory and device-visible operations.

Key Takeaways

  • Compiler order is not necessarily CPU-visible order.
  • Modern CPUs may reorder memory accesses for performance.
  • Memory barriers enforce ordering without changing program correctness.
  • Different barrier APIs provide different ordering guarantees.
  • Correct memory ordering is essential for SMP synchronization, MMIO, and DMA.