Skip to content

Day65 - Linux Kernel RCU APIs and Usage

Topic

Linux Kernel RCU API usage and deferred reclamation model.

Goals

  • Understand how Linux RCU APIs map to the Mini RCU model.
  • Understand the meaning of rcu_read_lock() and rcu_read_unlock().
  • Understand why rcu_dereference() and rcu_assign_pointer() are paired.
  • Understand the difference between synchronize_rcu() and call_rcu().
  • Understand how struct rcu_head enables callback-based deferred reclamation.

Completed Labs

Lab1 - RCU API Usage Simulation

Implemented a simplified userspace model of:

rcu_read_lock();
rcu_read_unlock();
rcu_dereference();
rcu_assign_pointer();
synchronize_rcu();

The lab demonstrated the blocking RCU reclamation flow:

publish new object
    ↓
wait for grace period
    ↓
free old object

Lab2 - call_rcu() Deferred Reclamation Simulation

Implemented:

struct rcu_head {
    struct rcu_head *next;
    void (*func)(struct rcu_head *head);
};

and simulated:

call_rcu(&old->rcu, rcu_object_free_cb);

The lab demonstrated that call_rcu() is non-blocking. The writer only enqueues the old object, while a reclaimer thread later waits for a grace period and executes the callback.

Key Understanding

RCU is not a lock.

RCU separates three operations:

  1. Object publication
  2. Reader-side access
  3. Object reclamation

The main API mapping is:

Concept API
Enter RCU read-side critical section rcu_read_lock()
Leave RCU read-side critical section rcu_read_unlock()
Safely load an RCU-protected pointer rcu_dereference()
Safely publish a new pointer rcu_assign_pointer()
Blocking grace period wait synchronize_rcu()
Non-blocking deferred reclamation call_rcu()

Important Notes

rcu_dereference() is not just a normal pointer load.

It provides acquire semantics so that when a reader observes a published pointer, it also observes the initialized contents of the object.

rcu_assign_pointer() is not just a normal pointer store.

It provides release semantics so that object initialization happens before the pointer becomes visible to readers.

Lab Observations

In Lab1, readers continued to safely use the old object after the writer published a new object. The writer waited for synchronize_rcu() before freeing the old object.

In Lab2, the writer called call_rcu() and exited immediately. The old object was later freed by the reclaimer thread after the grace period completed.

This confirms the difference between:

API Behavior
synchronize_rcu() Blocking reclamation
call_rcu() Asynchronous reclamation

Design Limitation

The labs used a simplified global reader counter:

atomic_fetch_add(&g_reader_count, 1);
atomic_fetch_sub(&g_reader_count, 1);

This is useful for learning, but it is not how Linux Kernel RCU scales.

A global reader counter would cause cacheline contention on multi-core systems. Real Linux RCU uses quiescent state tracking and per-CPU state instead.

Summary

Day65 completed the transition from Mini RCU concepts to Linux Kernel RCU API usage.

The most important conclusion is:

RCU is a publication and reclamation framework.
It allows readers to access old objects safely while writers publish new versions.
Old objects are reclaimed only after a grace period.

Next

Day66 will focus on Linux Kernel RCU internals:

  • Why global reader counters do not scale
  • Quiescent state concept
  • Per-CPU RCU state
  • Grace period detection
  • Context switch as quiescent state
  • High-level Tree RCU architecture