Skip to content

Day56 - POSIX Shared Memory IPC

Overview

Day56 focuses on Linux POSIX shared memory IPC architecture.

This lab starts from the minimal shm_open() + mmap() flow, then gradually extends into:

  • independent process shared memory
  • process-shared synchronization
  • AF_UNIX control channel
  • eventfd notification
  • epoll-driven IPC

The final architecture combines:

  • shared memory as the data path
  • eventfd as the notification path
  • UNIX domain socket as the control path

POSIX Shared Memory Basics

POSIX shared memory provides a mechanism for multiple processes to access the same memory pages.

Typical flow:

shm_open()
ftruncate()
mmap()

Unlike normal process memory:

process A memory != process B memory

shared memory explicitly maps the same kernel-backed pages into multiple processes.

Shared Memory Object Lifecycle

POSIX shared memory objects behave similarly to file objects.

Example:

/dev/shm/day56_shm

Typical lifecycle:

create -> map -> use -> unmap -> unlink

Important distinction:

  • close() releases fd references
  • munmap() removes virtual mapping
  • shm_unlink() removes the shared object name

Actual memory is released only when:

  • all mappings are gone
  • all references are closed
  • object has been unlinked

Fork-based Shared Memory

Initial demo:

parent
  └─ fork()
       └─ child

Both processes inherit:

  • shared memory mapping
  • shm fd

This demonstrates:

MAP_SHARED pages are truly shared

Changes from child become immediately visible to parent.

Independent Process Shared Memory

The architecture was later upgraded into:

writer process
reader process

without using fork() inheritance.

The writer:

  • creates shared memory
  • initializes shared data
  • owns object lifetime

The reader:

  • opens existing shared memory
  • attaches via mmap
  • reads shared data

This better reflects real Linux IPC architecture.

Process-shared pthread Mutex

Shared memory introduces race conditions.

A process-shared mutex was added:

pthread_mutexattr_setpshared(&attr,
                             PTHREAD_PROCESS_SHARED);

The mutex object itself must reside inside shared memory:

struct shm_data {
    pthread_mutex_t lock;
};

Important:

  • mutex must be initialized after mmap()
  • only the owner process initializes the mutex
  • all processes use the same mutex object

Reader Mapping Permissions

One important issue discovered during development:

pthread_mutex_lock()
requires writable mapping

Even a reader-only process must use:

PROT_READ | PROT_WRITE

because mutex internal state changes during lock/unlock.

Using only:

PROT_READ

caused glibc abort / segmentation fault.

AF_UNIX Control Channel

A UNIX domain socket was added as the IPC control path.

Socket roles:

writer = server
reader = client

Control socket responsibilities:

  • connection management
  • eventfd passing
  • future IPC extension

Socket types explored:

  • SOCK_STREAM
  • SOCK_SEQPACKET

SCM_RIGHTS File Descriptor Passing

The writer creates:

eventfd()

then passes the fd to the reader using:

SCM_RIGHTS
sendmsg()
recvmsg()

Important concept:

shared memory cannot transfer fd ownership
UNIX socket control channel is required

eventfd Notification Architecture

Final IPC architecture:

shared memory = data path
eventfd       = notification path
unix socket   = control path

Writer flow:

lock shared memory
update shared data
unlock shared memory
write(eventfd)

Reader flow:

epoll_wait()
read(eventfd)
lock shared memory
consume data
unlock shared memory

epoll-driven Shared Memory IPC

The reader was upgraded from polling:

sleep(1)
periodic read

into:

epoll_wait(eventfd)

This creates a fully event-driven IPC architecture.

Shared Activation State

A shared activation flag was added:

bool activation;

Writer termination flow:

activation = false
eventfd notify

Reader wakes up and exits gracefully.

Key Linux IPC Concepts Learned

Shared Memory

  • kernel-backed shared pages
  • independent process attachment
  • explicit synchronization required

pthread Mutex

  • process-shared synchronization
  • userspace synchronization object
  • futex-backed architecture

eventfd

  • event notification counter
  • epoll integration
  • lightweight userspace notification primitive

UNIX Domain Socket

  • control plane IPC
  • fd passing
  • process coordination

Final Architecture

                +------------------+
                |   shm_writer     |
                |------------------|
                | shared memory    |
                | pthread mutex    |
                | eventfd owner    |
                | UNIX socket srv  |
                +---------+--------+
                          |
                    SCM_RIGHTS
                          |
                          v
                +------------------+
                |   shm_reader     |
                |------------------|
                | epoll_wait()     |
                | eventfd consume  |
                | shared memory rd |
                +------------------+

Files

shm_common.h
fd_passing.h
epoll_helper.h
unix_socket_helper.h

shm_writer.c
shm_reader.c

Next Steps

Possible future extensions:

  • multi-reader architecture
  • ring buffer shared queue
  • lock-free shared structures
  • robust pthread mutex
  • semaphore-based synchronization
  • shared memory packet protocol
  • multi-client event routing
  • zero-copy IPC architecture