Day104 - DMA Termination and Synchronization¶

Today's Goal¶

Understand how the Linux DMA Engine framework terminates outstanding DMA transactions and establishes a safe synchronization boundary before client resources are released.

The main goals were:

Understand why stopping DMA hardware is not sufficient for safe resource cleanup.
Understand the difference between DMA termination and synchronization.
Study dmaengine_terminate_async().
Study dmaengine_synchronize().
Study dmaengine_terminate_sync().
Understand the corresponding struct dma_device controller operations.
Understand descriptor state during termination.
Analyze races between DMA completion, callbacks, and termination.
Understand client resource lifetime during DMA shutdown.
Trace a real virt-dma-based controller termination path.

What I Learned¶

Termination and Synchronization Solve Different Problems¶

DMA termination and synchronization operate on different parts of the DMA lifecycle.

Termination stops or retires outstanding DMA transactions:

DMA Running
    |
    +── Normal ─────────→ Hardware Completion
    |
    └── Terminate ──────→ Stop / Abort Hardware
                              ↓
                       Retire Outstanding
                         DMA Transactions

However, stopping the hardware does not prove that all software completion activity has finished.

A completion interrupt, deferred completion handler, or callback may already be pending or running.

Therefore, safe cleanup requires a separate synchronization boundary:

Terminate DMA
      ↓
Outstanding DMA Work Stopped or Retired
      ↓
Completion Processing May Still Exist
      ↓
Synchronize
      ↓
DMA-Related Asynchronous Activity Quiescent
      ↓
Safe Client Cleanup

The central mental model is:

Termination stops the DMA transaction lifecycle; synchronization closes the remaining asynchronous execution lifetime before client resources are reclaimed.

`dmaengine_terminate_async()`¶

dmaengine_terminate_async() requests termination of outstanding DMA activity without establishing a complete synchronization boundary for client cleanup.

Conceptually:

dmaengine_terminate_async()
        ↓
device_terminate_all()
        ↓
Controller-Specific Termination
        ↓
return

A successful return indicates that the controller termination operation returned successfully.

It does not guarantee that:

Deferred completion processing has finished.
A pending callback has executed.
A running callback has returned.
Callback parameters or other client resources are safe to release.

This makes asynchronous termination useful when termination must be initiated separately from later synchronization and cleanup.

`dmaengine_synchronize()`¶

dmaengine_synchronize() invokes the controller driver's synchronization operation:

dmaengine_synchronize()
        ↓
device_synchronize()

The exact implementation is controller-specific.

Depending on the provider, synchronization may involve:

Waiting for required hardware quiescence.
Synchronizing interrupt or deferred completion processing.
Waiting for a running callback.
Preventing pending deferred callbacks from executing.
Reclaiming terminated descriptors.

The important client-facing property is that the relevant old DMA completion activity has reached the provider-defined quiescent point before cleanup continues.

`dmaengine_terminate_sync()`¶

dmaengine_terminate_sync() combines termination and synchronization into a synchronous cleanup operation.

Conceptually:

dmaengine_terminate_sync()
        │
        ├── dmaengine_terminate_async()
        │
        └── dmaengine_synchronize()
        │
        ▼
return

This is useful when the caller can perform the complete termination and synchronization sequence immediately.

When the current execution context cannot safely perform the required synchronization, the termination and synchronization phases can instead be separated.

Descriptor State During Termination¶

Termination may occur while descriptors are at different lifecycle stages.

Descriptor State	Main Concern During Termination
Submitted	Retire queued work before it can be issued.
Issued	Prevent queued work from becoming active later.
Active	Stop or abort the hardware transfer and retire the transaction.
Completed / callback pending	The transaction may already be complete, but software completion activity may still require synchronization.

The exact descriptor lists and state transitions are controller-specific.

A generic DMA Engine client should not assume that every provider uses the same internal descriptor queues.

Completion and Termination Can Race¶

A DMA transfer may complete at nearly the same time that another execution context begins termination.

Conceptually:

Termination Path                 DMA IRQ / Completion Path

terminate
    │
    │                           Hardware Completion
    │                                  │
    └──────────── race ────────────────┘

The controller driver must serialize the relevant descriptor state transitions so that a descriptor is not incorrectly processed through conflicting completion and termination paths.

Locking may protect:

Descriptor ownership.
Queue membership.
Active descriptor identity.
Completion decisions.
Termination decisions.

However, locking does not replace synchronization.

A callback may execute outside the descriptor lock, so acquiring the same lock in the termination path does not prove that an already scheduled or running callback has quiesced.

Locking
    ↓
Descriptor and State Consistency


Synchronization
    ↓
Asynchronous Execution Lifetime and Quiescence

Callback Behavior During Synchronization¶

The virt-dma path studied uses tasklet-based deferred completion processing.

If a tasklet containing a pending callback has not started when synchronization kills the tasklet, that callback does not need to execute.

If the callback is already running, synchronization waits for that execution to finish.

Therefore:

Synchronization guarantees callback quiescence, not that every previously scheduled callback will execute.

This reinforces the Day103 distinction between hardware completion and callback execution.

Client Lifecycle Must Prevent New DMA Work¶

Termination does not automatically prevent the DMA client itself from creating new transactions.

This becomes especially important when a completion callback prepares or submits another DMA descriptor.

The client should first transition into its own stopping state:

Client RUNNING
      ↓
Client STOPPING
      ↓
Prevent New DMA Submissions
      ↓
Terminate Outstanding DMA
      ↓
Synchronize
      ↓
Client STOPPED

DMA transaction state and client operational state represent different concepts.

For example, dmaengine_tx_status() describes the state of a DMA transaction. It does not answer whether the client driver currently allows another DMA transaction to be submitted.

A client-specific state such as RUNNING, STOPPING, or STOPPED is therefore useful when callback-driven resubmission must be controlled.

Safe Client Resource Cleanup¶

A DMA descriptor may indirectly reference several client resources:

DMA Descriptor
    │
    ├── DMA Buffer
    ├── Callback
    └── Callback Parameter
            ↓
       Client Context

Stopping DMA hardware protects against future hardware accesses, but deferred completion processing may still access the DMA buffer or client context.

The safe cleanup ordering is therefore:

Prevent New DMA Work
        ↓
Terminate Outstanding DMA
        ↓
Synchronize Completion Activity
        ↓
Release DMA Buffers
        ↓
Release Callback / Client State
        ↓
Release DMA Channel

The exact client resource cleanup order depends on the driver, but every resource must remain valid until all asynchronous execution paths that may reference it have quiesced.

virt-dma Source Analysis¶

The virt-dma helpers provide one implementation model for descriptor termination and deferred completion synchronization.

Important helpers studied included:

vchan_next_desc()
vchan_terminate_vdesc()
vchan_get_all_descriptors()
vchan_synchronize()

A terminated descriptor may be moved into desc_terminated instead of being destroyed immediately.

This reinforces an important lifecycle distinction:

Transaction Terminated
        ≠
Descriptor Object Destroyed

Descriptor reclamation may occur later during synchronization.

virt-dma is an implementation helper and should not be treated as mandatory behavior for every DMA controller driver.

XDMA Controller Source Analysis¶

A Xilinx XDMA controller path was studied as a concrete virt-dma-based implementation example.

The termination path first stops the hardware and then processes software descriptor state.

The front descriptor in the issued list receives special handling:

Front of desc_issued
        ↓
Remove from Current List
        ↓
dma_cookie_complete()
        ↓
vchan_terminate_vdesc()
        ↓
desc_terminated

The remaining virt-dma descriptor lists are then collected and moved into the termination-side descriptor state.

One important source observation was:

dma_cookie_complete(&vd->tx);
vchan_terminate_vdesc(vd);

These operations are not contradictory.

dma_cookie_complete() updates the generic cookie bookkeeping so that the transaction is no longer tracked as outstanding through the normal cookie mechanism.

vchan_terminate_vdesc() moves the descriptor object into the termination lifecycle.

Therefore:

Cookie Completion
        ≠
Normal Successful DMA Completion
        ≠
Callback Execution
        ≠
Descriptor Reclamation

The use of dma_cookie_complete() in this termination path is an implementation detail of the controller path studied and must not be generalized to every DMA controller.

XDMA Synchronization Source Analysis¶

The XDMA synchronization path also demonstrated that device_synchronize() is not necessarily limited to software callback synchronization.

The implementation may perform controller-specific synchronization before using the virt-dma synchronization helper.

Conceptually:

device_synchronize()
        ↓
Controller-Specific Synchronization
        │
        ├── Hardware Quiescence, if required
        ├── IRQ / Deferred Processing Quiescence
        ├── Callback Quiescence
        └── Descriptor Reclamation, if applicable
        ↓
QUIESCENT

These operations do not represent a generic fixed ordering.

The exact synchronization mechanism and ordering are provider-specific.

Reusable Diagram¶

The termination and synchronization lifecycle is summarized by:

docs/assets/diagrams/kernel-driver/dma/
├── dma-termination-synchronization-lifecycle.drawio
└── dma-termination-synchronization-lifecycle.svg

The diagram emphasizes the boundary between termination and synchronization and shows that client cleanup occurs only after old DMA-related asynchronous activity has quiesced.

Important Observations¶

Stopping DMA hardware is not sufficient to establish safe client resource cleanup.
DMA termination and completion synchronization are separate lifecycle operations.
dmaengine_terminate_async() does not by itself establish a safe cleanup boundary.
A successful asynchronous termination return does not guarantee that callbacks or deferred completion processing have quiesced.
dmaengine_synchronize() establishes the synchronization boundary required before releasing resources referenced by old DMA activity.
dmaengine_terminate_sync() combines termination and synchronization.
Submitted, issued, active, and completed descriptors have different concerns during termination.
Completion and termination may race.
Controller locking protects descriptor state transitions but does not replace callback synchronization.
A pending callback may be prevented from executing, while a callback already running must finish before synchronization returns.
The client must stop producing new DMA transactions before termination begins.
DMA transaction state and client operational state are separate concepts.
Client callback parameters and DMA buffers must remain valid until relevant asynchronous activity has quiesced.
Transaction termination does not imply immediate descriptor destruction.
Cookie completion does not by itself mean normal successful DMA completion.
virt-dma provides useful termination and synchronization helpers but is not mandatory for all DMA controller drivers.
Controller-specific source behavior must not be generalized into DMA Engine framework guarantees.

Lab¶

No Day104 lab was created.

The work focused on DMA Engine termination semantics, synchronization boundaries, concurrency races, resource lifetime, and Linux kernel source analysis.

A synthetic lab would not accurately reproduce the controller-specific hardware abort, interrupt, deferred completion, and synchronization behavior studied during the session.

Summary¶

Day104 completed the shutdown side of the DMA transaction lifecycle.

The key lifecycle is:

Client STOPPING
        ↓
Prevent New DMA Work
        ↓
DMA Termination
        ↓
Outstanding Transactions Retired
        ↓
DMA Synchronization
        ↓
Completion Activity Quiescent
        ↓
Safe Client Cleanup

The most important distinction is that termination stops outstanding DMA transaction activity, while synchronization ensures that remaining asynchronous completion activity can no longer race with client resource cleanup.

Next Plan¶

Day105 will study:

Cyclic DMA and Periodic Callbacks

The next topic will focus on:

Cyclic DMA descriptors.
Period-based DMA operation.
Period callbacks.
The difference between periodic callbacks and normal transaction completion.
Cyclic DMA termination behavior.
Relevant DMA Engine APIs and controller responsibilities.

Day104 - DMA Termination and Synchronization¶

Today's Goal¶

What I Learned¶

Termination and Synchronization Solve Different Problems¶

dmaengine_terminate_async()¶

dmaengine_synchronize()¶

dmaengine_terminate_sync()¶

Descriptor State During Termination¶

Completion and Termination Can Race¶

Callback Behavior During Synchronization¶

Client Lifecycle Must Prevent New DMA Work¶

Safe Client Resource Cleanup¶

virt-dma Source Analysis¶

XDMA Controller Source Analysis¶

XDMA Synchronization Source Analysis¶

Reusable Diagram¶

Important Observations¶

Lab¶

Summary¶

Next Plan¶

`dmaengine_terminate_async()`¶

`dmaengine_synchronize()`¶

`dmaengine_terminate_sync()`¶