Skip to content

Day102 — DMA Descriptor Submission and Completion

Objective

Trace the Linux DMA Engine source code to understand how a prepared DMA descriptor moves through submission, scheduling, hardware execution, completion, and callback processing.

The goal is to build the complete execution path:

Prepare
  ↓
Submit
  ↓
Issue
  ↓
Schedule
  ↓
Execute
  ↓
Complete
  ↓
Callback

Lab Type

Kernel source walkthrough.

This lab does not require a custom kernel module or DMA hardware experiment.

The focus is understanding the interaction between:

  • DMA Engine framework
  • virt-dma
  • DMA controller driver
  • DMA hardware completion path

Part 1 — Trace Descriptor Submission

Start from:

dmaengine_submit()

Trace the call into:

desc->tx_submit(desc)

For a controller using virt-dma, inspect:

vchan_tx_submit()

Identify where:

  • The DMA cookie is assigned.
  • The descriptor leaves the allocated state.
  • The descriptor enters the submitted list.

Expected flow:

dmaengine_submit()
        ↓
desc->tx_submit()
        ↓
vchan_tx_submit()
        ↓
dma_cookie_assign()
        ↓
desc_submitted

Observation

dmaengine_submit() does not directly start DMA hardware execution.

Submission makes the prepared transaction part of the channel's pending work.

Part 2 — Trace dma_async_issue_pending()

Start from:

dma_async_issue_pending(chan)

Trace the controller operation:

device_issue_pending()

For a controller using virt-dma, inspect:

vchan_issue_pending()

Identify the descriptor-list transition:

desc_submitted
      ↓
desc_issued

Observation

dma_async_issue_pending() makes submitted descriptors available for controller scheduling.

It does not guarantee immediate hardware execution.

Therefore:

Issued does not mean running.

Part 3 — Inspect the Issued Descriptor

Inspect:

vchan_next_desc()

Determine whether the function removes the descriptor from desc_issued.

Observation

vchan_next_desc() returns the descriptor at the head of the issued list.

It does not remove the descriptor.

Conceptually:

desc_issued

+--------+    +--------+
| Desc A | -> | Desc B |
+--------+    +--------+
     ^
     |
vchan_next_desc()

Descriptor A remains linked into desc_issued.

Part 4 — Find the Ownership Transition

Follow a DMA controller driver that consumes the descriptor returned by vchan_next_desc().

Look for controller-specific code that removes the descriptor from the issued list, commonly using:

list_del(&vd->node);

The transition is:

virt-dma

desc_issued
     |
     | vchan_next_desc()
     | list_del()
     v

Controller Driver

active descriptor

Observation

The active descriptor is normally controller-specific state.

virt-dma does not require a generic desc_active list.

The controller driver owns the transition from an issued descriptor to an active hardware transaction.

Part 5 — Trace Hardware Execution

After the controller consumes the descriptor, identify where the controller:

  1. Extracts hardware-specific transaction information.
  2. Programs the DMA controller registers or descriptors.
  3. Starts the DMA transfer.
  4. Records the descriptor as active controller state.

Conceptually:

desc_issued
     ↓
Controller selects descriptor
     ↓
Controller-private active state
     ↓
Program DMA hardware
     ↓
DMA running

Observation

DMA Engine manages the transaction abstraction, while the controller driver translates that transaction into hardware-specific execution.

Part 6 — Trace DMA Completion

Follow the controller's DMA completion interrupt path.

Identify where the controller determines that the active descriptor has completed.

For a virt-dma based controller, trace the path into:

vchan_cookie_complete()

Then inspect:

dma_cookie_complete()

Identify:

  • Cookie completion.
  • Descriptor completion-list handling.
  • Deferred completion scheduling.

Expected conceptual flow:

DMA Hardware
      ↓
Completion IRQ
      ↓
Controller Driver
      ↓
vchan_cookie_complete()
      ↓
dma_cookie_complete()
      ↓
desc_completed
      ↓
tasklet_schedule()

Part 7 — Inspect the Locking Contract

Inspect the locking expectations around:

vchan_cookie_complete()

Determine:

  • Who acquires the virtual channel lock.
  • Whether vchan_cookie_complete() releases the lock.
  • When the caller releases the lock.

Expected pattern:

spin_lock(&vc->lock)
        ↓
vchan_cookie_complete()
        ↓
spin_unlock(&vc->lock)

Observation

The lock assertion inside vchan_cookie_complete() verifies that the caller already holds the lock.

It does not release the lock.

Part 8 — Trace Deferred Callback Processing

Inspect:

tasklet_schedule(&vc->task)

Follow the virt-dma completion tasklet and determine where the completed descriptors are processed and the client callback is eventually invoked.

Conceptually:

Completion IRQ
      ↓
vchan_cookie_complete()
      ↓
tasklet_schedule()
      ↓
IRQ path returns
      ↓
Deferred completion processing
      ↓
Client callback

Observation

tasklet_schedule() schedules deferred kernel execution.

It does not synchronously execute the callback at the call site.

This also means that scheduling the tasklet while holding the channel lock does not mean the client callback executes while that lock is held.

Part 9 — Compare Two Consecutive Transactions

Consider two descriptors:

Descriptor A
Descriptor B

Trace the possible ordering when Descriptor A completes while Descriptor B is already issued.

Expected sequence:

A running
    ↓
A hardware completion
    ↓
Cookie A complete
    ↓
Callback A scheduled
    ↓
Controller selects B
    ↓
B starts running
    ↓
Deferred completion processing
    ↓
Callback A executes

Observation

Descriptor B can begin hardware execution before Descriptor A's callback executes.

Hardware scheduling and client callback processing are separate paths.

Final Execution Path

After completing the source walkthrough, the complete DMA transaction can be summarized as:

Client Driver
    |
    | prepare
    v
DMA Descriptor
    |
    | dmaengine_submit()
    v
Submitted
    |
    | dma_async_issue_pending()
    v
Issued
    |
    | controller scheduling
    v
Controller Active
    |
    | program hardware
    v
DMA Hardware
    |
    | completion IRQ
    v
Controller Completion
    |
    | dma_cookie_complete()
    v
Transaction Complete
    |
    | deferred processing
    v
Client Callback

Key Findings

  • dmaengine_submit() submits a descriptor but does not start DMA hardware execution.
  • dma_cookie_assign() assigns the transaction cookie during submission.
  • dma_async_issue_pending() moves work toward controller scheduling.
  • An issued descriptor may still wait before becoming active.
  • vchan_next_desc() only peeks at the next issued descriptor.
  • The controller driver removes the descriptor when it consumes the transaction.
  • Active descriptor state is controller-specific.
  • Hardware completion and cookie completion are distinct from callback execution.
  • tasklet_schedule() requests deferred completion processing.
  • The client callback executes outside the controller's original locked completion section.
  • The next DMA transaction may start before the previous transaction's callback executes.