Skip to content

Day102 — DMA Descriptor Submission and Completion

Goal

Understand how a prepared DMA descriptor progresses from submission to hardware execution and completion.

The focus is the complete execution path of one asynchronous DMA transaction:

Prepare
  ↓
Submit
  ↓
Issue
  ↓
Schedule
  ↓
Execute
  ↓
Complete
  ↓
Callback

What I Learned

Descriptor Submission

A prepared descriptor is submitted with:

dmaengine_submit(desc);

dmaengine_submit() dispatches through the descriptor's tx_submit() callback.

For controllers using virt-dma, this callback is commonly implemented by vchan_tx_submit().

The submission path assigns a DMA cookie and places the descriptor into the submitted state.

The important distinction is:

Submission does not start DMA hardware execution.

dma_cookie_assign() assigns a transaction cookie during descriptor submission.

The cookie identifies the asynchronous transaction and allows its completion state to be tracked later.

The client driver normally receives the cookie from dmaengine_submit() rather than calling dma_cookie_assign() directly.

Issuing Pending Descriptors

After one or more descriptors have been submitted, the client driver calls:

dma_async_issue_pending(chan);

The framework dispatches through the controller's device_issue_pending() operation.

For controllers using virt-dma, vchan_issue_pending() moves descriptors from:

desc_submitted
      ↓
desc_issued

An issued descriptor is available for controller scheduling.

However:

Issued does not mean running.

A descriptor in desc_issued may still wait while another transaction is active on the DMA channel.

Pending and Active Work

virt-dma manages generic descriptor lists such as:

  • desc_allocated
  • desc_submitted
  • desc_issued
  • desc_completed

The active descriptor is typically maintained by controller-specific state rather than a generic virt-dma active list.

This gives the following ownership model:

virt-dma
    |
    | desc_issued
    v
Controller Driver
    |
    | active descriptor
    v
DMA Hardware

vchan_next_desc() Only Peeks

One important source-code detail was that vchan_next_desc() returns the descriptor at the head of desc_issued, but does not remove it from the list.

Conceptually:

vd = vchan_next_desc(vc);

means:

Look at next issued descriptor
            |
            v
Descriptor remains in desc_issued

The controller driver removes the descriptor when it actually consumes the transaction, commonly with:

list_del(&vd->node);

Therefore:

vchan_next_desc() selects by inspection; controller-specific scheduling logic performs the actual ownership transition.

DMA Completion

When the hardware finishes a transaction, the controller handles the completion interrupt.

For a virt-dma based controller, the completion path commonly reaches:

vchan_cookie_complete(vd);

This path performs several logically separate operations:

Hardware completion
        ↓
dma_cookie_complete()
        ↓
Descriptor completion state
        ↓
tasklet_schedule()
        ↓
Deferred completion processing
        ↓
Client callback

Hardware completion and callback execution are therefore not the same event.

dma_cookie_complete() records that the transaction has completed.

The important distinction is:

Cookie complete
      ≠
Callback executed

The DMA Engine can record the transaction as complete before the client callback runs.

tasklet_schedule() and Callback Execution

vchan_cookie_complete() schedules deferred completion processing with:

tasklet_schedule(&vc->task);

A tasklet is not equivalent to an RTOS task.

Calling tasklet_schedule() does not immediately execute the callback or cause a priority-based RTOS-style task switch.

Instead, it requests deferred execution through the kernel's tasklet/softirq mechanism.

Locking During Completion

vchan_cookie_complete() expects the virtual channel lock to already be held.

The lock assertion verifies this calling contract; the function does not release the lock itself.

Conceptually:

Controller IRQ handler
        |
        v
spin_lock(&vc->lock)
        |
        v
vchan_cookie_complete()
        |
        v
spin_unlock(&vc->lock)

The callback itself is deferred and executes outside the controller's original locked completion section.

A useful distinction is:

Scheduling deferred callback processing while holding the lock is not the same as executing the callback while holding the lock.

The Next DMA Transaction Can Start Before the Callback

Descriptor completion processing does not necessarily block hardware scheduling.

For two descriptors A and B, a valid ordering is:

Descriptor A running
        ↓
A completes
        ↓
Cookie A complete
        ↓
Callback A scheduled
        ↓
Descriptor B selected
        ↓
Descriptor B running
        ↓
Callback A executes

Therefore:

Descriptor B may already be running before Descriptor A's callback executes.

This separates DMA hardware scheduling from software completion notification.

Complete Mental Model

A DMA transaction crosses three major responsibility domains:

Client Driver
    |
    | prepare / submit / issue
    v
DMA Engine + Controller Driver
    |
    | queue / schedule / complete
    v
DMA Hardware
    |
    | execute transfer
    v
Controller Driver + DMA Engine
    |
    | completion processing
    v
Client Callback

The responsibilities can be summarized as:

  • The DMA Engine manages the transaction abstraction and common state.
  • The Controller Driver schedules transactions and manages hardware-specific active state.
  • The DMA Controller executes the actual transfer.
  • Completion processing records transaction completion and eventually notifies the client.

Key Takeaways

  • dmaengine_submit() submits a prepared descriptor but does not start the hardware.
  • tx_submit() provides the descriptor-specific submission implementation.
  • dma_cookie_assign() assigns the transaction cookie during submission.
  • dma_async_issue_pending() makes submitted descriptors available for controller scheduling.
  • Issued descriptors may still wait before becoming active.
  • vchan_next_desc() only peeks at the next issued descriptor.
  • The Controller Driver removes a descriptor from desc_issued when it consumes it.
  • Active descriptor state is normally controller-specific.
  • dma_cookie_complete() records transaction completion independently from callback execution.
  • tasklet_schedule() requests deferred kernel execution; it is not an RTOS-style task switch.
  • Callback execution occurs outside the controller's original locked completion section.
  • The next DMA transaction may start before the previous transaction's callback executes.