VxWorks Cache Coherency: Fixing DMA Issues on MPC860

VxWorks Cache Coherency: Fixing DMA Issues on MPC860

Cache coherency is a critical consideration when developing device drivers for the VxWorks real-time operating system, particularly on embedded processors where the CPU and DMA-capable peripherals share physical memory without hardware cache snooping.

The Motorola MPC860 PowerQUICC family provides a representative example. Its PowerPC-compatible processor core, integrated communications hardware, and Fast Ethernet Controller (FEC) on supported variants rely on shared-memory structures to exchange data.

When the CPU’s data cache is enabled, the processor and DMA hardware may observe different versions of the same memory.

This can cause difficult-to-reproduce failures, including corrupted Ethernet frames, stale receive data, invalid buffer descriptor states, and intermittent communication errors.

The fundamental solution is to establish explicit memory ownership and ensure that CPU caches and DMA-visible memory are synchronized at every ownership transition.

VxWorks provides cache-management APIs such as cacheFlush(), cacheInvalidate(), cacheDmaMalloc(), and cacheDmaFree() to support these requirements.

This article explains the underlying problem, compares cache-safe memory allocation with explicit cache maintenance, and demonstrates practical techniques for managing transmit and receive buffers in MPC860 FEC drivers.

The implementation concepts primarily target classic VxWorks releases and MPC860-family hardware. Modern VxWorks 7 systems may require different DMA allocation, address translation, and cache synchronization interfaces.

🧠 1. Understanding Cache Coherency in VxWorks

#

1.1 Why CPU Caches Create DMA Problems

#

Modern processors use caches to reduce memory access latency.

Instead of fetching every value directly from main memory, the processor stores recently accessed data in a smaller, faster cache.

The MPC860 family includes separate instruction and data caches.

For device-driver development, the data cache is particularly important because peripheral devices may access the same memory independently through direct memory access (DMA).

Consider the following architecture:

+--------------------------+
|          CPU             |
|                          |
|     +--------------+     |
|     | Data Cache   |     |
|     +--------------+     |
+-------------+------------+
              |
              |
+-------------v------------+
|       Main Memory        |
|                          |
|   Shared DMA Buffers     |
|   Buffer Descriptors     |
+-------------+------------+
              |
              |
+-------------v------------+
|       DMA Controller     |
|                          |
|    Ethernet FEC / CPM    |
+--------------------------+

The CPU normally accesses memory through its cache hierarchy.

A DMA controller, however, reads or writes memory through its own hardware path.

If the hardware does not maintain cache coherency automatically, the CPU cache and main memory can contain different values.

This creates two fundamental problems.

Transmit Data Inconsistency

The CPU modifies a transmit buffer, but the updated data remains in the cache.

The DMA controller reads the older contents from main memory.

Receive Data Inconsistency

The DMA controller writes a newly received packet into main memory, but the CPU continues reading an outdated copy from its cache.

Both problems can occur even when the driver uses valid pointers and correctly configured DMA descriptors.

1.2 Write-Through Cache Mode

#

In write-through mode, CPU writes are propagated to the cache and the underlying memory system.

Conceptually:

CPU Write
    |
    +----> Data Cache
    |
    +----> Main Memory

This reduces the possibility of DMA reading stale CPU-generated data.

However, write-through caching does not automatically solve receive-side coherency.

When DMA writes directly into memory, previously cached data may remain valid from the CPU’s perspective even though the underlying memory has changed.

Consequently, a CPU read may return stale data unless the relevant cache lines are invalidated.

Write-through behavior must also be considered alongside write buffers, synchronization instructions, and the processor’s memory-ordering rules.

1.3 Copy-Back Cache Mode

#

In copy-back mode, CPU writes initially update cached data.

Main memory may not receive the changes until the cache line is written back.

Conceptually:

CPU Write
    |
    v
+-------------------+
| Data Cache        |
| Modified / Dirty  |
+-------------------+
    |
    | Later write-back
    v
+-------------------+
| Main Memory       |
+-------------------+

This creates potential coherency problems in both directions.

For transmit operations, the DMA controller may read stale memory because the CPU’s modifications remain cached.

For receive operations, the CPU may read stale cached data after DMA updates main memory.

An additional risk exists when dirty cache lines overlap a DMA receive buffer. A later write-back could overwrite data that the peripheral has already written.

Therefore, copy-back configurations require careful coordination of cache flushing, invalidation, memory alignment, and DMA ownership.

1.4 Why the MPC860 Requires Software Cache Management

#

The MPC860 family does not provide general hardware cache coherence for external DMA transactions.

The processor therefore depends on software-managed cache maintenance or cache-inhibited memory mappings for shared DMA data.

This is particularly important when working with:

• Fast Ethernet Controller transmit buffers.
• Fast Ethernet Controller receive buffers.
• Transmit and receive buffer descriptors.
• Communications Processor Module resources.
• External DMA-capable peripherals.
• Shared memory regions accessed by multiple hardware agents.

A driver cannot assume that CPU reads and DMA accesses automatically observe identical memory contents.

For additional background on VxWorks device-driver architecture and DMA synchronization, see VxWorks 5.4 Device Driver Development: Character and Block Drivers.

⚙️ 2. General Solutions for VxWorks Cache Coherency

#

Two common strategies address DMA-related cache coherency problems:

1. Allocating memory with DMA-safe cache attributes.
2. Using explicit cache maintenance for cacheable memory.

The appropriate strategy depends on the processor, BSP, hardware DMA requirements, and performance constraints.

2.1 Using Non-Cacheable DMA Buffers

#

One straightforward approach is to use a memory region that the CPU accesses without data caching.

When shared memory is mapped as cache-inhibited, CPU operations access the underlying memory system rather than retaining ordinary cached copies.

This avoids many software cache maintenance requirements.

Advantages include:

• Simpler CPU–DMA synchronization.
• No dirty data cache lines for the non-cacheable region.
• Predictable descriptor visibility.
• Reduced risk of stale cached descriptor fields.

Disadvantages include:

• Higher CPU memory-access latency.
• Potentially reduced packet-processing performance.
• Increased external memory traffic.
• Additional BSP memory-mapping requirements.

Non-cacheable memory is especially useful for small control structures that are frequently exchanged between the CPU and DMA hardware.

2.2 Configuring Non-Cacheable Memory in the BSP

#

In classic VxWorks BSPs, physical memory mappings and cache attributes may be configured through the system memory descriptor table.

A common location is:

sysLib.c

The relevant data structure is often:

sysPhysMemDesc[]

The following example illustrates the general structure of a non-cacheable memory mapping.

/*
 * Illustrative VxWorks BSP memory descriptor.
 *
 * Region addresses and attribute values must
 * be adapted to the actual BSP and processor.
 */

#define DMA_REGION_BASE  0x02000000
#define DMA_REGION_SIZE  0x00010000

PHYS_MEM_DESC sysPhysMemDesc[] =
{
    {
        (void *)DMA_REGION_BASE,
        (void *)DMA_REGION_BASE,
        DMA_REGION_SIZE,

        VM_STATE_MASK_VALID |
        VM_STATE_MASK_WRITABLE |
        VM_STATE_MASK_CACHEABLE,

        VM_STATE_VALID |
        VM_STATE_WRITABLE |
        VM_STATE_CACHEABLE_NOT
    }
};

This is an illustrative descriptor entry, not a complete replacement for a BSP memory table.

The actual implementation must account for:

• MMU page-size requirements.
• Address alignment.
• Existing memory mappings.
• CPU-specific cache attributes.
• BSP-supported virtual-memory constants.
• Physical address accessibility by DMA hardware.

Changing memory attributes also requires ensuring that no conflicting cacheable aliases remain active.

On some systems, additional cache or TLB synchronization may be required when changing mappings dynamically.

2.3 Allocating DMA-Safe Memory with cacheDmaMalloc()

#

Classic VxWorks provides the following allocation interfaces:

void *cacheDmaMalloc(size_t size);

STATUS cacheDmaFree(void *pBuffer);

These functions are intended for memory shared with DMA-capable devices.

Example:

#include <vxWorks.h>
#include <cacheLib.h>
#include <stddef.h>

#define DMA_BUFFER_SIZE  1536

static void *dmaBuffer = NULL;

STATUS dmaBufferInit(void)
{
    dmaBuffer = cacheDmaMalloc(DMA_BUFFER_SIZE);

    if (dmaBuffer == NULL)
        return ERROR;

    return OK;
}

STATUS dmaBufferRelease(void)
{
    STATUS status;

    if (dmaBuffer == NULL)
        return OK;

    status = cacheDmaFree(dmaBuffer);

    if (status == OK)
        dmaBuffer = NULL;

    return status;
}

An important distinction is that DMA-safe allocation does not universally mean the returned memory is non-cacheable.

Depending on the VxWorks release, processor architecture, and BSP implementation, the allocator may return memory with different caching characteristics.

On systems without suitable MMU-based memory attributes, the implementation may use cacheable memory and require explicit cache management.

Drivers must inspect the target’s cache library behavior instead of assuming a particular allocation policy.

Additionally, a CPU-accessible pointer returned by cacheDmaMalloc() is not necessarily the physical or bus address required by a DMA controller.

Address translation must follow the target’s documented DMA mapping interfaces.

🔄 3. Explicit Cache Flush and Invalidate Operations

#

When DMA buffers remain cacheable, the driver must explicitly synchronize memory before transferring ownership between the CPU and peripheral.

3.1 cacheFlush(): Preparing Data for DMA Reads

#

The cacheFlush() operation makes modified CPU cache data visible to the underlying memory system.

Its classic VxWorks interface is:

STATUS cacheFlush
    (
    CACHE_TYPE cache,
    void *address,
    size_t bytes
    );

For a device that reads a memory buffer, the general sequence is:

1. CPU writes data into the buffer.
2. Driver flushes the relevant cache range.
3. Driver performs any required memory synchronization.
4. Driver transfers buffer ownership to the DMA controller.
5. DMA reads the updated data.

Example:

/*
 * Synchronize a transmit buffer before DMA.
 */

STATUS dmaPrepareTx
    (
    void *pBuffer,
    size_t length
    )
{
    if ((pBuffer == NULL) || (length == 0))
        return ERROR;

    if (cacheFlush(DATA_CACHE,
                   pBuffer,
                   length) != OK)
    {
        return ERROR;
    }

    return OK;
}

The cache operation must complete before the hardware is instructed to read the buffer.

On platforms with additional memory-ordering requirements, the driver may also need processor-specific synchronization instructions or BSP-provided barriers.

3.2 cacheInvalidate(): Reading Data Written by DMA

#

The cacheInvalidate() operation invalidates cached data associated with a memory region.

Subsequent CPU reads must obtain valid data through the memory hierarchy instead of using the discarded cache contents.

The classic interface is:

STATUS cacheInvalidate
    (
    CACHE_TYPE cache,
    void *address,
    size_t bytes
    );

For a completed DMA receive operation, the general sequence is:

1. DMA writes data into main memory.
2. Hardware indicates that the transfer is complete.
3. Driver performs required completion synchronization.
4. Driver invalidates the relevant CPU cache lines.
5. CPU reads the received data.

Example:

/*
 * Synchronize a received buffer after DMA.
 */

STATUS dmaCompleteRx
    (
    void *pBuffer,
    size_t length
    )
{
    if ((pBuffer == NULL) || (length == 0))
        return ERROR;

    if (cacheInvalidate(DATA_CACHE,
                        pBuffer,
                        length) != OK)
    {
        return ERROR;
    }

    return OK;
}

The driver must not invalidate dirty cache lines containing unrelated data.

For noncoherent DMA, receive buffers should be prepared before DMA ownership begins, and cache-line sharing with unrelated writable data should be avoided.

3.3 CACHE_DMA_FLUSH and CACHE_DMA_INVALIDATE

#

Classic VxWorks configurations may also provide DMA-oriented cache macros:

CACHE_DMA_FLUSH(address, bytes);

CACHE_DMA_INVALIDATE(address, bytes);

These interfaces use the platform’s DMA cache-management configuration.

They are commonly associated with memory managed through the DMA cache allocation mechanisms.

The relationship between the interfaces can be summarized as follows:

These functions and macros should not be treated as universally interchangeable.

Their behavior can vary by VxWorks version, architecture, and BSP.

Use the interface documented for the specific allocation method and DMA subsystem.

For modern VxWorks 7 driver design, see PCIe Device Driver Development on VxWorks 7, which discusses DMA memory allocation, address translation, and cache synchronization in the VxBus environment.

🏗️ 4. MPC860 Architecture and DMA Memory Organization

#

4.1 MPC860 PowerQUICC Overview

#

The Motorola MPC860 is a 32-bit embedded processor family based on the PowerPC architecture.

Major components include:

• PowerPC-compatible processor core.
• Instruction and data caches.
• Memory management units.
• System Integration Unit.
• Communications Processor Module.
• Dual-port internal RAM.
• Serial communication controllers.
• Fast Ethernet Controller on supported MPC860 variants.

The MPC860T and certain related variants include integrated Fast Ethernet functionality.

The device’s DMA-capable communication hardware can access memory independently of the CPU, making correct memory attributes and cache maintenance essential.

4.2 Shared Memory in the FEC Architecture

#

The MPC860T Fast Ethernet Controller communicates with software through control registers, buffer descriptors, and packet buffers.

The relevant memory architecture can be represented as:

+--------------------------------+
|          PowerPC CPU           |
|                                |
|        Data Cache              |
+---------------+----------------+
                |
                v
+--------------------------------+
|       External Main RAM        |
|                                |
|  TX Buffer Descriptor Ring     |
|  RX Buffer Descriptor Ring     |
|                                |
|  TX Packet Buffers             |
|  RX Packet Buffers             |
+---------------+----------------+
                ^
                |
+---------------+----------------+
|        MPC860T FEC             |
|                                |
|  Control / Status Registers    |
|  DMA Processing Logic          |
|  Ethernet MAC                  |
+--------------------------------+

A critical architectural distinction is that the classic MPC860T FEC driver places its transmit and receive buffer descriptor rings in external main memory.

The controller’s internal memory contains communication control resources and associated parameter information.

It is therefore inaccurate to assume that the FEC descriptor rings should always be placed inside the processor’s internal dual-port RAM.

The MPC860 family also has substantially less than 64 KB of actual internal dual-port RAM; the exact organization depends on the processor variant.

Descriptor location must follow the specific FEC architecture and driver requirements.

4.3 Transmit and Receive Buffer Descriptors

#

A Buffer Descriptor (BD) communicates packet-buffer information between software and the Ethernet controller.

A typical descriptor contains:

• Status and control flags.
• Data length.
• Buffer address.

Conceptually:

/*
 * Illustrative FEC buffer descriptor layout.
 *
 * Exact hardware layout, byte ordering, and
 * bit definitions must match the MPC860T manual.
 */

typedef struct
{
    volatile UINT16 status;
    volatile UINT16 length;
    volatile UINT32 bufferAddress;
} FEC_BD;

For the MPC860T FEC, transmit and receive descriptors are typically organized into rings.

The DMA controller and CPU update descriptor fields as ownership changes.

Because both agents access these structures, descriptors must be placed in a memory region with suitable DMA coherence properties.

4.4 Descriptor Alignment and Memory Attributes

#

The classic Motorola FEC END driver requires buffer descriptors to be placed in memory suitable for DMA access.

Relevant requirements include:

• Descriptor alignment.
• Valid DMA-visible addresses.
• Appropriate cache attributes.
• Correct buffer ownership.
• Proper descriptor status updates.
• Consistent endianness.

The classic motFecEnd driver documentation specifies 8-byte alignment and non-cacheable memory for descriptor storage.

When an appropriate preallocated descriptor region is not supplied, the driver may allocate the descriptor rings through cacheDmaMalloc().

These requirements are different from those of packet-data buffers, which may remain cacheable when correct flush and invalidate operations are implemented.

🧩 5. Recommended Cache Coherency Strategy for MPC860 FEC Drivers

#

A practical FEC driver should distinguish between descriptor memory and packet-data memory.

5.1 Keep Buffer Descriptors DMA-Coherent

#

Transmit and receive buffer descriptors are small control structures accessed frequently by both the CPU and FEC.

For the classic MPC860T FEC, use DMA-safe descriptor memory with the cache attributes required by the hardware and driver.

Possible allocation strategies include:

1. Reserving a non-cacheable region in external main memory through the BSP.
2. Using cacheDmaMalloc() when its target-specific implementation provides suitable descriptor memory.
3. Using another documented DMA-safe allocation mechanism supported by the platform.

The descriptor rings should remain allocated throughout normal driver operation.

5.2 Use Cacheable Memory for Packet Data

#

Packet-data buffers are substantially larger than descriptors.

For example, a conventional Ethernet frame may require a buffer of approximately 1,518 bytes without a VLAN tag or additional hardware-specific overhead.

Using cacheable memory for packet payloads can improve CPU performance.

A typical design is:

This arrangement balances correctness and performance.

However, the memory configuration must follow the actual BSP and FEC driver implementation.

5.3 Separate CPU and DMA Ownership

#

Every DMA buffer should have a clearly defined owner.

For transmit operations:

CPU owns buffer
       |
       v
CPU prepares packet
       |
       v
Flush cache
       |
       v
Transfer ownership to FEC
       |
       v
FEC transmits packet
       |
       v
FEC signals completion
       |
       v
CPU reclaims buffer

For receive operations:

CPU prepares RX buffer
       |
       v
Synchronize buffer for DMA
       |
       v
Transfer ownership to FEC
       |
       v
FEC writes received packet
       |
       v
FEC signals completion
       |
       v
Invalidate cache
       |
       v
CPU processes packet
       |
       v
Reinitialize RX descriptor

This ownership model prevents the CPU and DMA controller from modifying the same buffer simultaneously.

💻 6. Implementing Cache Coherency in an MPC860 FEC Driver

#

6.1 Transmit Path: Flushing Data Before DMA

#

In the transmit path, the CPU prepares an Ethernet frame and places it in a transmit buffer.

If the buffer is cacheable, its contents must be synchronized before the FEC reads the memory.

The original driver concept can be refined as follows:

#include <vxWorks.h>
#include <cacheLib.h>
#include <string.h>

/*
 * Illustrative MPC860T FEC transmit preparation.
 *
 * Requirements:
 * - pTxBuffer is DMA-accessible.
 * - txBd resides in DMA-coherent memory.
 * - dmaAddress is the validated FEC-visible address.
 * - The descriptor is currently CPU-owned.
 * - Hardware-specific status bits are defined by the BSP.
 */

STATUS fecPrepareTx
    (
    FEC_BD *txBd,
    void *pTxBuffer,
    UINT32 dmaAddress,
    const void *pData,
    size_t length
    )
{
    if (txBd == NULL ||
        pTxBuffer == NULL ||
        pData == NULL ||
        length == 0 ||
        length > 1518)
    {
        return ERROR;
    }

    /*
     * Copy packet data into the DMA buffer.
     */
    memcpy(pTxBuffer, pData, length);

    /*
     * Make CPU-written data visible to the FEC.
     */
    if (cacheFlush(DATA_CACHE,
                   pTxBuffer,
                   length) != OK)
    {
        return ERROR;
    }

    /*
     * Prepare the descriptor.
     *
     * This function does not transfer ownership
     * to hardware or activate transmission.
     */
    txBd->bufferAddress = dmaAddress;
    txBd->length = (UINT16)length;

    return OK;
}

The example deliberately separates packet preparation from hardware activation.

Before setting the transmit descriptor’s hardware-ownership bit, the driver must ensure:

• The descriptor fields are complete.
• The DMA address is valid.
• The packet buffer has been flushed.
• Required memory-ordering barriers have executed.
• The descriptor is not still owned by the FEC.

The final descriptor status update and transmit activation must use the correct MPC860T register definitions and ordering requirements.

6.2 Why Flushing Must Occur Before Transmission

#

Consider the following incorrect sequence:

/*
 * Incorrect DMA sequence.
 */

/* CPU modifies cached buffer. */
memcpy(pTxBuffer, pData, length);

/* Hardware DMA is activated too early. */
fecStartDma();

/* Cache flush happens after DMA starts. */
cacheFlush(DATA_CACHE, pTxBuffer, length);

The FEC may begin reading main memory before the CPU’s modified cache contents have been written back.

The device could transmit stale or partially updated data.

The correct ordering is:

/*
 * Correct conceptual ordering.
 */

memcpy(pTxBuffer, pData, length);

cacheFlush(DATA_CACHE, pTxBuffer, length);

/*
 * Perform required memory synchronization.
 * Then publish the descriptor and activate DMA.
 */
fecStartDma();

Here, fecStartDma() represents a BSP-specific hardware activation routine rather than a standard VxWorks API.

Real implementations must also guarantee that descriptor writes are visible before the device receives ownership.

6.3 Receive Path: Invalidating Data After DMA

#

The receive path requires the opposite synchronization operation.

The FEC writes an incoming Ethernet frame into a receive buffer.

Before the CPU processes the data, the driver must ensure that cached copies do not hide the DMA updates.

Example:

#include <vxWorks.h>
#include <cacheLib.h>

/*
 * Illustrative receive completion.
 *
 * Call only after hardware has completed
 * the DMA write and returned ownership
 * of the receive descriptor to the CPU.
 */

STATUS fecCompleteRx
    (
    void *pRxBuffer,
    size_t bufferCapacity,
    size_t receivedLength
    )
{
    if (pRxBuffer == NULL ||
        receivedLength == 0 ||
        receivedLength > bufferCapacity)
    {
        return ERROR;
    }

    /*
     * Discard stale cached copies of the
     * received data.
     */
    if (cacheInvalidate(DATA_CACHE,
                        pRxBuffer,
                        receivedLength) != OK)
    {
        return ERROR;
    }

    /*
     * The CPU may now safely inspect the
     * received frame, subject to the
     * platform's DMA synchronization rules.
     */

    return OK;
}

This function assumes the buffer was correctly prepared before being handed to the FEC.

In particular, dirty CPU cache lines must not remain capable of overwriting the DMA receive data.

The driver must also ensure that DMA has completed before accessing the buffer.

6.4 Preparing Receive Buffers Before DMA

#

Receive-buffer preparation is as important as receive completion.

A common safe sequence is:

1. Allocate a DMA-accessible buffer.
2. Ensure cache-line alignment.
3. Avoid sharing cache lines with unrelated writable data.
4. Perform the cache operation required before device ownership.
5. Program the receive descriptor.
6. Transfer ownership to the FEC.

Example:

/*
 * Illustrative receive-buffer preparation.
 *
 * The buffer must be dedicated to DMA and
 * satisfy the BSP's cache-line requirements.
 */

STATUS fecPrepareRx
    (
    void *pRxBuffer,
    size_t bufferSize
    )
{
    if (pRxBuffer == NULL || bufferSize == 0)
        return ERROR;

    /*
     * Buffer must not contain unrelated
     * dirty CPU data in overlapping lines.
     */
    if (cacheFlush(DATA_CACHE,
                   pRxBuffer,
                   bufferSize) != OK)
    {
        return ERROR;
    }

    if (cacheInvalidate(DATA_CACHE,
                        pRxBuffer,
                        bufferSize) != OK)
    {
        return ERROR;
    }

    return OK;
}

This example illustrates the required ownership preparation rather than a universally optimal cache-maintenance sequence.

Some BSPs provide DMA synchronization operations that combine the necessary write-back and invalidation behavior.

Use those interfaces when available.

The driver must not allow the CPU to modify the receive buffer after ownership has been transferred to the device.

📐 7. Cache-Line Alignment and Buffer Management

#

7.1 Why Cache-Line Alignment Matters

#

Cache operations often affect entire cache lines rather than individual bytes.

The MPC860 family uses 16-byte cache blocks.

If two independent buffers share a cache line, invalidating one buffer can also invalidate data belonging to the other.

For example:

Cache Line: 16 Bytes

+-------------------------------+
| Buffer A        | Buffer B    |
+-------------------------------+
  0             7  8          15

If Buffer A is used for DMA reception while Buffer B contains modified CPU data, invalidating the shared cache line can discard valid changes to Buffer B.

This is known as false sharing at the cache-line level.

For DMA buffers, the problem is not simply performance degradation; it can cause data loss.

7.2 Recommended Alignment Strategy

#

DMA receive buffers should generally:

• Begin at an appropriate cache-line boundary.
• Have allocated extents rounded to the required alignment.
• Avoid sharing cache lines with unrelated writable objects.
• Remain exclusively owned by the DMA controller during device access.
• Use the BSP’s documented allocation and synchronization mechanism.

For a 16-byte cache-line size:

#define CACHE_LINE_SIZE  16U

#define ALIGN_UP(value, alignment) \
    (((value) + (alignment) - 1U) & \
     ~((alignment) - 1U))

Example:

size_t frameSize = 1518;

size_t allocSize =
    ALIGN_UP(frameSize, CACHE_LINE_SIZE);

/*
 * allocSize becomes 1520 bytes.
 *
 * The allocation itself must also return
 * a correctly aligned starting address.
 */

Rounding the size does not guarantee that the returned pointer is aligned.

The allocation mechanism must independently satisfy the alignment requirement.

For actual hardware, verify alignment constraints for both the CPU cache and the DMA controller.

7.3 Physical Address Translation

#

An additional concern is the difference between CPU-visible and DMA-visible addresses.

A pointer returned by a memory allocator may represent a virtual address.

The FEC requires an address it can use on its memory bus.

These addresses are not necessarily identical.

Conceptually:

CPU Virtual Address
         |
         v
Memory Mapping / Translation
         |
         v
DMA-Visible Address
         |
         v
FEC Buffer Descriptor

Modern VxWorks BSPs may expose DMA mapping interfaces for converting CPU addresses into device-visible addresses.

Legacy platforms may use identity mappings or other BSP-specific conventions.

Drivers should not blindly cast a CPU pointer into an integer and assume it is a valid DMA address.

Incorrect address translation can cause memory corruption even when cache maintenance is otherwise correct.

🚨 8. Common VxWorks Cache Coherency Bugs

#

8.1 Ethernet Frames Contain Old Data

#

Symptom

Packets are transmitted with stale payloads or previously used buffer contents.

Likely Cause

The transmit buffer was modified in the CPU cache, but the changes were not flushed before DMA.

Resolution

Flush the transmit buffer before publishing the descriptor ownership state and starting hardware transmission.

8.2 Received Frames Contain Corrupted Data

#

Symptom

Incoming Ethernet packets contain incorrect bytes even though hardware reports successful reception.

Likely Cause

The CPU reads stale cache contents or dirty cache lines overwrite DMA-generated data.

Resolution

Use the correct receive-buffer preparation and completion synchronization sequence, with strict DMA ownership enforcement.

8.3 Descriptor Status Never Changes

#

Symptom

The driver repeatedly observes the same transmit or receive descriptor state.

Likely Cause

Descriptor memory is cacheable without adequate synchronization, or the driver is examining a stale cached copy.

Resolution

Use DMA-coherent descriptor memory as required by the MPC860T FEC driver.

8.4 Driver Works Only with Caches Disabled

#

Symptom

The Ethernet driver operates correctly with the data cache disabled but fails after caching is enabled.

Likely Cause

The driver relies on implicit memory coherence and lacks required cache maintenance.

Resolution

Audit every CPU-to-DMA and DMA-to-CPU ownership transfer, including descriptor handling and memory barriers.

8.5 Failures Occur Only Under Heavy Traffic

#

Symptom

Communication works during basic tests but experiences random errors under sustained network load.

Possible Causes

• Buffer ownership races.
• Incorrect cache synchronization.
• Cache-line sharing.
• Descriptor ring corruption.
• Reuse of buffers before DMA completion.
• Incorrect interrupt or DMA completion handling.

Resolution

Stress-test the driver with simultaneous transmit and receive activity while validating descriptor ownership and buffer lifetime.

🧪 9. Testing and Verifying Cache Coherency

#

9.1 Functional Validation

#

Cache-coherency testing should cover more than successful initialization and a single packet transfer.

Recommended validation scenarios include:

1. Repeated transmission of changing packet patterns.
2. Continuous reception of maximum-length Ethernet frames.
3. Simultaneous transmit and receive traffic.
4. Reuse of transmit buffers after completion.
5. Receive-buffer replacement under load.
6. Descriptor ring wraparound.
7. Cache-enabled versus cache-disabled operation.
8. DMA transfers during high CPU utilization.
9. Network interruptions and recovery.
10. Long-duration stress testing.

Each test should verify packet integrity and descriptor consistency.

9.2 Testing with Known Data Patterns

#

A useful technique is transmitting predictable byte patterns.

For example:

#include <stdint.h>
#include <stddef.h>

void fillTestPattern
    (
    uint8_t *pBuffer,
    size_t length,
    uint8_t seed
    )
{
    size_t i;

    for (i = 0; i < length; ++i)
    {
        pBuffer[i] = (uint8_t)(seed + i);
    }
}

The receiving endpoint can verify the pattern and detect corruption.

Changing the seed for each transfer helps expose stale-cache failures that might be hidden when identical payloads are repeatedly transmitted.

9.3 Measuring Performance

#

Non-cacheable buffers improve coherency simplicity but may reduce CPU-side throughput.

Cacheable buffers improve CPU access performance but require synchronization operations.

Useful performance metrics include:

Measurements should be performed on the actual target hardware with the final BSP and cache configuration.

9.4 Debugging Considerations

#

Cache-coherency failures can be sensitive to timing.

Adding diagnostic logging, breakpoints, or debugger memory inspections may change execution timing and alter the observed behavior.

A reliable debugging approach should combine:

• Hardware status registers.
• Descriptor ring inspection.
• Buffer ownership tracing.
• Packet sequence numbers.
• Memory integrity checks.
• Interrupt and DMA completion counters.
• Controlled cache configuration experiments.

Avoid assuming a problem is fixed simply because it disappears when debugging instrumentation is enabled.

🚀 10. Applying These Principles to Modern VxWorks 7 Systems

#

Although the MPC860 example focuses on classic PowerPC hardware, the same fundamental issues appear in modern noncoherent DMA systems.

Contemporary embedded platforms may use:

• ARM-based processors.
• PowerPC and Power Architecture processors.
• PCIe DMA controllers.
• FPGA accelerators.
• Ethernet controllers.
• Storage controllers.
• Custom high-speed peripheral interfaces.

However, modern SoCs may provide additional hardware coherency features or DMA mapping infrastructure.

As a result, the correct implementation can differ significantly from legacy MPC860 drivers.

10.1 Modern DMA Design Considerations

#

When developing VxWorks 7 drivers, evaluate:

• Whether the DMA device participates in hardware cache coherence.
• Whether memory is coherent or noncoherent.
• Whether an IOMMU or address translation unit is present.
• Whether the DMA allocation API returns cacheable memory.
• Whether cache synchronization is required before or after transfers.
• Whether DMA mapping and unmapping operations are required.
• Whether the target uses physically indexed or virtually indexed caches.
• Whether memory barriers are required for descriptor publication.
• Whether DMA resources are shared across multiple CPU cores.

These characteristics should determine the driver design.

10.2 Avoid Unnecessary Cache Maintenance

#

Cache maintenance is not always required on hardware-coherent systems.

Unnecessary operations can introduce overhead and, when used incorrectly, may interfere with valid memory contents.

Therefore, drivers should rely on the platform’s documented DMA ownership and synchronization model.

The general design principle remains unchanged:

Before transferring ownership, ensure that the next hardware or software agent can observe the correct memory contents.

For legacy MPC860 systems, this often requires explicit cache operations.

For modern VxWorks 7 systems, DMA mapping APIs or hardware coherence may handle some or all of the synchronization requirements.

✅ 11. Conclusion

#

Cache coherency is one of the most important correctness requirements in DMA-capable VxWorks device drivers.

On processors such as the Motorola MPC860, the CPU cache and DMA hardware do not automatically maintain a consistent view of shared memory.

Without appropriate synchronization, drivers may experience stale packet data, corrupted buffers, invalid descriptor states, and intermittent communication failures.

Two principal techniques address these problems:

1. DMA-safe, non-cacheable memory for small shared control structures such as FEC buffer descriptors.
2. Cacheable packet buffers with explicit cache synchronization to preserve CPU performance while ensuring correct DMA operation.

For the MPC860T Fast Ethernet Controller, a robust implementation should keep descriptor rings in suitably coherent external memory and carefully coordinate cacheFlush() and cacheInvalidate() operations around packet-buffer ownership transitions.

Correct memory alignment, physical address translation, synchronization barriers, and DMA completion handling are equally important.

The original design approach—separating descriptor memory from frequently allocated packet buffers—remains a useful optimization strategy, provided that each memory region satisfies the hardware and BSP requirements.

These principles extend beyond the MPC860 to Ethernet controllers, PCIe devices, FPGA accelerators, storage interfaces, and other DMA-based peripherals.

Reliable DMA driver development requires more than allocating shared memory: it requires explicit ownership rules, correct cache visibility, and systematic validation under real operating conditions.

By applying these practices, VxWorks developers can eliminate many difficult cache-related failures while maintaining the performance and predictability expected of real-time embedded systems.