VxWorks 7 PCIe Driver Development with VxBus 2.0

Table of Contents

VxWorks 7 PCIe Driver Development with VxBus 2.0

PCI Express (PCIe) has become one of the most important high-speed interconnect technologies in embedded and industrial systems. FPGA accelerators, high-speed ADCs and DACs, network adapters, storage controllers, and specialized industrial I/O devices all rely on PCIe to exchange data with embedded processors.

In VxWorks 7, the preferred architecture for developing modern PCIe device drivers is the VxBus 2.0 framework. VxBus provides a structured driver model that simplifies hardware discovery, resource management, interrupt handling, and device lifecycle management while maintaining the deterministic behavior required by real-time applications.

A well-designed VxBus PCIe driver typically provides:

Automatic device discovery and matching.
Standardized BAR and interrupt resource management.
Support for PCI configuration-space access.
Flattened Device Tree (FDT) integration where applicable.
SMP-aware driver behavior.
Clean probe, attach, and detach lifecycles.
Integration with DMA and interrupt subsystems.

This guide presents a practical PCIe driver architecture using a PLX/Broadcom-style PCIe switch or bridge as an example.

🧩 1. Define the Driver Control Structure
#

Every VxBus driver should maintain a private control structure containing the state associated with each device instance.

/* plxPcieDrv.h */

#ifndef __PLX_PCIE_DRV_H__
#define __PLX_PCIE_DRV_H__

#include <vxWorks.h>
#include <hwif/vxBus.h>
#include <subsys/pci/vxbPciLib.h>
#include <subsys/int/vxbIntLib.h>

#define PLX_VENDOR_ID   0x10B5      /* PLX / Broadcom */
#define PLX_DEVICE_ID   0x8725      /* Example device */

typedef struct plxPcieDrvCtrl
    {
    VXB_DEV_ID      pDev;           /* VxBus device handle */
    void *          barBase;        /* Virtual BAR0 address */

    VXB_RESOURCE *  pResMem;        /* Memory BAR resource */
    VXB_RESOURCE *  pResIrq;        /* Interrupt resource */

    int             irq;            /* IRQ number */

    SEM_ID          isrSem;         /* Optional ISR-to-task semaphore */

    /* Additional driver state */
    /* DMA channels, locks, statistics, etc. */

    } PLX_PCIE_DRV_CTRL;

#endif /* __PLX_PCIE_DRV_H__ */

The control structure serves as the driver’s central state container.

pDev identifies the VxBus device instance, while barBase stores the virtual address associated with a mapped PCIe BAR.

The resource pointers track resources allocated through VxBus and must be released during the driver’s cleanup path.

A semaphore can optionally be used to allow the interrupt service routine to perform minimal work while delegating heavier processing to a dedicated task.

🔍 2. Implement the Probe Function
#

The probe method determines whether the driver supports a particular PCIe device.

A basic implementation can match the PCI Vendor ID and Device ID:

LOCAL STATUS plxPcieProbe
    (
    VXB_DEV_ID pDev
    )
    {
    UINT16 vendorId = 0;
    UINT16 deviceId = 0;

    /* Read Vendor ID */
    if (vxbPciConfigRead16
        (
        pDev,
        PCI_CFG_VENDOR_ID,
        &vendorId
        ) != OK)
        {
        return ERROR;
        }

    /* Read Device ID */
    if (vxbPciConfigRead16
        (
        pDev,
        PCI_CFG_DEVICE_ID,
        &deviceId
        ) != OK)
        {
        return ERROR;
        }

    /* Match supported hardware */
    if (vendorId == PLX_VENDOR_ID &&
        deviceId == PLX_DEVICE_ID)
        {
        return OK;
        }

    return ERROR;
    }

The important point is to use the VxBus PCI configuration helpers rather than legacy PCI configuration APIs when developing a VxBus-based driver.

Returning OK indicates that the driver recognizes the device and allows VxBus to continue with the attach phase.

Production drivers can make matching more selective by checking additional information such as:

PCI revision ID.
Subsystem Vendor ID.
Subsystem Device ID.
PCI class code.
Device capabilities.
Hardware-specific configuration.

This becomes particularly useful when a single driver supports multiple related devices.

🛠️ 3. Attach the Device and Allocate Resources
#

The attach method is the central initialization stage of a VxBus PCIe driver.

It typically allocates the private control structure, obtains BAR resources, acquires the interrupt resource, connects the ISR, initializes synchronization objects, and configures the hardware.

LOCAL STATUS plxPcieAttach
    (
    VXB_DEV_ID pDev
    )
    {
    PLX_PCIE_DRV_CTRL * pCtrl;
    VXB_RESOURCE_ADR *  pResAdr;
    int                 i;

    /* Allocate driver control structure */
    pCtrl = (PLX_PCIE_DRV_CTRL *)
            vxbMemAlloc(sizeof(PLX_PCIE_DRV_CTRL));

    if (pCtrl == NULL)
        return ERROR;

    bzero((char *)pCtrl, sizeof(PLX_PCIE_DRV_CTRL));

    pCtrl->pDev = pDev;

    /*
     * Allocate a usable memory BAR.
     */
    for (i = 0; i < VXB_MAXBARS; i++)
        {
        pCtrl->pResMem =
            vxbResourceAlloc(pDev, VXB_RES_MEMORY, i);

        if (pCtrl->pResMem != NULL)
            {
            pResAdr =
                (VXB_RESOURCE_ADR *)pCtrl->pResMem->pRes;

            if (pResAdr != NULL)
                {
                pCtrl->barBase =
                    (void *)pResAdr->virtual;

                /*
                 * pResAdr->start   = physical address
                 * pResAdr->size    = BAR size
                 * pResAdr->pHandle = bus access handle
                 */

                break;
                }

            vxbResourceFree(pDev, pCtrl->pResMem);
            pCtrl->pResMem = NULL;
            }
        }

    if (pCtrl->barBase == NULL)
        {
        vxbMemFree(pCtrl);
        return ERROR;
        }

    /*
     * Allocate interrupt resource.
     */
    pCtrl->pResIrq =
        vxbResourceAlloc(pDev, VXB_RES_IRQ, 0);

    if (pCtrl->pResIrq == NULL)
        {
        vxbResourceFree(pDev, pCtrl->pResMem);
        vxbMemFree(pCtrl);
        return ERROR;
        }

    pCtrl->irq =
        (int)(long)vxbResourceAdrsGet
        (
        pDev,
        VXB_RES_IRQ,
        0
        );

    /*
     * Store private data for later retrieval.
     */
    vxbDevSoftcSet(pDev, pCtrl);

    /*
     * Connect interrupt handler.
     */
    if (vxbIntConnect
        (
        pDev,
        pCtrl->pResIrq,
        plxPcieIsr,
        pCtrl
        ) != OK)
        {
        /* Cleanup omitted here for brevity. */
        return ERROR;
        }

    /*
     * Enable interrupt.
     */
    if (vxbIntEnable
        (
        pDev,
        pCtrl->pResIrq
        ) != OK)
        {
        vxbIntDisconnect
            (
            pDev,
            pCtrl->pResIrq
            );

        /* Cleanup omitted here for brevity. */
        return ERROR;
        }

    /*
     * Optional ISR-to-task synchronization.
     */
    pCtrl->isrSem =
        semBCreate(SEM_Q_PRIORITY, SEM_EMPTY);

    /*
     * Device-specific initialization.
     */
    plxPcieHwInit(pCtrl);

    return OK;
    }

The BAR allocation loop asks VxBus for available memory resources. Once a valid resource is returned, VXB_RESOURCE_ADR provides information about the resource, including its virtual address, physical address, size, and access handle.

One important engineering principle is complete error-path cleanup. If interrupt allocation fails after a BAR has already been allocated, the BAR must be released before returning.

The same rule applies to every subsequent initialization stage.

🧱 4. Understand PCIe BAR Mapping
#

PCIe Base Address Registers (BARs) describe the memory or I/O resources exposed by the endpoint.

For memory BARs, the VxBus resource layer provides the driver with a mapped address that can be used for device register access.

Conceptually, the mapping looks like:

PCIe Device
     │
     ├── BAR0
     │     └── Physical device registers
     │
     ▼
VxBus PCI resource manager
     │
     ▼
Virtual address mapping
     │
     ▼
Driver barBase
     │
     ▼
Device register access

The driver should not assume that BAR0 is always the correct resource. Real hardware may expose multiple BARs with different purposes.

For example:

BAR0 → Control and status registers.
BAR2 → Large DMA buffer window.
BAR4 → Doorbell or queue registers.

The hardware datasheet should always be used to determine which BAR corresponds to which function.

⚡ 5. Design a Deterministic Interrupt Service Routine
#

The interrupt service routine should remain short and deterministic.

A simple example is:

LOCAL void plxPcieIsr
    (
    void * pArg
    )
    {
    PLX_PCIE_DRV_CTRL * pCtrl =
        (PLX_PCIE_DRV_CTRL *)pArg;

    UINT32 status;

    /*
     * Read interrupt status.
     * Offset is device-specific.
     */
    status =
        *(volatile UINT32 *)
        ((char *)pCtrl->barBase + 0x04);

    /*
     * Clear active interrupt bits.
     * Device-specific semantics.
     */
    *(volatile UINT32 *)
        ((char *)pCtrl->barBase + 0x04) = status;

    if (status != 0)
        {
        /*
         * Defer substantial processing to a task.
         */
        if (pCtrl->isrSem != NULL)
            semGive(pCtrl->isrSem);
        }
    }

A good real-time ISR generally performs only the minimum work necessary to acknowledge the hardware event and preserve its state.

Avoid performing expensive processing, blocking operations, or complex library calls from interrupt context.

A common architecture is:

PCIe Interrupt
      │
      ▼
    ISR
      │
      ├── Read status
      ├── Clear interrupt
      └── Signal semaphore
              │
              ▼
       High-priority task
              │
              ├── Process event
              ├── Handle DMA
              └── Notify application

This design improves determinism while keeping the interrupt response path short.

🧮 6. Use VxBus Register Accessors
#

Register access should preferably use the VxBus accessor mechanisms because they account for bus-specific access attributes and endianness requirements.

static inline UINT32 plxRead32
    (
    PLX_PCIE_DRV_CTRL * pCtrl,
    UINT32 offset
    )
    {
    return vxbRead32
        (
        pCtrl->pResMem->pRes->pHandle,
        (UINT32 *)
        ((char *)pCtrl->barBase + offset)
        );
    }

static inline void plxWrite32
    (
    PLX_PCIE_DRV_CTRL * pCtrl,
    UINT32 offset,
    UINT32 value
    )
    {
    vxbWrite32
        (
        pCtrl->pResMem->pRes->pHandle,
        (UINT32 *)
        ((char *)pCtrl->barBase + offset),
        value
        );
    }

For situations where the mapping and access requirements are completely understood, direct volatile access can also be used:

#define PLX_REG_READ(pCtrl, off) \
    (*(volatile UINT32 *) \
    ((char *)(pCtrl)->barBase + (off)))

#define PLX_REG_WRITE(pCtrl, off, val) \
    (*(volatile UINT32 *) \
    ((char *)(pCtrl)->barBase + (off)) = (val))

The preferred approach depends on the VxWorks BSP, processor architecture, device requirements, and bus-access semantics.

For production hardware, register accesses should always be verified against the device’s programming manual.

🧩 7. Register the Driver with VxBus
#

The driver must expose its methods through a VxBus method table.

LOCAL VXB_DRV_METHOD plxPcieMethods[] =
    {
    {
        VXB_DEVMETHOD_CALL(vxbDevProbe),
        (FUNCPTR)plxPcieProbe
    },

    {
        VXB_DEVMETHOD_CALL(vxbDevAttach),
        (FUNCPTR)plxPcieAttach
    },

    /*
     * Optional:
     * {
     *     VXB_DEVMETHOD_CALL(vxbDevDetach),
     *     (FUNCPTR)plxPcieDetach
     * }
     */

    VXB_DEVMETHOD_END
    };

LOCAL VXB_DRV plxPcieDrv =
    {
    { NULL },
    "plxPcie",
    "PLX PCIe Switch/Bridge Driver",
    VXB_BUSID_PCI,
    0,
    0,
    plxPcieMethods,
    NULL
    };

VXB_DRV_DEF(plxPcieDrv);

A BSP or module initialization routine can register the driver with:

STATUS plxPcieDrvRegister(void)
    {
    return vxbDrvAdd(&plxPcieDrv);
    }

Once registered, VxBus can match discovered PCIe devices against the driver’s probe method and invoke the attach method for supported hardware.

🧹 8. Implement a Complete Detach Path
#

A production driver should provide a clean detach path whenever the deployment model requires device removal, hot-plug support, or module unloading.

LOCAL STATUS plxPcieDetach
    (
    VXB_DEV_ID pDev
    )
    {
    PLX_PCIE_DRV_CTRL * pCtrl =
        vxbDevSoftcGet(pDev);

    if (pCtrl == NULL)
        return ERROR;

    /*
     * Stop interrupt generation first.
     */
    if (pCtrl->pResIrq != NULL)
        {
        vxbIntDisable
            (
            pDev,
            pCtrl->pResIrq
            );

        vxbIntDisconnect
            (
            pDev,
            pCtrl->pResIrq
            );
        }

    /*
     * Release resources.
     */
    if (pCtrl->pResMem != NULL)
        {
        vxbResourceFree
            (
            pDev,
            pCtrl->pResMem
            );
        }

    if (pCtrl->pResIrq != NULL)
        {
        vxbResourceFree
            (
            pDev,
            pCtrl->pResIrq
            );
        }

    if (pCtrl->isrSem != NULL)
        semDelete(pCtrl->isrSem);

    vxbDevSoftcSet(pDev, NULL);

    vxbMemFree(pCtrl);

    return OK;
    }

Cleanup should occur in the reverse order of initialization whenever possible.

This prevents resources from remaining active after the device has been removed or the driver has been unloaded.

🚀 9. Add DMA for High-Bandwidth PCIe Devices
#

PCIe devices such as FPGA accelerators, storage controllers, and high-speed acquisition cards often depend heavily on DMA.

Rather than transferring large payloads through CPU-driven register accesses, a DMA engine can move data directly between the device and system memory.

A typical data path looks like:

PCIe Endpoint
      │
      │ DMA
      ▼
System Memory
      │
      ▼
Application / Processing Task

For VxWorks 7, DMA implementation should use the appropriate VxBus DMA interfaces and account for the platform’s cache-coherency model.

Particular attention should be paid to:

DMA address width.
Cache coherency.
Memory alignment.
Scatter/gather support.
IOMMU configuration where applicable.
DMA buffer lifetime.
Synchronization between CPU and device.
Interrupt completion handling.

DMA implementation is highly dependent on the BSP, processor, PCIe controller, and endpoint hardware, so the device datasheet and platform documentation should be treated as authoritative.

🐞 10. Debug the Driver from the VxWorks Shell
#

VxWorks provides useful shell commands for inspecting the VxBus and PCIe topology.

-> vxbPciShow()
-> vxbDevShow()
-> vxbDevPathShow()

During development, it is also useful to print the resources discovered during attachment:

printf
    (
    "PLX PCIe: BAR0 virtual = %p, IRQ = %d\n",
    pCtrl->barBase,
    pCtrl->irq
    );

A useful debugging sequence is:

Confirm the PCIe device is enumerated.
Verify Vendor ID and Device ID.
Confirm BAR sizes and addresses.
Verify the driver probe succeeds.
Confirm attach completes.
Check interrupt allocation.
Verify register reads and writes.
Trigger a known hardware interrupt.
Validate DMA transfers.
Stress the device under SMP and high interrupt loads.

This staged approach makes it easier to isolate enumeration, resource, interrupt, and data-path problems.

🛡️ 11. Follow Real-Time PCIe Driver Best Practices
#

A robust VxWorks PCIe driver should follow several core engineering principles.

Keep ISRs Short
#

Do the minimum amount of work required to acknowledge the interrupt and capture necessary state.

Protect Shared State
#

Use appropriate synchronization primitives when data is accessed concurrently by ISRs and tasks.

ISR
 │
 ├── Update minimal state
 │
 └── Signal task
          │
          ▼
      Worker task
          │
          └── Protected shared state

The synchronization mechanism should be selected according to the execution context and real-time requirements.

Prefer MSI or MSI-X
#

When supported by both the PCIe endpoint and BSP, MSI/MSI-X generally provides a cleaner interrupt architecture than legacy INTx.

Multiple MSI-X vectors can also allow different device functions or queues to be assigned to different interrupt handlers.

Validate Every Hardware Access
#

Register offsets, bit definitions, reset behavior, interrupt-clearing semantics, and DMA requirements should always be derived from the hardware reference manual.

Handle Missing Hardware Gracefully
#

The driver should tolerate cases where:

The device is not present.
BAR allocation fails.
The device reports an unexpected revision.
The endpoint is not fully initialized.
Interrupt resources are unavailable.
DMA initialization fails.

Test Under Realistic Load
#

PCIe drivers should be tested under:

High interrupt rates.
Sustained DMA traffic.
SMP operation.
CPU contention.
Repeated device resets.
Error and recovery conditions.
Long-duration stress tests.

A driver that works during a simple functional test may still fail under sustained system load.

🔄 12. Recommended VxBus PCIe Driver Architecture
#

A complete driver can be organized around the following lifecycle:

                VxBus Device Discovery
                         │
                         ▼
                     probe()
                         │
                 Device Supported?
                    /          \
                  No            Yes
                  │              │
                  ▼              ▼
                Exit           attach()
                                  │
                    ┌─────────────┼─────────────┐
                    ▼             ▼             ▼
                 Allocate       BARs          IRQ
                 Softc         Resources      Setup
                    │             │             │
                    └─────────────┼─────────────┘
                                  ▼
                           Hardware Init
                                  │
                                  ▼
                            Runtime State
                                  │
                       ┌──────────┴──────────┐
                       ▼                     ▼
                      ISR                  DMA
                       │                     │
                       ▼                     ▼
                 Worker Task          Data Processing
                       │                     │
                       └──────────┬──────────┘
                                  ▼
                               detach()
                                  │
                                  ▼
                         Disable + Disconnect
                                  │
                                  ▼
                          Free All Resources

This separation keeps hardware discovery, initialization, runtime processing, and cleanup clearly defined.

🎯 Conclusion
#

Developing a PCIe device driver for VxWorks 7 with VxBus 2.0 becomes significantly more manageable when the driver follows the framework’s intended lifecycle.

The fundamental sequence is straightforward:

Define a per-device control structure.
Implement a precise probe method.
Allocate and map PCIe BAR resources during attach.
Acquire and configure interrupt resources.
Connect a short, deterministic ISR.
Move substantial processing into tasks.
Use appropriate VxBus register-access and DMA interfaces.
Register the driver through the VxBus framework.
Implement complete error recovery and cleanup.
Stress-test the driver under SMP, interrupt, and DMA workloads.

The PLX/Broadcom-style implementation presented here provides a useful starting skeleton for PCIe endpoints, bridges, and accelerator devices. However, production deployment requires adapting BAR selection, register offsets, interrupt semantics, DMA handling, reset behavior, and synchronization to the exact hardware and BSP.

With a clean VxBus architecture, careful resource management, and disciplined real-time design, VxWorks 7 can provide a maintainable foundation for high-speed PCIe devices used in industrial automation, aerospace, networking, data acquisition, storage, and other mission-critical embedded systems.