PCIe Linux Driver Understanding in Brief

PCIe Wires and Voltages
 The Memory bus
 The I/O bus
 The bus clock Bus locking / wait states
 The Bridge and Delayed Transaction
 Bursts
 Bus mastering, bus arbitration, DMA

PCI	PCIe
Very slow bus in today’s terms (66 MHz)	Ø  It’s not a bus, it’s a packet network of 32 bit elements Ø  Hotpluggable, energy saving Ø  Each device has its own dedicated wires Ø  Transaction Layer Packets (TLPs) instead of bus transactions Ø  Posted and non-posted bus requests Ø  Data flow control, error detection and retransmit Ø  Full backward compatibility and interoperability with legacy PCI (including INTx) Ø  Lanes: 1x, 2x, 4x, 8x, 16x Speeds: PCIe 1.0: 2.5 Gb/s raw bits, 2.0 Gb/s data bits, parallel to 62.5 MHz 32-bit bus...
Memory space (32/64 bits), I/O space, configuration space
Bus Mastering (DMA) and bursts
Bridges
Vendor ID, Device ID, class, subclass
pci.ids, lspci and kernel support of a specific device
Automatic allocation of dedicated addresses segments (“BAR addresses”) by OS (BIOS on PCs) and interrupts
Delayed Transaction not allowed on I/O writes. (but is on Memory writes) PCI I/O Mapped I/O - 16 bits addressing on x86. Depends on architecture. I/O Mapped I/O - PCI allows 32 bits. I/O Mapped I/O  - /proc/ioports for your own computer’s mapping Memory mapped I/O- Read prefetching Memory mapped I/O- Write reordering Memory mapped I/O- Cache coherency and flushing Memory mapped I/O- IOMMU Memory mapped I/O- /proc/iomem for your own computer’s mapping. Interrupt- Physical interrupt lines are limited Interrupt- PCI has INTA, INTB, INTC, INTD Interrupt- Interrupt sharing: Everyone answers the phone Interrupt- Edge triggered can’t be shared, and double interrupts are missed Interrupt- Level triggered can Interrupt- MSI: Message Signalled Interrupt Interrupt- Check out /proc/interrupts to get the nudges.

1. PCIe Driver Hot-Plug Mechanism:

Hot-Plug/Hot-Swap solution provide methods to replace modules without turning system off, keeping operating system services running correctly after component removal and restarting or shutting down software associated to removed device.

2. Hardware Support for Hot-Plug Mechanism:

1. Hot-Plug Controller 

2. Card Slot Power Switching and Card Reset Logic 

3. Power Indicator and Attention Indicator

4. Attention Button 

5. Card Present Detection Pins

3. Software Support for Hot-Plug Mechanism

1. Hot-Plug Service 

2. Hot-Plug Driver 

3. Device Drivers 

4. User Interface

4. PCI Driver Model vs. PCI Express Driver Model

 Standard PCI Driver Model allows to load one driver for one device 

 Standard PCI Express Ports support up to four different functions 

 PCI Express Port Bus Driver was designed to support PCI Express functionalities in PCI Driver Mode

5. PCI Express Bus Driver
1. Hot-Plug
2. Power Management
3. Virtual Channel
4. Advanced Error Reporting

6. uDev Rules:

uDev rules

  ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:03:00.0", ATTR{vendor}=="0x10EE", ATTR{device}=="0x0008", RUN+="/bin/bash /etc/develop/check_firmware 0x10EE0008" 

ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:02:00.0", ATTR{vendor}=="0x10EE", ATTR{device}=="0x009", RUN+="/bin/bash /etc/develop/check_firmware 0x10EE0018

7. Device parameters

There are two types of PCI Express Port:

Ø  The Root Port

Ø  The Switch Port

PCI Express Port Bus Driver

To enable multiple service drivers running simultaneously requires

having a PCI Express Port Bus driver, which manages all populated

PCI Express Ports and distributes all provided service requests

to the corresponding service drivers as required. Some key

advantages of using the PCI Express Port Bus driver are listed below:

        - Allow multiple service drivers to run simultaneously on

          a PCI-PCI Bridge Port device.

        - Allow service drivers implemented in an independent

          staged approach.

        - Allow one service driver to run on multiple PCI-PCI Bridge

          Port devices.

        - Manage and distribute resources of a PCI-PCI Bridge Port

          device to requested service drivers.

Configuring the PCI Express Port Bus Driver vs. Service Drivers

Including the PCI Express Port Bus driver depends on whether the PCI

Express support is included in the kernel config. The kernel will

automatically include the PCI Express Port Bus driver as a kernel

driver when the PCI Express support is enabled in the kernel.

Enabling Service Driver Support

PCI device drivers are implemented based on Linux Device Driver Model.

All service drivers are PCI device drivers.

pcie_port_service_register

int pcie_port_service_register(struct pcie_port_service_driver *new)

This API replaces the Linux Driver Model's pci_register_driver API. A

service driver should always calls pcie_port_service_register at

module init. Note that after service driver being loaded, calls

such as pci_enable_device(dev) and pci_set_master(dev) are no longer

necessary since these calls are executed by the PCI Port Bus driver.

pcie_port_service_unregister

void pcie_port_service_unregister(struct pcie_port_service_driver *new)

pcie_port_service_unregister replaces the Linux Driver Model's

pci_unregister_driver. It's always called by service driver when a

module exits.

Sample Code

Below is sample service driver code to initialize the port service

driver data structure.

static struct pcie_port_service_id service_id[] = { { .vendor = PCI_ANY_ID, .device = PCI_ANY_ID, .port_type = PCIE_RC_PORT, .service_type = PCIE_PORT_SERVICE_AER, }, { /* end: all zeroes */ } }; static struct pcie_port_service_driver root_aerdrv = { .name = (char *)device_name, .id_table = &service_id[0], .probe = aerdrv_load, .remove = aerdrv_unload, .suspend = aerdrv_suspend, .resume = aerdrv_resume, }; Below is a sample code for registering/unregistering a service driver. static int __init aerdrv_service_init(void) { int retval = 0; retval = pcie_port_service_register(&root_aerdrv); if (!retval) { /* * FIX ME */ } return retval; } static void __exit aerdrv_service_exit(void) { pcie_port_service_unregister(&root_aerdrv); } module_init(aerdrv_service_init); module_exit(aerdrv_service_exit);

Ø  Since all service drivers of a PCI-PCI Bridge Port device are allowed to run simultaneously, 

Ø  Below lists a few of possible resource conflicts with proposed solutions.

1.  MSI and MSI-X Vector Resource

Ø  Once MSI or MSI-X interrupts are enabled on a device, it stays in this mode until they are disabled again.

Ø  Since service drivers of the same PCI-PCI Bridge port share the same physical device, if an individual service driver enables or disables MSI/MSI-X mode it may result unpredictable behaviour.

Solution:

        To avoid this situation all service drivers are not permitted to

        switch interrupt mode on its device. 

Ø  The PCI Express Port Bus driver is responsible for determining the interrupt mode and this should be transparent to service drivers. 

Ø  Service drivers need to know only the vector IRQ assigned to the field irq of struct pcie_device, which is passed in when the PCI Express Port Bus driver probes each service driver. 

Ø  Service drivers should use (struct pcie_device*)dev->irq to call request_irq/free_irq. 

Ø  In addition, the interrupt mode is stored in the field interrupt_mode of struct pcie_device.

2.  PCI Config Registers

Ø  Each service driver runs its PCI config operations on its own

capability structure except the PCI Express capability structure,    in which Root Control register and Device Control register are shared

between (PCI Express Power Management)PME and (Advanced Error Reporting)AER.

Ø  It assumes that all service drivers will be well behaved and not overwrite other service driver's configuration settings.

PCI/PCIe device working


static const struct pci_device_id foo_ids[] = {
{
.class = 0x000000,
.class_mask = 0x000000,
.vendor = 0xdead,
.device = 0xbeef,
.subvendor = PCI_ANY_ID,
.subdevice = PCI_ANY_ID,
.driver_data = 0 /* Arbitrary data, hence zero */
},
{ /* End: all zeroes */ }
};
MODULE_DEVICE_TABLE(pci, foo_ids);


static struct pci_driver foo_driver = {
.name = "foo",
.id_table = foo_ids,
.probe = foo_probe,
.remove = __devexit_p(foo_remove),
};
static int __init foo_init(void)
{
    return pci_register_driver(&foo_driver);
}
static void __exit foo_exit(void)
{
    pci_unregister_driver(&foo_driver);
}
module_init(foo_init);
module_exit(foo_exit);

The probe method
static int __devinit foo_probe(struct pci_dev *pdev,
const struct pci_device_id *ent)
{
struct foo_privdata *privdata;
int rc = 0;
int i;
privdata = kzalloc(sizeof(*privdata), GFP_KERNEL);
if (!privdata) {
printk(KERN_ERR "foo: Failed to allocate memory.\n");
return -ENOMEM;
}
pci_set_drvdata(pdev, privdata);
//Enabling the device

rc = pci_enable_device(pdev);
if (rc) {
printk(KERN_ERR "foo: pci_enable_device() failed.\n");
goto no_enable;
}

// Checking that BAR 0 is defined and memory mapped
if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM)) {
printk(KERN_ERR "foo: Incorrect BAR configuration.\n");
rc = -ENODEV;
goto bad_bar;
}
rc = pci_request_regions(pdev, "foo");
if (rc) {
printk(KERN_ERR "foo: pci_request_regions() failed.\n");
goto failed_request_regions;
}

//Assuming that our device’s configuration header requests 128 bytes on BAR 0:
privdata->registers = pci_iomap(pdev, 0, 128);
if (!privdata->registers) {
printk(KERN_ERR "foo: Failed to map BAR 0.\n");
goto failed_iomap0;
}
iowrite32(1, &privdata->registers[0x10]);
mmiowb(); /* Some holy water never hurts */
//registers is pointer to u32.
//Reading a bit

for (i=0; i<16; i++)
printk(KERN_INFO "foo: Register 0x%x = 0x%x \n",i, ioread32(&privdata->registers[i]));

//Also common in the kernel: inp, outp, writel, readl, writew, readw,writeb, readb, ioread8, //ioread16, iowrite8, iowrite16


//Enabling mastership and MSI
pci_set_master(pdev);
/* Set up a single MSI interrupt */
if (pci_enable_msi(pdev)) {
printk(KERN_ERR "foo: Failed to enable MSI.\n");
rc = -ENODEV;
goto failed_enable_msi;
}

//Checking DMA 32 or 64 bit? 
if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(64)))
privdata->dma_using_dac = 1;
else if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(32)))
privdata->dma_using_dac = 0;
else {
printk(KERN_ERR "foo: Failed to set DMA mask.\n");
rc = -ENODEV;
goto failed_dmamask;
}

// Allocating aligned memory
int memorder = 2;
privdata->mem = (u32 *) __get_free_pages(GFP_KERNEL |
__GFP_DMA | __GFP_ZERO, memorder);
if (!privdata->mem) {
printk(KERN_ERR "foo: Failed to allocate memory.\n");
rc = -ENOMEM;
goto failed_dmamem;
}
printk(KERN_INFO "foo: Got %ld bytes at 0x%x\n",
PAGE_SIZE * (1 << memorder),
(u32) privdata->mem);

// DMA mapping
dma_mem = pci_map_single(pdev, privdata->mem,
PAGE_SIZE * (1 << memorder),
PCI_DMA_FROMDEVICE);
if (pci_dma_mapping_error(pdev, dma_mem)) {
printk(KERN_ERR "foo: Failed to map memory).\n");
rc = -ENOMEM;
goto failed_dmamap;
}

 dma mem is a dma addr t, which may be a u64 or a u32, depending on the CONFIG X86 64 or CONFIG HIGHMEM64G kernel compilation flags
//Preparing for DMA
iowrite32( (u32) (dma_mem & 0xffffffff),
&privdata->registers[0x0c]);
iowrite32( ( (u32) ( ( dma_mem >> 32) & 0xffffffff) ),
&privdata->registers[0x0d]);
iowrite32( privdata->dma_using_dac,
&privdata->registers[0x08]);
mmiowb(); /* Hope the holy water works */

Warning: Is the PCI device aware of the host’s endianess?
// Setting up the interrupt handler
// waitq is of type wait queue head t
init_waitqueue_head(&privdata->waitq);
rc = request_irq(pdev->irq, foo_msi, 0, "foo", privdata);
if (rc) {
printk(KERN_ERR "foo: Failed to register MSI.\n");
rc = -ENODEV;
goto failed_register_msi;
}

//The MSI handle:
static irqreturn_t foo_msi(int irq, void *data)
{
struct foo_privdata *privdata = data;
privdata->flag = 1;
wake_up_interruptible(&privdata->waitq);
return IRQ_HANDLED;
}
 Important: On a non-MSI handler, checking that our device really
Asserted the interrupt is mandatory.

DMA Write Cycle:
privdata->flag = 0;
iowrite32( (u32) 1, &privdata->registers[0x09]); /* GO! */
wait_event_interruptible(privdata->waitq,
(privdata->flag != 0));
if (privdata->flag == 0) {
printk(KERN_ERR "foo: Interrupted!.\n");
rc = -ENODEV;
goto failed_wait_irq;
}

pci_dma_sync_single_for_cpu(pdev, privdata->dma_mem, PAGE_SIZE * (1 <<  memorder),PCI_DMA_FROMDEVICE);
for (i=0; i<16; i++)
printk(KERN_INFO "foo: Read at index %d: %x\n",i, privdata->mem[i]);

pci_dma_sync_single_for_device(pdev, privdata->dma_mem, PAGE_SIZE * (1 <<  memorder),PCI_DMA_FROMDEVICE);

 For multiple operation map and unmap for each iteration of DMA Cycle.
 Consistent DMA 
 “Streaming” DMA mapped and unmapped every use 
 “Streaming” DMA mapped once (as shown here) 
 Scatter-gather See DMA-API-HOWTO.txt in the kernel docs

pci_unmap_single(pdev, privdata->dma_mem,
PAGE_SIZE * (1 << memorder),
PCI_DMA_FROMDEVICE);
free_pages ((unsigned long) privdata->mem, memorder);
free_irq(pdev->irq, privdata);
pci_disable_msi(pdev);
pci_clear_master(pdev); /* Nobody seems to do this */
pci_iounmap(pdev, privdata->registers);
pci_release_regions(pdev);
pci_disable_device(pdev);