PCI Express Switch Enumeration Using VMM-Based DesignWare Verification IP

Keith Young, Corporate Applications Engineer

PCI Express (PCIe) utilizes a point to point interconnect and uses switches to fan out and expand the number of PCIe connections in a system. Upon system boot up a critical task is the discovery or enumeration process of all the devices in the PCIe tree so they can be allocated by the system software. During the enumeration process the system software discovers all of the switch and endpoint devices that are connected to the system, determines the memory requirements and then configures the PCIe devices. The PCIe switch devices represent a special case in this process as their configuration is unique and separate from that of PCIe endpoints. In the simulation testbench environment however, only their configuration is required; the discovery process is not strictly necessary as the number of PCIe devices are known ahead of time. This paper will elucidate the process of switch configuration using Synopsys' PCI Express VMM simulation class libraries for SystemVerilog.

First, while the discovery process is not needed within the testbench environment, the testbench must still select the bus numbers and memory address of all the devices. These bus numbers are shown in Figure 1 as Bus#2, Bus#3, Bus#4, and Bus#5. Figure 1 shows the testbench that is used in the remainder of the article.

Figure 1. Example Testbench With Bus Numbers Assigned

This testbench in Figure 1 is testing a 3 port switch with one upstream port and two downstream ports. The switch in this example has no internal memory so as to simplify the configuration process. The enumeration process selects the bus numbers for each device that has a connection. For this example, the upstream port of the switch will be assigned bus #2. In the switch device, the internal bus between the upstream port and the two downstream ports will be assigned bus #3. The PCIe End Point (EP) devices that are downstream from the switch are Bus #4 and Bus # 5. (see Figure 1). The device numbers for both EP's will be 0.

The next step in the enumeration process is to determine the memory size of each device. Again, this step is skipped because the testbench environment sets the memory size of each device. To simplify this example, the switch will not have any internal memory, and each EP device will have 2M of memory. The EP device on bus 4 will start at address 2M. The EP on bus 5 will start at address 4M. The address range that the Switch will support is from 2M through 6M. Each EP device will only support 32-bit addresses.

Now that the bus numbers, device numbers and address range are known, the Switch can be configured. Each port of the Switch has its own configuration space and the format of the configuration space is type CONFIG 1. Refer to Table 1.

Byte Offset	Byte 3	Byte 2	Byte 1	Byte 0
0x00	Device ID		Vendor ID
0x04	Status Register		Command Register
0x08	Class Code			Revision ID
0x0C	BIST(0x00)	Header Type	Latency Timer	Cache Line Size
0x10	Base Address Register 0
0x14	Base Address Register 1
0x18	Secondary Latency Timer	Subordinate Bus Number	Secondary Bus Number	Primary Bus Number
0x1C	Secondary Status		I/O Limit	I/O Base
0x20	Memory Limit		Memory Base
0x24	Prefetchable Memory Limit		Prefetchable Memory Base
0x28	Prefetchable Base Upper 32 Bits
0x2C	Prefetchable Limit 32 Upper Bits
0x30	I/O Limit Upper 16 Bits		I/O Base Upper 16 Bits
0x34	Reserved			CapPtr
0x38	Expansion ROM Base Address
0x3C	Bridge Control		Interrupt Pin	Interrupt Line

Table 1. PCI Configuration Space Header - Type 1

There are a minimum of 6 registers that need to be programmed for the Switch to support configuration and memory accesses to an EP device. They are:

1.	Command	: Enable memory access
2.	Primary Bus	: Bus number of upstream bus
3.	Secondary Bus	: Bus number of bus directly attached device
4.	Subordinate Bus	: Bus number of farthest downstream device
5.	Memory Limit	: Maximum memory address
6.	Memory Base	: Starting memory address

Each port of the switch has its own CONFIG-1 configuration space. Each of these CONFIG-1 spaces must be programmed. The upstream port needs to be programmed first. The following table shows the configuration parameters and their values that need to be programmed.

Register	Address	Register	Value
Command[15:0]	0x04	01	16'h0004
Primary Bus[7:0]	0x18	06	8'b00000010
Secondary Bus[7:0]	0x18	06	8'b00000011
Subordinate Bus[7:0]	0x18	06	8'b00000110
Memory Base[15:0]	0x20	08	16'h0020
Memory Limit[15:0]	0x20	08	16'h0060

Table 2. Upstream Port Configuration Values

The upstream port configuration space is programmed using CONFIG-0 packets. CONFIG-0 packets use the bus number, device number, function number, and register number to access the register to be programmed. The following VMM task will program the upstream Switch port with the previously listed values.

//  Program Upstream Port
//    1)  Write Command register
//    2)  Write Subordinate, Secondary and Primary Bus Numbers
//    3)  Write Memory Limit and Memory Base
task  pcie_rvm_env::Upstream_switch_config();
  begin
      bit test;
      bit status;
  
      dw_vip_pcie_tlp_transaction  cfg0_wr = 
                                   new ( , dw_vip_pcie_tlp_transaction::CFG_WR_0);
      dw_vip_pcie_tlp_transaction  cfg_tlp;

      //  Write COMMAND REGISTER
      //  Enable memory cycles
      test = cfg0_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h02; m_bvDevNum == 5'h00; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h001;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00000004;
      };

      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   
      $cast(cfg_tlp, cfg_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    

      //  Write Subordinate Bus, Secondary Bus and Primary Bus Numbers:  
      test = cfg0_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h02; m_bvDevNum == 5'h00; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h006;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00050302;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

     //  Write Memory Limit and Memory Base
      test = cfg0_wr.randomize() with { 
          m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h02; m_bvDevNum == 5'h00;
	   m_bvFuncNum == 3'h0; m_bvRegNum == 10'h008;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00600020;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   
  end
endtask

The downstream port (Device 1 ) has the following values as shown in Table 3.

Register	Address	Register	Value
Command[15:0]	0x04	01	16'h0004
Primary Bus[7:0]	0x18	06	8'b00000011
Secondary Bus[7:0]	0x18	06	8'b00000100
Subordinate Bus[7:0]	0x18	06	8'b00000100
Memory Base[15:0]	0x20	08	16'h0020
Memory Limit[15:0]	0x20	08	16'h0040

Table 3. Downstream Port Configuration Values

The downstream ports configuration space is programmed using CONFIG-1 packets. CONFIG-1 packets also use bus number, device number, function number, and register number to access the register to be programmed. The following VMM task will program downstream device 1 of the downstream port.

//  Program Device 1 Downstream Port
//    1)  Write Command register
//    2)  Write Subordinate, Secondary and Primary Bus Numbers
//    3)  Write Memory Limit and Memory Base
task  pcie_rvm_env::Downstream_device_1_switch_config();
  begin
      bit test;
  
      dw_vip_pcie_tlp_transaction  cfg1_wr = 
                                   new ( , dw_vip_pcie_tlp_transaction::CFG_WR_1);
      dw_vip_pcie_tlp_transaction  cfg_tlp;

      //  Write COMMAND REGISTER
      //  Enable memory cycles
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h001;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00000004;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg1_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   


      //  Write Subordinate Bus, Secondary Bus and Primary Bus Numbers:  
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h006;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00040403;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

     //  Write Memory Limit and Memory Base
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h008;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00400020;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   


  end
endtask

The downstream port (Device 2 ) has the following configuration values as shown in Table 4.

Register	Address	Register	Value
Command[15:0]	0x04	01	16'h0004
Primary Bus[7:0]	0x18	06	8'b00000011
Secondary Bus[7:0]	0x18	06	8'b00000101
Subordinate Bus[7:0]	0x18	06	8'b00000101
Memory Base[15:0]	0x20	08	16'h0040
Memory Limit[15:0]	0x20	08	16'h0060

Table 4. Downstream Port Configuration Values

//  Program Device 2 Downstream Port
//    1)  Write Command register
//    2)  Write Subordinate, Secondary and Primary Bus Numbers
//    3)  Write Memory Limit and Memory Base
task  pcie_rvm_env::Downstream_device_2_switch_config();
  begin
      bit test;
  
      dw_vip_pcie_tlp_transaction  cfg1_wr = 
                                   new ( , dw_vip_pcie_tlp_transaction::CFG_WR_1);
      dw_vip_pcie_tlp_transaction  cfg_tlp;

      //  Write COMMAND REGISTER
      //  Enable memory cycles
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h02; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h001;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00000004;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg1_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   


      //  Write Subordinate Bus, Secondary Bus and Primary Bus Numbers:  
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h02; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h006;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00050503;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

     //  Write Memory Limit and Memory Base
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h008;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00600040;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

  end
endtask

The Switch is now ready to accept configuration and memory packets that are for the EP devices connected to the downstream ports of the switch.

In summary, PCIe designs must go through the process of Switch enumeration to discover available switches, and then to configure them. This paper has shown how you can use VMM based PCIe Verification IP and SystemVerilog to perform the task of configuration during the process of Switch enumeration. The examples use the rich set of class, protocol, and packet capabilities of the VMM models to perform the configuration task.

Search Tools