Welcome to Part 1: The Basics of this comprehensive guide on developing custom firmware for full device emulation using FPGA-based DMA hardware. This section is designed to walk you through foundational concepts, preparing you to create, customize, and test your first custom firmware. After completing this part, you'll have a solid understanding of how to gather device information, configure your FPGA, and emulate a PCIe device.
For further assistance, feedback, or collaboration, feel free to connect with the community:
Discord - VCPU (Join Now)
- Introduction
- Key Definitions
- Device Compatibility
- Project Hierarchy Overview
- Requirements
- Gathering Donor Device Information
- Initial Firmware Customization
- Vivado Project Setup and Customization
- Building, Flashing, and Testing
- Troubleshooting
The primary goal of this guide is to help developers create custom firmware to emulate PCIe devices using FPGA-based DMA hardware. The first part of this guide focuses on the basics, walking through the essential steps for firmware development, device information gathering, environment setup, and initial firmware customization. This will lay the groundwork for more advanced custom firmware development covered in Part 2.
This guide is intended for:
- Beginners: People new to firmware development, FPGA programming, or PCIe device emulation.
- Firmware Developers: Engineers creating custom firmware for testing, emulation, or specific hardware functionality.
- Hardware Engineers: Professionals working on hardware design and testing using FPGAs.
- Security Researchers: Individuals leveraging hardware-based security tools and custom emulated devices.
Understanding the following key concepts is essential for effective firmware development:
- DMA (Direct Memory Access): A feature that allows hardware devices to access the system’s memory independently of the CPU, often used for fast data transfers.
- FPGA (Field Programmable Gate Array): A programmable integrated circuit that allows developers to create custom hardware designs.
- PCIe (Peripheral Component Interconnect Express): A fast data transfer interface used for connecting hardware devices to a computer's motherboard.
- TLP (Transaction Layer Packet): The data packet format used in PCIe communication, containing control and data information.
- BAR (Base Address Register): Registers in PCIe devices that specify the memory regions mapped to system memory for device communication.
- Device Serial Number (DSN): A unique identifier used by some devices for enhanced identification and emulation accuracy.
This guide focuses on FPGA-based devices that are compatible with PCILeech-FPGA and capable of emulating PCIe devices through DMA operations. Supported hardware includes:
- Squirrel (35T)
- EnigmaX1 (75T)
- ZDMA (100T)
- Kintex-7
Each of these devices can be programmed using Xilinx Vivado and can perform full PCIe device emulation tasks.
Several important PCIe-related configurations must be addressed for successful emulation:
- IOMMU/VT-d: Disable Intel’s Virtualization Technology for Directed I/O to allow unrestricted DMA access.
- Kernel DMA Protection: Disable Kernel DMA protection features to ensure unrestricted memory access.
- PCIe Slot Requirements: Ensure that the FPGA card is installed in a compatible PCIe slot (e.g., x1, x4, x16) that matches the card’s capabilities.
To successfully develop, test, and run custom firmware, the following system setup is recommended:
- Host System:
- Processor: Multi-core CPU (Intel i5/i7 or equivalent)
- Memory: Minimum 16GB of RAM
- Storage: SSD with at least 100GB of free space
- Operating System: Windows 10/11 (64-bit) or a compatible Linux distribution such as Ubuntu or Debian
- Peripheral Devices:
- JTAG Programmer: Required for flashing firmware onto the FPGA
- Available PCIe Slot: Ensure your system has an available PCIe slot for the FPGA card
To make effective changes and implement custom firmware, it’s crucial to understand the project structure.
Here is an example of the structure you’ll encounter when working on the PCILeech-FPGA project:
📦 pcileech-wifi-main
┣ 📂 ip
┃ ┣ 📂 100t
┃ ┃ ┣ 📜 bram_bar_zero4k.xci
┃ ┃ ┣ 📜 bram_pcie_cfgspace.xci
┃ ┃ ┣ 📜 clk_wiz_0.xci
┃ ┃ ┗ 📜 pcileech_cfgspace_writemask.coe
┣ 📂 pcie_7x
┃ ┣ 📂 zdma
┃ ┃ ┣ 📜 pcie_7x_0.v
┃ ┃ ┣ 📜 pcie_7x_0_axi_basic_rx.v
┃ ┃ ┗ 📜 pcie_7x_0_pcie_top.v
┣ 📂 src
┃ ┣ 📜 pcileech_pcie_a7.sv
┃ ┣ 📜 pcileech_pcie_cfg_a7.sv
┃ ┣ 📜 pcileech_squirrel_top.sv
┣ 📜 build.md
┣ 📜 vivado_build.tcl
┣ 📜 generate-squirrel.bat
┗ 📜 README.md
ipDirectory: Contains the FPGA Intellectual Property (IP) cores for various components like the PCIe configuration space and DMA buffers.pcie_7xDirectory: Houses PCIe-specific components including PCIe top modules and DMA logic for FPGA communication with PCIe devices.srcDirectory: Contains Verilog files that define the FPGA’s behavior. Key files include:pcileech_pcie_a7.sv: Implements the PCIe interface logic.pcileech_pcie_cfg_a7.sv: Contains the PCIe configuration space logic.pcileech_squirrel_top.sv: Top-level module for the Squirrel FPGA.
- Build Scripts:
generate-squirrel.bat: Automates the generation of Vivado projects for the Squirrel FPGA.vivado_build.tcl: Tcl script used to build the project in Vivado.
To get started with firmware development
, you’ll need the following hardware:
- Donor PCIe Device: This is the device you plan to emulate. Typical examples include network adapters or storage controllers.
- FPGA Card: A DMA-capable FPGA device such as the Squirrel (35T) or EnigmaX1 (75T).
- JTAG Programmer: Necessary for flashing the firmware onto the FPGA card.
- Xilinx Vivado: Required for FPGA development. This is used to synthesize and implement the custom firmware.
- Visual Studio Code: A code editor for writing and modifying the Verilog/SystemVerilog source code.
- PCILeech-FPGA Repository: The base code repository where you’ll find the necessary scripts and source files.
- PCIe Device Scanning Tools: Tools for extracting configuration data from PCIe devices.
- Arbor: A commercial PCIe device scanning tool (14-day trial available).
- Alternatives: Linux’s
lspcicommand or PCI-Z for Windows.
-
Install Xilinx Vivado
- Visit the Xilinx Vivado Download Page and download the appropriate version for your FPGA device.
- Follow the installation instructions to install Vivado on your system.
-
Install Visual Studio Code
- Download Visual Studio Code from here.
- Install any necessary extensions for Verilog/SystemVerilog support.
-
Clone the PCILeech-FPGA Repository
- Open a terminal or command prompt, then run the following commands:
git clone https://github.com/ufrisk/pcileech-fpga.git cd pcileech-fpga
- Open a terminal or command prompt, then run the following commands:
-
Set Up a Clean Development Environment
- It is recommended to use a dedicated machine or virtual environment for development, especially if you plan to work with sensitive data. Ensure no other FPGA projects or tools interfere with your setup.
Accurate emulation depends on gathering the correct data from the donor device, including configuration details like Device ID, Vendor ID, and Base Address Registers (BARs).
-
Install Arbor
- Download from Arbor’s Website and create an account to access the trial version.
- Install the software on your host system.
-
Scan PCIe Devices
- Launch Arbor and navigate to the Local System tab.
- Click Scan/Rescan to detect all connected PCIe devices.
-
Identify the Donor Device
- Locate the donor device in the scan results and select it to view its detailed configuration.
-
Open a Terminal
-
List PCI Devices
- Run the following command:
lspci -vvv
- Run the following command:
-
Identify Donor Device
- Locate the donor PCIe device by name or class (e.g., a network adapter).
-
Extract Information
- Take note of important information such as Device ID, Vendor ID, Subsystem ID, BAR sizes, and capabilities.
-
Download PCI-Z
- Download from the PCI-Z Website.
-
Run PCI-Z
- Launch the software and view the list of connected PCI devices.
-
Identify Donor Device
- Locate the device from the list and extract the necessary information.
For accurate emulation, record the following details from the donor device:
- Device ID: The unique identifier for the hardware device.
- Vendor ID: The identifier for the manufacturer.
- Subsystem ID: The identifier for any subsystems associated with the device.
- Revision ID: The version number of the hardware.
- BARs (Base Address Registers): The size, type (memory or I/O), and layout of the device’s memory regions.
- Capabilities: Features like MSI/MSI-X, power management, and PCIe link speed.
- Device Serial Number (DSN): If available, the unique serial number of the device.
The PCIe configuration space defines how the FPGA is identified by the host system. To spoof the donor device, you’ll need to customize this space with the correct values.
-
Open
pcileech_pcie_cfg_a7.svin Visual Studio Code. -
Modify Device and Vendor IDs
- Replace the default Device ID and Vendor ID with the values from the donor device:
cfg_deviceid <= 16'h1234; // Replace with donor Device ID cfg_vendorid <= 16'hABCD; // Replace with donor Vendor ID
- Replace the default Device ID and Vendor ID with the values from the donor device:
-
Modify Subsystem IDs
- Update the Subsystem IDs to match the donor device:
cfg_subsysid <= 16'h5678; // Replace with donor Subsystem ID cfg_subsysvid <= 16'hABCD; // Typically same as Vendor ID
- Update the Subsystem IDs to match the donor device:
-
Set Revision IDs and Class Codes
- Set the device class and revision:
cfg_rev_id <= 8'h01; // Replace with donor Revision ID cfg_class_code <= 24'h020000; // Example for Network Controller
- Set the device class and revision:
-
Configure BARs
- Define the number and size of the Base Address Registers (BARs):
cfg_bar_0 <= 32'hFFFFFFF0; // 32-bit Memory BAR
- Define the number and size of the Base Address Registers (BARs):
-
Locate DSN Configuration
- Insert the donor device’s serial number:
cfg_dsn <= 64'h01000000684CE000; // Replace with donor DSN
- Insert the donor device’s serial number:
-
Handle Missing DSN
- If the donor device does not have a DSN, use zeros or generate a unique DSN:
cfg_dsn <= 64'h0000000000000000;
- If the donor device does not have a DSN, use zeros or generate a unique DSN:
-
Open Vivado
-
Launch the Tcl Console
- In the Vivado interface, open the Tcl Console from the toolbar or navigation menu.
-
Navigate to the Project Directory
- Use the
cdcommand to navigate to the PCILeech project:cd C:/path/to/pcileech-fpga/PCIeSquirrel
- Use the
-
Generate the Vivado Project
- Run the following command to generate the project:
source vivado_generate_project_squirrel.tcl -notrace
- Run the following command to generate the project:
-
Open the PCIe IP Core Configuration
- In Vivado, navigate to the Sources pane and find
pcileech_squirrel_top. - Locate the
pcie_7x_0module, right-click it, and choose Customize IP.
- In Vivado, navigate to the Sources pane and find
-
Modify Device and Vendor IDs
- In the customization window, enter the correct Device ID and Vendor ID from the donor device.
-
Configure BARs
- Set the correct number, type, and size of BARs based on the donor device’s configuration.
-
Set Revision and Class Codes
- Ensure that the revision and class codes match the donor device.
-
Lock the IP Core
- Lock the IP core to prevent Vivado from overwriting these changes during synthesis:
set_property is_managed false [get_files pcie_7x_0.xci]
- Lock the IP core to prevent Vivado from overwriting these changes during synthesis:
-
Run Synthesis
- In Vivado, click Run Synthesis to synthesize the design.
- Check for errors and warnings, resolving them as needed.
-
Run Implementation
- Once synthesis is complete, run Implementation to generate the necessary files for your FPGA.
-
Generate Bitstream
- Click Generate Bitstream and wait for the process to complete.
-
Connect FPGA via JTAG
- Connect your JTAG programmer to the FPGA and ensure it is powered on.
-
Open Vivado Hardware Manager
- In Vivado, go to Open Hardware Manager and select Open Target to detect the connected FPGA device.
-
Program the FPGA
- Right-click on the FPGA device and select Program Device. Choose the generated bitstream file and click Program.
-
Verify Device Detection
- Use Device Manager (Windows) or
lspci(Linux) to check if the FPGA is detected as the donor device.
- Use Device Manager (Windows) or
-
**Test Device Functionality
**
- Depending on the device, perform basic tests such as checking network connectivity (for network cards) or accessing storage (for storage devices).
- Test Performance
- Use benchmarking tools to verify that the device performs as expected, comparing results with the original hardware.
If the host system does not detect the FPGA, try the following:
- Check Physical Connections: Ensure the FPGA card is properly seated in the PCIe slot.
- Review Configuration Settings: Verify the correctness of the Device ID, Vendor ID, and BAR configurations.
- BIOS/UEFI Settings: Ensure that DMA protection features are disabled and that the system supports external DMA devices.
If you experience issues with memory mapping or accessing the device:
- Verify BAR Sizes and Addresses: Double-check that the BAR sizes and address ranges match those of the donor device.
- Memory Alignment: Ensure that BARs are properly aligned to the system memory.
- Resolve Overlapping BARs: Avoid assigning overlapping memory regions to different BARs.
For performance or data transfer issues:
- Optimize FIFO Sizes: Adjust FIFO buffers for better data handling.
- TLP Payload Sizes: Set the maximum TLP payload size to match the donor device’s specifications.
- PCIe Link Settings: Ensure the PCIe link speed and width match the host system's PCIe capabilities.
In Part 2, we’ll dive into advanced techniques for customizing and optimizing your custom firmware for full device emulation. This section will cover more intricate topics such as modifying PCIe parameters for specific emulation tasks, advanced debugging, and techniques for improving emulation accuracy. You’ll also learn how to incorporate additional capabilities like power management, virtual functions, and advanced PCIe features.
By the end of Part 2, you’ll be well-equipped to tackle more sophisticated firmware development challenges and optimize your FPGA design for specific use cases.
- Advanced Firmware Customization
- Emulating Device-Specific Capabilities
- Transaction Layer Packet (TLP) Emulation
- Advanced Debugging Techniques
- Emulation Accuracy and Optimizations
- Best Practices for Firmware Development
- Incorporating ekknod's FPGA DMA Firmware
- Implementing Additional Features
- Appendices
- Conclusion
- Additional Resources
- Support and Community
The accuracy of emulation greatly depends on how well the FPGA-based device replicates the donor device’s PCIe configuration, link speed, and width. Fine-tuning these settings will allow the host system to interact with the FPGA as if it were the original hardware.
-
Open
pcileech_pcie_cfg_a7.svin your editor. -
Locate the PCIe Link Speed and Width Configurations:
- Ensure the values for link speed and width are consistent with the donor device.
pcie_link_speed <= 4'b0010; // PCIe Gen2 (5 GT/s) pcie_link_width <= 8'b00000100; // x4 lanes
Example: For a Gen2 x4 PCIe device, set the link speed to
4'b0010and the width to8'b00000100.
PCIe capability pointers are essential for ensuring the host system can access advanced PCIe features like MSI/MSI-X, power management, and error reporting.
-
Set Capability Pointer:
- Adjust the capability pointer to match that of the donor device.
capability_pointer <= 8'h40; // Starting location for PCIe capabilities
-
Extended Capabilities:
- You can extend this section by adding or modifying support for features like Advanced Error Reporting (AER) or Latency Tolerance Reporting (LTR).
AER_CAP_VERSION <= 4'b0001; // Enable AER support
Base Address Registers (BARs) allow PCIe devices to map their internal memory to system memory. Proper configuration of BARs is critical for the host system to communicate with the emulated device.
-
Open
pcileech_pcie_cfg_a7.sv. -
Modify the BAR Size Values to match the donor device’s BARs.
Example: For a 16KB BAR0:
bar0_size <= 32'h00004000; // 16KB BAR0
-
Repeat for Additional BARs as needed for BAR1, BAR2, etc.
-
Ensure BAR address spaces are contiguous and don’t overlap. Overlapping BARs can cause memory mapping errors in the host system.
bar0_addr <= 32'hF0000000; // Example BAR0 address bar1_addr <= 32'hF0004000; // Example BAR1 address
-
Align Memory Regions: Make sure BARs are aligned according to the donor device’s configuration.
Properly emulating power management and interrupts is crucial for seamless system integration. The host system expects certain behaviors from the device when it transitions between power states or handles interrupts.
-
Set Power Management Capabilities:
- Adjust the power management settings to emulate the donor device's power features.
PM_CAP_VERSION <= 4'b0011; // Power Management version 3.0 PM_CAP_D1SUPPORT <= 1'b1; // Support for D1 power state PM_CAP_D2SUPPORT <= 1'b0; // No support for D2
-
Power State Transitions:
- Implement logic for transitioning between power states (D0, D1, D3) as per the donor device’s capabilities.
-
Enable MSI or MSI-X in
pcileech_pcie_cfg_a7.sv.MSI_CAP_64_BIT_ADDR_CAPABLE <= 1'b1; // Enable 64-bit address for MSI cfg_interrupt <= 1'b1; // Enable MSI interrupts
-
Route Interrupt Signals correctly to ensure that the host system receives the expected interrupt behavior from the emulated device.
assign cfg_interrupt_di = cfg_int_di; assign cfg_interrupt_assert = cfg_int_assert;
Many devices include advanced capabilities beyond standard PCIe behavior, such as custom power management or proprietary features. Emulating these device-specific capabilities enhances the fidelity of the emulation.
-
Enable AER to match the donor device’s configuration:
- Advanced Error Reporting (AER) allows the device to report and log PCIe errors.
AER_CAP_VERSION <= 4'b0001; // AER capability version 1 AER_CAP_NEXTPTR <= 8'h00; // No next capability
Ensure that the emulated device supports dynamic link speed negotiation as seen in the donor device.
- Set the PCIe Link Speed:
- The FPGA should
negotiate the link speed based on host system capabilities.
```verilog
pcie_link_speed <= 4'b0010; // Gen2 speed (5 GT/s)
```
Certain devices have proprietary features unique to the vendor. These must be carefully implemented to ensure that the FPGA behaves identically to the original device.
Use PCIe traffic analysis tools (covered in Section 14.2) to monitor TLPs and identify vendor-specific behaviors.
Create custom logic to handle any vendor-specific registers or commands.
reg [31:0] vendor_specific_reg;
always @(posedge clk) begin
if (vendor_specific_write_enable) begin
vendor_specific_reg <= vendor_specific_data_in;
end
endTLPs are the building blocks of PCIe communication. Accurately crafting TLPs ensures smooth interactions between the host system and the emulated device.
A TLP consists of three key components:
- Header: Contains control information such as address, length, and type of transaction.
- Data Payload: The actual data being transferred.
- Tail: Includes additional control information and checksums.
-
Use PCIe traffic analysis tools (e.g., Wireshark with PCIe extensions or Teledyne LeCroy Telescan PE) to monitor TLP traffic from the donor device.
-
Analyze TLP Patterns:
- Look for recurring TLP patterns, such as memory reads/writes or configuration accesses.
Once you understand the structure of TLPs, you can craft custom TLPs to match specific operations from the donor device.
// TLP for Memory Write
tlp_header <= {2'b10, 5'b00000, 3'b000, 1'b0, 1'b0, 2'b00, 10'b0000000001}; // Fmt, Type, etc.
tlp_address <= 64'h0000000012345678; // Target address
tlp_data <= 32'hDEADBEEF; // Data payload// TLP for Configuration Write
tlp_header <= {2'b10, 5'b00101, 3'b000, 1'b0, 1'b0, 2'b00, 10'b0000000001}; // Fmt, Type, etc.
tlp_address <= 32'h00000010; // Configuration register address
tlp_data <= 32'h00000001; // Data to writeThe Integrated Logic Analyzer (ILA) in Vivado allows you to capture and analyze real-time signals inside the FPGA. This tool is indispensable for debugging custom firmware.
-
Open Vivado and navigate to Tools > Insert Logic Analyzer.
-
Select Signals: Choose the signals you want to monitor (e.g., TLP data or PCIe interrupts).
-
Configure Triggers: Set up triggers to capture specific events, such as TLP generation or MSI interrupts.
In addition to Vivado’s ILA, external tools allow for comprehensive PCIe traffic analysis.
-
Install Wireshark with PCIe support.
-
Capture Traffic: Use Wireshark to capture PCIe traffic and filter for relevant TLPs.
-
Install Telescan PE for advanced PCIe protocol analysis.
-
Monitor PCIe Traffic: Capture and analyze traffic between the FPGA and host system.
To ensure seamless integration and undetectable device emulation, follow these advanced optimization techniques.
Ensure that all clocks within the FPGA are synchronized with the PCIe link clock to avoid timing issues during communication.
clk_div <= clk / 2; // Example of clock division for synchronization-
Implement State Machines: Use state machines to manage different operational states of the emulated device.
-
Handle System Requests Dynamically: Monitor incoming TLPs and adjust the device’s response accordingly.
- Test Frequently: Regularly test the firmware after each modification.
- Document Changes: Maintain detailed documentation to track changes.
- Use Git for Version Control: Ensure every significant change is tracked.
- Branching Strategy: Use feature branches to manage complex changes.
ekknod’s firmware advancements focus on enhancing FPGA-based DMA performance, particularly for Wi-Fi and multimedia applications.
Ensure your firmware properly emulates transitions between power states to match modern energy-efficient devices.
Shadow configuration space allows you to dynamically modify PCIe configuration registers during runtime. Use this technique to emulate complex devices.
By completing Part 2: Advanced Tutorials, you have gained a deeper understanding of advanced firmware customization techniques. You are now prepared to handle complex emulation tasks, optimize your design for specific hardware, and troubleshoot advanced issues.
If you need further assistance or want to collaborate, join our Discord community: