Mitysom-A5E PCIe Gen 3x4 RP Example Project¶
This page is a work in progress. When this warning is removed, the instructions should be considered finalized.
Overview¶
This example demonstrates using the Agilex 5 PCIe Hard IP in a root port (RP) configuration at Gen 3 (8 Gbps) link rates with x4 data lanes. In this example, the m.2 M-Key (typically supporting NVME SSD drives) of the MitySOM-A5E Standard Development Kit is used for testing the interface.
The PCIe RP project is heavily based on the work provided in Altera's PCIe Root Port System Example Design for the Agilex 5 E-Series Premium Development Kit. The project files are located on their github page. At the time of this writing, the Altera reference project was supported for Quartus Pro 25.1. This example project has been advanced to Quartus Pro 25.3.1.
Supported Hardware¶
- MitySOM-A5E Standard Development Kit Baseboard (all variants)
- A5ED-B64-188-SRC-X (default)
- A5ED-B64-144-SRC-X
- A5ED-B66-188-SRC-X
- A5ED-B66-144-SRC-X
This example was successfully tested using the a revision -1 MitySOM-A5E Standard Development Kit card and the
following drives:
- Samsung 970 EVO 512 GB SSD Drive
- Samsung MZVLB256HAHQ-000L7
- Samsung 990 EVO 2 TB SSD Drive
Known Issues and Limitations¶
While some success with this project is reported, there is still ongoing work at Critical Link to
resolve the following issues and limitations:
- With properly detected and mounted drivers, several NVME completion events appear to be missed by
the IRQ handler (but determined to be complete using polling) resulting in error messages similar
to:[ 93.178195] nvme nvme0: I/O tag 640 (3280) QID 2 timeout, completion polled
when the drive is under heavy load. This has the effect of reducing throughput performance. Critical
Link is working to determine the root cause, which may include pipelining stalls in the design or
perhaps be related to testing using engineering silicon (SR0) during the testing. - Several drives tested were not successfully detected or mounted, including those listed below. Critical
Link is working with Altera to determine the root cause.- Samsung 980 EVO 1 TB SSD drive - detected but generated errors while mounted / accessed
- Intel SSDPEKKW256G7
- Toshiba THNSN5128GPU7
Requirements¶
- Quartus Prime Pro Edition 25.3.1
- AArch64 cross-compiler toolchain (e.g.,
aarch64-none-linux-gnu-gccoraarch64-linux-gnu-gcc) - Reference SD card with a compatible Linux filesystem
- This project is configured for HPS-first boot
Building the FPGA Design¶
Compile the Linux Kernel, DTS, and Modules¶
The Linux Kernel source is also needed to compile dts overlays and kernel modules. Also, since we are compiling a kernel module out of tree we need to compile the Linux kernel and install it onto the SD card so that the versions will match, therefore, checkout the Linux Kernel source and compile it. This can be done following the instructions found here: Linux_Kernel
Make sure to compile the kernel Image, dtbs and modules and install them onto the SD card following the instructions in the previous link
You can use the toolchain previously source to compile the kernel. Once the kernel is compiled, set the following environment variable to point to the source.
export KERNEL_SRC=/path/to/linux-socfpga
Programming the Board¶
Flash the JIC into the on-board QSPI NOR flash via JTAG using the Quartus programmer. If multiple boards are connected, jtagconfig can be used to identify cable indices.
jtagconfig --setparam 1 JtagClock 16M quartus_pgm -c 1 -m jtag -o "ip;output_files/a5e.jic@2"
After flashing, power-cycle the board.
Also copy the updated boot.scr to the SD card boot partition:
cp scripts/boot.scr /path/to/sdcard/boot/
Running the Test(s)¶
Ensure device is properly detected by the kernel¶
Power off and power on the board. Allow the board to boot to a linux command line prompt.
If the FPGA and kernel drivers / configuration have been setup correctly, you should be able to see several messages in the kernel boot log indicating that the PCIe drivers were loaded and the SSD drive was detected by the NVME framework:
root@mity-a5e:~# dmesg | grep -i pcie [ 0.000000] OF: reserved mem: initialized node pcie-rootport-buf@83ffff000, compatible id shared-dma-pool [ 0.000000] OF: reserved mem: 0x000000088faff000..0x000000088fafffff (4 KiB) nomap non-reusable pcie-rootport-buf@83ffff000 [ 0.062690] /soc@0/pcie@200000000: Fixed dependency cycle(s) with /soc@0/pcie@200000000 [ 0.068545] /soc@0/pcie@200000000: Fixed dependency cycle(s) with /soc@0/pcie@200000000 [ 10.362405] altera-pcie 60000000.pcie: host bridge /soc@0/pcie@200000000 ranges: [ 10.375261] altera-pcie 60000000.pcie: MEM 0x0050100000..0x005fffffff -> 0x0000100000 [ 10.386059] altera-pcie 60000000.pcie: PCI host bridge to bus 0000:00 [ 10.409129] pci 0000:00:00.0: [1172:0000] type 01 class 0x060400 PCIe Root Port [ 10.459084] pci 0000:01:00.0: [144d:a808] type 00 class 0x010802 PCIe Endpoint [ 10.536734] pcieport 0000:00:00.0: enabling device (0000 -> 0002) root@mity-a5e:~# dmesg | grep -i nvme [ 12.058673] nvme nvme0: pci function 0000:01:00.0 [ 12.064028] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 12.075802] nvme nvme0: missing or invalid SUBNQN field. [ 12.082428] nvme nvme0: D3 entry latency set to 8 seconds [ 12.112861] nvme nvme0: 4/0/0 default/read/poll queues [ 12.144769] nvme0n1: p1
You can also check the PCIe Link status with the command below. The LnkSta (Link Status) will indicate the link speed and number of lanes that the interface is using.
root@mity-a5e:~# lspci -vv -s 01:00.0
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd SSD 970 EVO/PRO
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 0
Region 0: Memory at 50100000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=0 offset=00003000
PBA: BAR=0 offset=00002000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158 v1] Power Budgeting <?>
Capabilities: [168 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [188 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: nvme
Kernel modules: nvme
Checking for the nvme device:
root@mity-a5e:~# ls -ltr /dev/nvm* brw-rw---- 1 root disk 259, 1 May 29 18:48 /dev/nvme0n1p1 brw-rw---- 1 root disk 259, 0 May 29 18:48 /dev/nvme0n1 crw------- 1 root root 242, 0 May 29 18:48 /dev/nvme0
Mount the drive (optional - create a filesystem)¶
Most NVME SSD drives should present one working partition when received new. For this test, we will install an ext4 filesystem on the partition and mount it to a temporary mount location. Note: if you already have a working filesystem created on the drive, you can skip the first step below.
Create the Filesystem
root@mity-a5e:~# mkfs.ext4 /dev/nvme0n1p1
Mount the filesystem
root@mity-a5e:~# mkdir -p /mnt root@mity-a5e:~# mount /dev/nvme0n1p1 /mnt [ 595.724473] EXT4-fs (nvme0n1p1): mounted filesystem e26321f7-f86d-4dc0-95ac-ed9de55816a3 r/w with ordered data mode. Quota mode: disabled. root@mity-a5e:~# df -h /mnt Filesystem Size Used Available Use% Mounted on /dev/nvme0n1p1 228.2G 39.1G 177.4G 18% /mnt
Test Drive Performance¶
The fio command can be used to test the read/write performance of the drive. A sample command that was used is shown below. The FIO command provides a high level of flexibility in terms of active threads and types of read/write operations performed. Users are encouraged to experiment with the options and review the manual page for the tool.
root@mity-a5e:~# fio --name=nvme --filename=/mnt/test --size=31768M --rw=write --bs=512k --iodepth=8 --numjobs=1 --ioengine=libaio --direct=1 --verify=crc32c
Go to top
