POWER9 Hardware Compatibility List/PCIe Devices

From RCS Wiki
Jump to navigation Jump to search

Compatibility rules

In general, any PCIe device will work providing that an open source driver is available for your operating system. There are some exceptions:

  • Hardware bugs. POWER does not permit errant DMA accesses. If a device tries to access areas of host memory which it is not permitted to access, the device is shut down immediately. This is dissimilar to x86 platforms, which simply silently ignore such attempts. Some badly designed I/O devices have bugs causing them to attempt DMA accesses to random areas of host memory; these devices are unlikely to function correctly on POWER systems unless a workaround is available. Note that devices in full bypass mode may legally have access to all host memory, to avoid this and test driver IOMMU setup pass "iommu=nobypass" to the kernel at startup.
  • I/O space. Starting with POWER9, access to the legacy PCI I/O space is no longer supported; devices or drivers which rely on this will not function. The legacy I/O space has been deprecated for as long as PCIe has existed; generally this will only affect very old PCIe devices which use PCIe to PCI bridge chips to attach old PCI devices to the bus, or genuine legacy PCI devices attached via PCIe/PCI Expansion. A small subset of these devices may require legacy I/O space support.
  • Incomplete memory addressing support. The PCIe architecture specifies a 64-bit address space. Some I/O devices try to economize on this by only implementing e.g. 40 bits for their addressing, rendering them incapable of addressing host memory which lies above address 240. (Firmware patches to work around this are pending.)
  • Bifurcation limits. Arbitrary PCIe lane bifurcation is not supported. Devices which split a PCIe slot into multiple connectors (for example, PCIe to M.2 adaptors) will not work unless they have a PCIe switch chip, although the first connector will generally work.

Troubleshooting

If a PCIe device is in a broken state due to being attached/detached from a VM, or due to a transition from Petitboot to the main OS, you may be able to fix it by issuing a hot reset. A script for performing a PCIe hot reset is at File:Pcie hot reset.sh. For background on PCIe resets and how a hot reset differs from the function-level reset performed by echo 1 > /sys/bus/pci/devices/$dev/reset, see Alex Forencich's explanation on Stack Exchange.

NICs

Working

  • 3Com 3C905C-TX-M EtherLink 10/100 PCI Ethernet (via PCIe/PCI Expansion)
  • Broadcom BCM5719
  • Chelsio T520-SO-CR (dual port 10Gb/s, cxgb4 driver)
  • Chelsio T6225-SO-CR
  • DEC FDDIcontroller/PCI (DEFPA) PCI FDDI (via PCIe/PCI Expansion)
    • Works automatically as from Linux kernel version 5.13.
    • Works with Linux kernel versions before 5.13 as long as the defxx driver has been compiled with the CONFIG_DEFXX_MMIO option, which may not be the case with standard distribution kernels as the option was not on by default for historical reasons.
      This is because the PFI ASIC used as the PCI interface with the DEC FDDIcontroller/PCI adapter supports both port I/O and MMIO for main ASIC's (PDQ) CSR access, however as noted above the Power Systems Host Bridge 4 (PHB4) PCIe root complex used with POWER9 microprocessors does not support I/O Read or I/O Write commands required for port I/O.
  • Mellanox ConnectX-6 EN 200Gb/s Adapter Card (supports CAPI)
  • Silicom PE210G2SPI9/PE210G2SPI9A/PE210G2SPI9B dual port 10 GbE SFP+ adapter (Chipset: Intel 82599ES) (ixgbe driver)
  • Silicom PE310G4SPI9/PE310G4SPI9LA quad port 10 GbE SFP+ adapter (Chipset: Two Intel 82599ES controllers behind one PLX PEX 8724 switch) (ixgbe driver)
  • Unbranded 4-port Gigabit Ethernet adapter (Chipset: Four Realtek RTL8111F controllers behind one ASMedia ASM1184e switch) (r8169 driver, firmware optional)
    • The card be identified by the "NET111-V1.0" text on its PCB.
  • ASUS XG-C100F (single port 10GbE SFP+, AQUANTIA AQtion Linux "atlantic" driver)
    • Driver compiled by default on PowerPC (and others) from this commit.
    • There is proprietary firmware written into EEPROM from the factory but it does not seem to be writeable, nor does it have to be loaded by the Linux kernel.

Non-working

  • Mellanox ConnectX IB QDR (mlx4 driver)

Wireless Cards

Working

  • Alfa AWUS036NHA Wireless USB Adaptor
    • open source firmware (ath9k_htc) [1]
  • Intel Dual-Band Wireless-AC 7260 (Bluetooth untested)
  • Intel Dual-Band Wireless-AC 8260 (Bluetooth untested)
  • Intel Dual-Band Wireless-AC 9260 (Bluetooth untested)
  • Intel® Wi-Fi 6 AX200 (Bluetooth untested)
    • When loading older firmware versions (iwlwifi-cc-a0-50.ucode), and EEH is triggered. Disabling the iommu is required for functionality. Note, tx bitrate is not reported correctly.
    • When using recent firmware (iwlwifi-cc-a0-71.ucode), the card operates as expected.
  • ThinkPenguin TPE-NMPCIE
    • Chipset: Atheros AR9281
    • Linux driver: ath9k
    • Tested with StarTech PEX2MPEX; device is detected without trouble by Linux and NetworkManager; didn't try hooking up an antenna, so wasn't able to try connecting to networks.
  • TP-Link TL WN823N RTL8192EU [2]
    • remove "ARCH=$(ARCH)" on line 1710 of the makefile and it compiles fine
  • TP-LINK TL-WN725N V2 USB dongle
    • as of Linux 5.6.x, the rtl8188eu is in staging stage, so it is advised to compile the driver from [lwfinger/rtl8188eu]
  • Asus PCE-AX3000 Both Wi-Fi and Bluetooth worked out of the box with Ubuntu 21.10 (Bluetooth requires an additional USB connection, an internal USB to 9-pin cable works)

NVMe Drives

  • Samsung 950 PRO (with M.2 to PCIe adapter)
  • Samsung 960 EVO / PRO (with M.2 to PCIe adapter)
  • Samsung 970 PRO (with M.2 to PCIe adapter)
  • Samsung 970 EVO Plus (with M.2 to PCIe adapter)
  • Samsung 980 PRO (with M.2 to PCIe adapter)
  • Samsung PM983 (with AOC-SHG3-4M2P M.2 to PCIe adapter)
  • Intel SSD DC P3600 PCIe AIC (tested 1.6 TB)
  • Intel Optane 900P NVMe XPoint PCIe
  • Intel Optane 905P NVMe XPoint PCIe AIC
  • Intel Optane 905P NVMe Xpoint U.2, with included U.2 to M.2 cable plugged into an ASUS Hyper M.2 X4 mini.
  • WD Black PCIe (with M.2 to PCIe adapter)
  • MyDigitalSSD BPX 480GB (with M.2 to PCIe adapter)
  • Silicon Power US70 2000 GB (note that it is configured with 512 byte sectors by default and should be reformatted with nvme-cli to get better performance)

Known issues:

PCIe to M.2 Adapters

Working

Partially working

PCIe to U.2 Adapters

Working

  • High Point SSD7120 PCIe 3.0 x16 to 4x U.2 NVMe ports (Dedicated PCIe 3.0 x4 per port, with PCIe switch) tested by Gyakovlev (talk)
    • Tested with Icy Dock ToughArmor MB699VP-B 4xU2 enclosure. Neither above controller nor enclosure ships with cables, 4x SAS HD SFF-8643 cables required to connect drives.
    • 4x Optane 905P work fine with this combo.
  • High Point SSD7580A U.2 NVMe RAID controller (PCIe 4.0 x16, allows eight U.2 NVMe PCIe 4.0 SSDs to be connected)

PCIe to MiniPCIe Adapters

Working

  • StarTech PEX2MPEX (NewEgg)
    • Tested with ThinkPenguin TPE-NMPCIE.
    • Particularly of interest as a lot of the ASPEED and SiliconMotion GPU's have a MiniPCIe form factor.

SAS/SATA Storage Controllers

Working

  • IOCrest SI-PEX40062 (Chipset: Marvell 88SE9235, PCI id 1B4B:9235)
    • Marvell 88SE9230 chipset also confirmed to work
  • Kouwell PE-115H (Chipset: Marvell 88SE9130, PCI id 1b4b:9130)
  • LSI 9300/9200 SAS HBAs
    • May require updating to IT firmware on a x86 machine
  • PM8068-based SAS HBAs
  • Supermicro AOC-SLG3-4E2P 4-port OCuLink adapter
  • Jmicron JMB 363 SATA PCIe card. SATA ports work with Petitboot.
  • Broadcom MegaRAID 9460-8i
  • Broadcom MegaRAID 9460-16i
  • IOCrest IO-M2F585-5I (Chipset: JMicron JMB585, PCI IDs: 197b:0585)
  • Unbranded JMicron JMB363 SATA/IDE controller card, with one eSATA, one internal SATA, and one IDE (PATA) connector (Chipset: JMicron JMB363, PCI IDs: 197b:2363)
    • SATA ports work in Petitboot.
    • The IDE/PATA port doesn't work since it exclusively uses PCI I/O space access, which the POWER9 does not support. Because of this, PCI function 1, which is used for the IDE/PATA functionality, is not exposed by Linux and so will not appear in the output of lspci.

Non-working

  • AXAGON PCES-SA2 (ASMedia chipset)
  • SuperMicro AOC-SASLP-MV8 (mvsas driver)
  • MegaRAID 9341-8i - probably a bug in the firmware
  • Unbranded ASMedia ASM1166 SATA controller cards with six SATA connectors (Chipset: ASMedia ASM1166, PCI IDs: 1b21:1166)
    • Cards appear to be sold under different brands and with slightly different PCB layouts, but they can be identified by the following markings on their PCBs:
      • "PCE6SAT-A01" and "VER006S"
      • "PCI-E 3.0 TO SATA.6GB - 6Port" and "SU-SA3026"
    • Causes a PHB endpoint freeze during Skiboot's initialization of the PCI Express device, so the issues this chip is having can't be worked around with just a kernel patch.

Unknown if working (good candidates to test)

Optical Drives


Graphics Cards

No display? Check out the GPU Troubleshooting page.

AMD

All AMD GPUs currently have DMA issues (limited to 32-bit, which can cause crashes) due to missing Linux kernel support for DMA windows between 33 and 63 bits in length. The root cause is GPU vendors (and occasionally some non-GPU vendors) cutting costs and only including 40-bit capable (Intel-style) DMA controllers. A compatibility mode is expected to be included in Linux 5.4 and above that will resolve this issue.

  • AMD Radeon HD 5570 - Archaic (1GB VRAM, PCI 2.1) but much faster than the AST. This card (ASUS EAH5570 Silent) is passively cooled.
  • AMD Radeon HD 5850 - Must disable onboard VGA first. Currently has issues with only using 32-bit DMA.
  • AMD Radeon HD 6450 - Works with default settings (kernel: radeon, X: modesetting or radeon), tested in BE mode
  • AMD Radeon HD 6850 - Disable AST VGA with jumper. 32 bit.
  • AMD Radeon HD 7850 - Disabled onboard VGA. Using amdgpu is highly unstable, radeon driver is usable but has issues with only using 32-bit DMA.
  • AMD Radeon HD 7950 - Must disable onboard VGA first. Currently has issues with only using 32-bit DMA.
  • AMD Radeon R5 220
  • AMD Radeon R5 230 - Works in BE mode (use Option "AccelMethod" "EXA" for Xorg)
  • AMD Radeon R7 240
  • AMD Radeon R9 290X
  • AMD Radeon R9 Nano - Must disable onboard VGA first. Works perfectly with Linux 5.6.x to 5.12.x with 4K Pages. It is confirmed working 5.6.x and 5.10.6 (or newer) with 64K Pages (occasionally crash under heavy load for example Blender rendering). Unfortunately 5.7.x, 5.8.x, 5.9.x, 5.11.1 -> 5.11.11 and 5.12.x with 64K pages are known to crash. For 5.7.x (64K pages), you could workaround the crash by adding `amdgpu.dc=0` to grub config.
  • Sapphire GPRO 8200 (Polaris10 core) - Disable AST VGA with jumper, disabling in grub is not enough. Same form factor as WX7100, a single-slot RX 470 with 8GB of RAM and 4 DP outputs.
  • AMD Radeon RX 480
  • AMD Radeon RX 550 - Works with caveats (below) for particular card tested.
    • Card tested was Gigabyte GV-RX550D5-2GD in a Blackbird with Ubuntu 19.10, 5.3.0-24 and amdgpu with onboard VGA disabled by jumper. Suspect with tweaking would work without needing to disable VGA. Alas would lock up every day or two, to the point I replaced with a Sapphire RX580. Based on conversations w/Raptor suspect this was an issue with the model card I had rather than the RX550 itself.
  • AMD Radeon RX 560X
  • AMD Radeon RX 580 – Works with the amdgpu firmware from Ubuntu 19.04 and disabled onboard VGA
    • You may also need a kernel parameter like vga=797 if xinit complains about VESA[3]
    • The Sapphire NITRO+ Radeon RX 580 8GB card is a tight fit on the Blackbird planar if you're using the 2nd PCIe slot as it a bit wider than two slots (and is documented as such). Solved by removing the adapter plate for the PCIe-M2 adapter card being used and allowing it to sit in slot untethered.
  • AMD Radeon RX Vega 56 - Works with Debian Buster with amdgpu. Requires patches to work, somewhat unstable but usable. Cannot use AST Integrated VGA and AMDGPU at the same time without causing conflict. Not tested at this moment for use in petitboot or firmware.
  • AMD Radeon RX Vega 64 - Works with Fedora 32 with Linux kernel 5.5.0's amdgpu. The card does _NOT_ display in bootloader because vega10 firmwares failed to load correctly.
  • AMD Radeon Pro WX4100 (Polaris11 core) - May need at least linux 4.16 in order to get Xorg to work.
  • AMD Radeon Pro WX5100
  • AMD Radeon Pro WX7100 (Polaris10 core) - Available pre-installed on Talos II workstation, server, and desktop configurations.
  • AMD Radeon RX 5700 (Navi 10) - Requires kernel patches to enable display output.
  • AMD Radeon RX 5700 XT (Navi 10) - Requires kernel patches to enable display output.
  • AMD Radeon Pro W5500 (Navi 14) - "amdgpu" driver works with 4k kernel but not 64k. "fbdev" driver works with both page sizes.
  • AMD Radeon RX 6800 - see RX 6900 XT
  • AMD Radeon RX 6800 XT - see RX 6900 XT
  • AMD Radeon RX 6900 XT (Big Navi, Navi 21) - Not working with current 5.12 (in either 4K or 64K page sizes) - bug ticket

The core name is important when storing the firmware into the BOOTKERNFW partition in PNOR for use by skiroot.

DisplayLink

NVIDIA

  • NVIDIA Corporation G96 [GeForce 9500 GT] (rev a1) - Works in petitboot if onboard VGA is disabled. Currently has issues with only using 32-bit DMA. No firmware needed.
  • NVIDIA RTX 2070 - usable for compute, but not 3D acceleration; integrated by Raptor as part of the Talos II PowerAI Development System configuration

Other

  • ASPEED AST2500. Works in both the main OS (LE mode) and Petitboot. BE mode partially works (doesn't crash, but colors are wrong unless you apply a patch that is harmful to performance). On Linux 5.6+, 1920x1200 resolution is broken. ast Linux driver.

Non-working

Unknown if working (good candidates to test)

Sound Cards

Working

  • Creative Sound Blaster Audigy FX SB1570 PCIe 5.1 Sound Card
  • Creative Sound Blaster X-Fi Xtreme Fidelity PCIe Audio Sound Card (SB0880)
  • Creative Sound BlasterX AE-5 Plus worked out of the box with Ubuntu 21.10 (the audio config needs changing to use the stereo output otherwise the default is the S/PDIF)
  • AMD Radeon HD 5850 and 7950 (HDMI audio)
  • VANTEC NBA-120U (USB)
  • Sabrent USB External Stereo Sound Adapter (AU-MMSA)
  • Mackie Onyx Blackjack (USB) Recording Interface
  • RME HDSPe AIO (FreeBSD tested)
  • Leveraged Sabrent Bluetooth 4.0 USB adapter (model BT-UB40) to connect to wireless Bluetooth headphones, specifically Bose Quiet Comfort 35.
  • Bluetooth audio from an Asus PCE-AX3000 connecting to various devices, with the media keys also working to control playback

Non-working

  • ASUS Xonar SE - Contains ASMedia USB host controller with errant DMA access flaw
  • Creative Sound Blaster Audigy PCIe RX 7.1 - unable to enable emu10k1 driver on little-endian power9 kernel as driver requires ZONE_DMA

USB Host Controllers

Working

  • Insignia USB 3.0 PCI-e NS-PCCUP53 V1.0 (Chipset: NEC D720202)
  • AGAXO PCEU-23R (Chipset: Renesas uPD720202, PCI id 1912:0015)
  • Terminus Technology Inc. FE 2.1 7-port Hub
  • Sonnet Allegro Pro USB 3.0 PCIe USB3-PRO-4PM-E (Chipset: Four Fresco Logic FL1100EX controllers behind one PLX PEX 8608 switch)
  • Vantec UGT-PC345 4 Port USB 3.0 PCIe w/ Internal 20 pin (Chipset: Renesas uDP720201)
  • [4] (monster card with 8 ports of USB 3.0) four Renesas uPD720202 chips behind ASMedia 1806, card can be found under different brands but can be identified by the look. This should be the same card
  • En-Labs PCI-e to 4 Ports USB 3.1 GEN 1 (5Gbps) (USB Type-C +USB Type A w/ Internal 19Pin USB 3.0 Dual Port) PCI Express Card (Chipset: ASMedia ASM1042A)
    • Card appears to be sold under different brands, but can be identified by the "PCE3U1T-A31" and "VER 006S" markings on the PCB.
    • Requires kernel version 4.14.226, 4.19.181, 5.4.106, 5.10.24, 5.11.7, or later.
      • Petitboot's kernel will need to be updated to one of these versions (or later) in order to boot from any USB devices connected to this controller.
  • IOCrest SI-PEX20189 (Chipset: ASMedia ASM1142)
    • Requires kernel version 4.14.194, 4.19.139, 5.4.58, 5.7.15, 5.8.1, or later.
      • Petitboot's kernel will need to be updated to one of these versions (or later) in order to boot from any USB devices connected to this controller.
    • EEH errors may occur during long reads from multiple devices.
  • Ableconn PU31-2C-2 (Chipset: ASMedia ASM2142)
    • Requires kernel version 4.14.190, 4.19.135, 5.4.54, 5.7.11, 5.8.0, or later.
      • Petitboot's kernel will need to be updated to one of these versions (or later) in order to boot from any USB devices connected to this controller.
    • EEH errors may occur during long reads from multiple devices.
  • Semoic USB 3.1 to Type-C 2 Port Expansion Card (Chipset: ASMedia ASM3142, same PCI IDs as ASM2142)
    • Card appears to be sold under different brands, but can be identified by the "PCE2TYC-A31", "VER006", "USB 3.1 Type-C 2-Port Card", and "PCE-E 4X" markings on the PCB.
    • Requires kernel version 4.14.190, 4.19.135, 5.4.54, 5.7.11, 5.8.0, or later.
      • Petitboot's kernel will need to be updated to one of these versions (or later) in order to boot from any USB devices connected to this controller.
    • EEH errors may occur during long reads from multiple devices.
  • ORICO PE20-1C (Chipset: ASMedia ASM3242)
    • Requires kernel version 4.14.226, 4.19.181, 5.4.106, 5.10.24, 5.11.7, or later.
      • Petitboot's kernel will need to be updated to one of these versions (or later) in order to boot from any USB devices connected to this controller.

Non-working

In general, USB3 host controllers based on ASMedia chipsets are known to be problematic, due to ASMedia hardware or firmware bugs causing errant DMA accesses to invalid regions of host memory.

  • AXAGON PCEU-43V - chipset Via VL805 - PCI id 1106:3483
  • StarTech PEXUSB314A2V - 2x ASM1142 host controllers and a PCIe switch
    • This card completely fails to be detected.
  • QNINE USB 3.1 Gen2 (Type-A and Type-C) - ASM1142
    • This may work now with an updated kernel, but it hasn't been tested yet.
  • Rosewill RC-509 - ASM1142
    • This may work now with an updated kernel, but it hasn't been tested yet.
  • HighPoint RocketU 1022A (Chipset: ASMedia ASM1042)
    • Skiboot reports that the PCIe link is unstable when the card is connected directly, but it seems to work when the card is plugged in via a PCIe switch.
    • This chip seems somewhat unreliable, since USB reads can fail after only a few tens of gigabytes have been transferred.
  • HighPoint RocketU 1144A (Chipset: Chipset: Four ASMedia ASM1042 controllers behind one PLX PEX 8609 switch)
    • Skiboot sometimes times out when scanning for the ASM1042 controllers attached to the PEX 8609 ("Timeout waiting for downstream link"), resulting in some of the ports effectively being disabled until the next boot.
    • The ASM1042 controller seems somewhat unreliable.
      • Lots of resets on USB 3.
      • Long reads from a single USB hard drive can sometimes result in I/O errors.
      • Incompatible with some USB hard drives.

Unknown if working (good candidates to test)

TV Tuners

  • Hauppauge WinTV-quadHD (Chipset: Four Silicon Labs Si2157 tuners, four LG LG3306A demodulators, and two Conexant CX23888 PCIe interface chips behind one Pericom PI7C9X2G304EL PCIe switch)
  • Hauppauge WinTV HVR-850 (2040:7240) - ATSC - using Kaffeine

Firewire Host Controllers

Working

Unknown if working (good candidates to test)

Video Capture Cards

  • AVerMedia CE310B (Chipset: Conexant CX23888)
    • Requires kernel version 4.19.106, 5.4.22, 5.5.6, or later.

Serial Port Adapter Cards

Working

  • EXSYS EX-44072 PCI-Express 2x Serial RS-232 Board (Chipset: Oxford OXPCIe952)
  • EXSYS EX-44073 PCI-Express 2x Serial RS-232 Board (Chipset: Oxford OXPCIe952)
  • EXSYS EX-44171 PCI-Express 1x Serial RS-232 / 1x Parallel Multi I/O Card (Chipset: Oxford OXPCIe952)
    • The UARTs implemented with the Oxford Semiconductor OXPCIe952 PCIe ASIC can be strapped for either native or legacy operation. The EXSYS boards configure it for the native mode and therefore work with the PHB4 just fine.
      NB the PC parallel port is always a legacy PCIe device and therefore cannot work with the PHB4.

PCIe/PCI Expansion

Working

Retrieved from "https://wiki.raptorcs.com/w/index.php?title=POWER9_Hardware_Compatibility_List/PCIe_Devices&oldid=3782"