Troubleshooting/GPU

Background

Because OpenPOWER systems do not have a legacy graphics interface to fall back to, and as a result rely heavily on the running operating system and its drivers to handle display tasks, a few rough edges are exposed. This page attempts to document the current status of these rough edges and suggested workarounds pending actual fixes.

Common Issues

Bootloader does not show up on monitor(s) attached to a discrete GPU

Most modern discrete GPUs require firmware. As Talos™ II is aimed at a security-conscious audience, we do not currently include GPU firmware in the production firmware images. Instructions are available in the Users Guide to add firmware for your GPU to the PNOR if needed. Note that any added firmware may be able to access and modify data associated with the affected device(s); we strongly recommend you perform a security risk analysis before loading any firmware, and select open firmware where/if it is available.

If you are using a GPU that does not require firmware, or have already added any needed firmware files to the host PNOR, please ensure that the on-board VGA disable jumper (J10109) is capped. The bootloader output will preferentially show up on the on-board VGA port if it remains enabled.

Alternatively, you either use a serial console or VGA monitor / adapter to interact with the bootloader.

My AMD GPU works in petitboot but not the subsequent Linux OS

Older versions of the amdgpu driver (Linux 4.15 and below) have a bug where the connected outputs will not re-initialize after a kexec() while the driver is loaded. Kernel 4.16 and above does not appear to have this problem.

If you need to use kernel 4.15 or below, you can work around this issue by either:

• Enabling and using the VGA video output to access the bootloader (petitboot) -or-
• using a serial connection to control petitboot, and running the following commands prior to selecting an operating system via the petitboot menu:

echo 0 > /sys/class/vtconsole/vtcon1/bind
rmmod amdgpu

I want Petitboot via AST but the subsequent Linux OS console on a discrete GPU

If you don't want to put GPU firmware in the PNOR but still want Linux tty on the discrete graphics, you'll find that you'll always get output on the AST first no matter what. Blacklisting the ast module from loading is not sufficient on its own, you will need two kernel boot arguments:

modprobe.blacklist=ast video=offb:off

For example on Ubuntu, this can be accomplished by changing GRUB_CMDLINE_LINUX in /etc/default/grub accordingly, like

GRUB_CMDLINE_LINUX="modprobe.blacklist=ast video=offb:off"

and then running update-grub.

Xorg will not start / crashes when a discrete GPU is installed

Installing more than one GPU into an OpenPOWER system (for instance, when adding a discrete GPU) exposes all GPUs directly to the operating system -- there is no concept of a "primary" GPU like there is on x86. Xorg does not handle this gracefully, tending to crash during autoconfiguration. At least one bug report has been filed but fixing the root cause of this issue (incorrect Xorg drivers binding to underlying DRM devices) does not seem to be an Xorg priority. Furthermore, Xorg does not properly handle domains during autoconfiguration per another bug report on a similar issue. Community effort in getting proper fixes into Xorg would be very useful, as the Xorg developers may want to see that more than one or two systems are impacted by these bugs before working on resolving them.

Two workarounds are available:

Workaround 1: Disable on-board VGA

Disable the on-board VGA output via the VGA disable jumper, J10109. See the Users Guide for additional information.

Workaround 2: Select desired GPU at runtime

The workaround to keep both devices active, or to retain the ability to switch in the active operating system, is fairly simple, and consists of explicitly assigning Xorg drivers for each installed GPU. For this example we'll show how to fix Xorg on Debian with an AMD WX7100 discrete GPU installed.

Step 1: Locate Bus Numbers

root@talos:~# lspci | grep VGA
0000:01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 7100]
0005:02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)

Note the numbers to the left of the "VGA compatible controller" string. Each of these numbers is the PCI d:B:D.F number of the GPU, and is unique to the slot(s) you have your GPU(s) installed in. As a result of this slot dependence, bus IDs may differ from those shown in this example; always use your bus IDs when following the steps below. This slot dependence means that if you move your GPU to a different slot you will need to update the bus ID associated with that GPU.

Step 2: Create Xorg Configuration Snippet

root@talos:~# mkdir /etc/X11/xorg.conf.d

Create and open /etc/X11/xorg.conf.d/21-gpu-driver.conf for editing, then adjust the following template with your GPU information. Pay close attention to the BusID and Driver fields, as they must match your installed GPU(s). Note that Xorg uses decimal numbering, not hexadecimal like lspci, so you will need to convert the numbers within the colons of the lspci output to decimal in order to constrict a valid Xorg BusID. Furthermore, xorg doesn't use leading zeroes like lspci does; these must be stripped off when assembling the Xorg BusID. Finally, Xorg expects to see a BusID assembled as "PCI:B@d:D:F" (note Bus and Domain are swapped), and should not be assembled not using the format shown by lspci.

# AST2500
Section "Device"
    Identifier     "GPU0"
    Driver         "modesetting"
    BusID          "PCI:2@5:0:0"
    VendorName     "ASpeed Corporation"
EndSection

# WX7100
Section "Device"
    Identifier     "GPU1"
    Driver         "modesetting" # or amdgpu if you have xf86-video-amdgpu installed
    BusID          "PCI:1@0:0:0"
    VendorName     "AMD Corporation"
EndSection

# this is absolutely necessary, it tells xorg which GPU to use for the screen
Section "Screen"
    Identifier     "Screen0"
    Device         "GPU1"
EndSection

Save and exit the configuration snippet file, then restart Xorg. Your GPUs should now function as intended. If Xorg still does not start, make sure that the appropriate kernel driver (such as amdgpu for the example above, keep in mind that the Xorg driver and the kernel driver are separate and distinct) has been loaded:

root@talos:~# modprobe amdgpu

You can use the generic modesetting Xorg driver for AMD GPUs, or you can use amdgpu from xf86-video-amdgpu. The generic modesetting driver has been reported to work perfectly fine on a Talos with various GPUs, so there is likely no practical reason to use the driver-specific DDX.

Step 3 (optional): Disable Integrated Video

To disable the ASpeed VGA in the booted OS completely, you can use the modprobe.blacklist=ast approach on kernel command line, refer to the "I want Petitboot via AST but the subsequent Linux OS console on a discrete GPU" section above for more information. This method is universal/works on all distributions. The ASpeed VGA will still show up in lspci afterwards, which is normal, as you haven't disabled the hardware, just the driver.

With this done, it should be possible to remove the device section in the X.Org configuration file for the onboard VGA, but you can also just leave it there if you want, regardless of whether the driver is loaded or not.

There are alternative ways to blacklist the ast kernel driver. For example on Debian based systems, create a new file /etc/modprobe.d/ast-blacklist.conf and place the following line inside the new file:

blacklist ast

You may need to reboot if the ast DRM driver has already loaded. Alternatively, you may try to unbind and unload the ast driver as follows (assuming the ast driver is bound to vtcon0):

root@talos:~# echo 0 > /sys/class/vtconsole/vtcon0/bind
root@talos:~# rmmod ast

Monitor not detected in kernel 4.17+

Petitboot shows up fine, but there is no output for the host OS. It has been reported as bug 107049, the workaround is to append amdgpu.dc=0 to the kernel parameters. This is often associated with a host dmesg trace of

[drm] Cannot find any crtc or sizes

Xorg crashes or is laggy with the AST VGA GPU

Xorg seems to enable GLAMOR by default on many operating systems (such as Debian Buster). GLAMOR is a translation layer that converts 2D graphics operations to 3D graphics operations. This makes sense when 3D GPU acceleration is available, but when using a simple unaccelerated 2D GPU like the AST VGA GPU, the result is that 2D operations get converted to 3D operations by GLAMOR and are then converted back to 2D by llvmpipe, which introduces significant overhead.

In addition, on Debian Buster, GLAMOR has been observed to crash when used in conjunction with llvmpipe.

You can disable GLAMOR by saving the following text file as /usr/share/X11/xorg.conf.d/00-noglamoregl.conf:

Section "Device"
	Identifier "nogpu"
	Driver "modesetting"
	Option "Accelmethod" "none"
EndSection

Section "Module"
	Disable "glamoregl"
EndSection

This bug was probably fixed by commit 1e3c5d614ee33d9eac1d2cf6366feeb8341fc0f4 in upstream Xorg; it is unknown how quickly the fix will make its way to GNU/Linux distributions.

KDE is laggy with the AST VGA GPU

KDE's default compositor uses OpenGL. This makes sense when 3D GPU acceleration is available, but when using a simple unaccelerated 2D GPU like the AST VGA GPU, the result is that 2D operations get converted to 3D operations by KDE's compositor and are then converted back to 2D by llvmpipe, which introduces significant overhead.

To fix this, go to System Settings → Hardware → Display and Monitor → Compositor, and select XRender as the Rendering backend. You'll probably also want to select Smooth (slower) as the Scale method (it's still much faster than OpenGL, and it looks quite a bit better).

AMDGPU driver crashes after firmware update

The GPU only allows loading one firmware after an ASIC reset, so the firmware used by the skiroot kernel and the host kernel must be the same. See FreeDesktop.org bug 108585 for more details.