Updating Firmware

Purpose

This guide describes how to upgrade the firmware of your RCS OpenPOWER machine.

Applicability

All RCS OpenPOWER systems.

Overview

Your Raptor Computing Systems POWER9 mainboard contains three primary firmware components — a system control FPGA bitstream, the BMC software stack, and the host OpenPOWER firmware (PNOR). The BMC and host PNOR are easily upgradeable over a network connection.

In contrast, the FPGA bitstream is rarely changed. Should an FPGA bitstream update be desired, a direct SPI programming connection to a Flashrom compatible system is required via the provided FPGA programming header.

A list of current firmware versions for each supported product is available on the pages shown below:

• Talos II
• Blackbird

While we strongly encourage you to compile your own firmware components from the provided source, we also provide prebuilt firmware images for download. Please note that, in general, the only way to guarantee that you or your organization's security requirements are met is to download and audit the provided source code, then compile from that audited codebase. This is not unique to our systems; the nature of all software is that the binary form may be very difficult to analyze for undesired or unintended operation.

Updating the BMC firmware

Installed System Package revision v2.00 or higher

Starting with System Package v2.00, there is an easy point+click upgrade mechanism available for your BMC and PNOR components. Simply navigate to https://<your BMC IP address>, log in with your root credentials, and select Server Configuration-->Firmware.

We will also continue to support the SSH-based update mechanism, which is appropriate for large enterprise deployments and advanced users. Starting with System Package v2.00 installed on the BMC, you can transfer a new image-bmc file to the BMC and reboot to install the update without worrying about erasing any critical data from Flash.

Installed System Package revision lower than v2.00

Note: The BMC should never need to be fully reprogrammed. Erasing the entire BMC flash device will also erase U-Boot and its associated environment variables, requiring that they be reloaded from information printed on the mainboard itself. In particular, from the factory, the IPMI MAC address is stored in both the U-Boot loader via the default environment variable string, and the currently active U-Boot environment variables. This IPMI MAC address may also be found on the mainboard below CPU0 should you need to reprogram it for any reason.

The preferred method of updating the BMC is to take the BMC update files that you have either compiled or obtained from official Raptor Computing Systems sources, and to upload them to the BMC. Once uploaded, the BMC is able to self-update.

This procedure can be done with the host powered on, but it might result in artifacts such as power button starting to blink until the host is rebooted.

Transfer update files

Perform the upgrade by transferring the following two files to /run/initramfs/ on the BMC:

• image-kernel
• image-rofs

scp or any similar utility is capable of transferring the upgrade files. To avoid running out of disk space by placing these files in temporary directories, you can transfer them directly into /run/initramfs/ with something like scp image-{kernel,rofs} root@$TALOS_BMC_ADDR:/run/initramfs/.

Apply update

After transfer, reboot the BMC via the reboot command over SSH or the local BMC serial console. The firmware update will be applied automatically.

Default BMC login information is contained in the User's Guide.

Updating the OpenPOWER firmware

The host PNOR flash device, which contains the OpenPOWER firmware (Hostboot, Skiboot, and other host-level firmware components required to IPL your POWER9 system), can be modified in its entirety via the BMC.

With chassis power off, but standby power on, transfer the upgrade .pnor file to the BMC's /tmp/ directory, like this:

$ scp ./output/images/talos.pnor root@$TALOS_BMC_ADDR:/tmp/

Once the transfer is complete, log in to the BMC via SSH.

Execute the following command:

root@talos:~# pflash -E -p /tmp/<your PNOR file>

For all currently released system package versions, also execute the following command:

root@talos:~# pflash -P CVPD -c

If no errors occur with either command, the upgrade is complete. You should probably delete the .pnor file from the BMC's /tmp/ so that the BMC doesn't risk running out of RAM. Once you've done this, you may now power on and use your updated system.

The host may power itself off and back on up to two times before booting into the OS; this is normal after a firmware update.

Updating the FPGA firmware

Referencing the schematics provided on the included DVD, carefully connect your SPI programmer to the FPGA programming header. Apply standby power to your system but do not turn it on.

Using a current version of Flashrom with the upstream Atmel enablement patch applied, flash the new FPGA firmware binary to the on-board storage device.

Complete the upgrade by removing all power; wait for all LEDs on your system to extinguish, then reapply power. The upgrade is now complete and and you may use your system normally.

Troubleshooting

If Flashrom fails to recognize the chip, it may give errors such as "RDID byte 0 parity violation". If retrying the command does not work, remove power and move the FPGA switch (next to the OCuLink connector) from "RUN" to "RESET", then reapply standby power and try again.

If Flashrom recognizes the chip but repeatedly errors when trying to program it, try shortening the length of the wires connecting your programmer to the FPGA programming header and/or lowering the speed at which your programmer clocks the SPI bus before trying again. Ideally, connecting wires should be 3 inches (7½ cm) in length or shorter.

If using the Bus Pirate with Flashrom, the default SPI speed can be lowered using the `spispeed` parameter. For example, flashrom -p buspirate_spi:dev=/dev/ttyUSB0,spispeed=30K -c AT25DF021A -V -w Talos-ii-system-fpga-v1.08.rom - 30 kHz is the lowest speed option offered by Flashrom.

If your Talos II rev 1.00 board does not work after updating to FPGA firmware v1.08, use v1.07 instead. Owing to its use of an external oscillator only present on v1.01 and newer Talos II boards (and Blackbird), FPGA firmware v1.08 is incompatible with the rev 1.00 Talos II board.

Example

You need an ISP programmer, e.g. Bus Pirate or CH341a and a second computer with Linux with installed Flashrom. Version 1.1 already contains the required patch.

Using the Bus Pirate (v3.6 is recommended):

Read the steps below completely before you start.

Attention: If at any time an error occurs or the output differs from the one given here, check the previous steps or contact us on IRC.

---

Optional step: udev rule

Create the file /etc/udev/rules.d/98-buspirate.rules, and insert the following line:

SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", GROUP="users", MODE="0666", SYMLINK+="buspirate"

This has the advantage that the path to the programmer is always the same, no matter to which USB port the programmer is connected.

You must then reboot your computer or reload your udev rules with the following command (as root):

udevadm control --reload-rules

---

• connect the programmer to the second computer via USB
• disconnect the Talos/Blackbird from the power supply (pull the power cord or switch off the PSU)
• do not connect VCC to the FPGA header. The chip gets the power supply from the PSU of the Talos/Blackbird. Wait until the LEDs go out.
• connect the remaining wires of the programming cable to the FPGA header (see manual or schematics on the DVD)
• do not boot the Talos/Blackbird, just reconnect the power supply so that the LEDs light up again. Wait at least 30 seconds after you run the next point.
• check the connection between programmer and Talos/Blackbird with the following command in the terminal (as root):

  flashrom -p buspirate_spi:dev=/dev/buspirate

• for CH341a:

  flashrom -p ch341a_spi

You should get the following output:

flashrom v1.1 on Linux (x86_64)
flashrom is free software, get the source code at https://flashrom.org

Calibrating delay loop... OK.
Bus Pirate firmware 6.1 and older does not support SPI speeds above 2 MHz. Limiting speed to 2 MHz.
It is recommended to upgrade to firmware 6.2 or newer.
Found Atmel flash chip "AT25DF021A" (256 kB, SPI) on buspirate_spi.
 
No operations were specified.

• You don’t need to upgrade the firmware
• If you get a message like „unknown Atmel chip detected“, you carried out the command too quickly. Wait a little longer and try again.

• download the required firmware from the wiki.raptorcs.com site
• open a terminal and change to the directory where the firmware is located, e.g. Downloads
• execute the following command in the terminal (as root):

  flashrom -p buspirate_spi:dev=/dev/buspirate,spispeed=1M -w filename-of-the-firmware.bin -V

• for CH341a:

  flashrom -p ch341a_spi -w filename-of-the-firmware.bin -V

The command above deletes, writes and verifies the chip. You don’t need a higher spispeed as 1M. The flashing process only takes seconds.

At the end, an output should appear as follows:

Erase/write done.
Verifying flash... VERIFIED

• disconnect the power supply of the Talos/Blackbird until the LEDs go out. Wait 30 seconds.
• remove the Bus Pirate wires from the FPGA header.
• connect (or switch on) the power supply of the Talos/Blackbird.
• now you can boot the Talos/Blackbird again.

It may take a little longer for the first boot process. Do not panic.

Troubleshooting

Some hints for resolving firmware-related problems can be found here.

If OpenBMC works but you have boot problems into petitboot you can attach a serial (RS232) cable with a null modem adapter to your Raptor Talos or Blackbird motherboard and start logging the boot process on the other (Linux) computer with:

   sudo cat /dev/ttyUSB0 115200n8 | tee bootlog.txt

Replace the ttyUSB0 device name with the device name of your serial port. You can find the recognized serial ports via

   dmesg | egrep --color 'serial|ttyS'

Then switch on your Raptor via the power button (warm start).

If your other computer does not have serial RS232 port you can use an USB-to-serial converter cable and null modem adapter like these ones:

• VTOP USB serial RS232 Adapter - USB to RS232 Serial DB9 9 PIN COM Port Converter cable (about 20 USD)
• DeLock Adapter Sub-D 9Pin Bu/Bu null modem