Compiling Firmware

Purpose

The following steps can be used to compile and update the firmware on Raptor CS's OpenPOWER systems, such as the Talos II or Blackbird.

Applicability

These specific instructions are for the Talos II, though the process for compiling firmware for other OpenPOWER systems like Blackbird should be very similar.

Requirements

• At least 25GB of free hard drive space
• 16GB of free RAM

Building on Debian

The build system (op-build) has been primarily tested using Debian Stretch. Ensure you install the following packages:

# Packages needed for OpenPOWER Firmware builds
$ sudo apt install cscope ctags libz-dev libexpat-dev python texinfo build-essential g++ git bison flex unzip libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc wget bc rsync

# Packages needed for OpenBMC builds
$ sudo apt install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat

Building on other Linux distributions

If you are on a different distribution, such as Fedora 28, a Debian chroot is recommended:

$ sudo yum install debootstrap dpkg
$ sudo debootstrap stretch debian-chroot http://archive.debian.org/debian
$ sudo mount -t proc none debian-chroot/proc/
$ sudo mount -o bind /sys/ debian-chroot/sys/
$ sudo mount -o bind /dev/shm/ debian-chroot/dev/shm/

Enter the chroot and install the needed packages:

$ sudo chroot debian-chroot/
# apt install software-properties-common locales

# Packages needed for OpenPOWER Firmware builds
$ sudo apt install cscope ctags libz-dev libexpat-dev python texinfo build-essential g++ git bison flex unzip libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc wget bc rsync

# Packages needed for OpenBMC builds
$ sudo apt install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat

Also create a user inside the chroot to build under:

$ useradd -m build-user -s /bin/bash
$ su build-user
$ cd

You can now use the chroot to build the firmware.

To enter the chroot in the future, you can run the following from any terminal:

sudo chroot debian-chroot/
su build-user
cd

Building the OpenPOWER Firmware

Downloading the sources

Raptor CS maintains a public Git repository containing the complete source code for the firmware. To download the source code for Talos systems:

git clone -b raptor-v2.10 --recursive https://gitlab.raptorengineering.com/openpower-firmware/machine-talos-ii/op-build.git

To download the source code for Blackbird systems:

git clone -b raptor-v2.10 --recursive https://gitlab.raptorengineering.com/openpower-firmware/machine-blackbird/op-build.git

Note: The master branch is often in a non-functional state. The latest firmware branch (either raptor-v2.10, raptor-v2.00 or raptor-v1.05 at the time of this update) should be used.

Building the firmware

Before building the firmware, check the README.md file to ensure that all needed packages are installed.

The firmware can then be built using the following commands:

$ cd talos-op-build
$ git submodule update
$ . op-build-env
$ export KERNEL_BITS=64 # needed when building on ppc64, or libopenssl will try to build in 32 bit mode
$ op-build talos_defconfig
$ op-build

You can pass -j<num-cores> to perform a parallel build (op-build invokes make), though this may result in very high memory usage.

If the build completes successfully, the final firmware image is at output/images/talos.pnor.

Rebuilding an individual package

To rebuild an individual package (such as Hostboot) and recreate the talos.pnor image, run:

$ op-build pkgname-rebuild openpower-pnor-rebuild

where pkgname is the name of the package to rebuild.

For example:

$ op-build hostboot-rebuild openpower-pnor-rebuild

Note when recompiling hostboot into a PNOR image with openpower-pnor-rebuild, it is usually recommended to force a machine XML rebuild as well:

$ rm -rf output/build/machine-xml-*
$ rm -rf output/build/hostboot-*
$ ./op-build openpower-pnor-rebuild

Installing the OpenPOWER firmware

Transfer image to BMC

Copy the firmware to the BMC:

$ scp ./output/images/talos.pnor root@$TALOS_BMC_ADDR:/tmp/

Establish BMC sessions

At this point, you should connect two SSH sessions to OpenBMC. In the first session, run the following to display the console during boot:

$ ssh -p 2200 root@$TALOS_BMC_ADDR

The console log will be useful in debugging any issues with the firmware that could occur.

In the second session, get a shell on the BMC via SSH:

$ ssh root@$TALOS_BMC_ADDR
root@talos:~#

Ensure the system is off before proceeding:

root@talos:~# obmcutil state
CurrentBMCState     : xyz.openbmc_project.State.BMC.BMCState.Ready
CurrentPowerState   : xyz.openbmc_project.State.Chassis.PowerState.Off
CurrentHostState    : xyz.openbmc_project.State.Host.HostState.Off

The CurrentHostState must be Off before continuing with the procedure. If the CurrentHostState is not Off, please turn off the machine:

root@talos:~# obmcutil chassisoff

Running the firmware temporarily

You can test the firmware without installing it, though this requires using the v2.00+ BMC firmware which contains an updated mboxd with support for file-backed pnor images.

First, stop mboxd:

root@talos:~# systemctl stop mboxd

Restart mboxd with the additional -b argument:

root@talos:~# mboxd -f 64M -w 1M -b file:/tmp/talos.pnor

You can now test the new firmware image by starting the machine:

root@talos:~# obmcutil poweron

When you have finished testing the image, stop the machine:

root@talos:~# obmcutil poweroff

Note: Ensure the machine is off before proceeding. Verify this by running obmcutil state.

Finally, terminate mboxd and restart the normal mboxd:

root@talos:~# systemctl start mboxd

Flashing the firmware

Warning: Some PNOR firmware updates may require a BMC update to function. Before flashing ensure that your installed BMC firmware is capable of booting the image. It is also possible to downgrade using these instructions in the event of a non-functioning firmware image.

Ensure the system is off.

Perform the update:

root@talos:~# pflash -E -p /tmp/talos.pnor

Start the machine:

root@talos:~# obmcutil poweron

Note: The machine may reboot multiple times when first booted after a firmware update. This is normal; do not interrupt the process.

Troubleshooting the OpenPOWER Firmware

General advice

Always upgrade PNOR and BMC together

Many mismatched PNOR/BMC version combinations lead to weird failures.

Try downgrading the PNOR+BMC firmware

Firmware package 1.04 seems the most reliable at updating the SBE SEEPROM inside the POWER9 chip package.

Always use processor socket 0 for SBE updates

The BMC firmware and/or FSI driver seem to either forget to update the SBE SEEPROM in the second CPU socket, leading to a boot with only CPU 0 active. When you get a brand new chip you need to install it in CPU socket 0 leaving socket 1 empty, wait for the double-reboot to update the SEEPROM, and then you can move that chip to socket 1 if you like.

Try unplugging the HSF fan power during SBE update

Not kidding about this. The BMC is insanely complicated — it's got an entire operating system in there for some reason. It even has systemd. The BMC's systemd often gets into a funky loop restarting hwmon over and over and over, interrupting the SBE SEEPROM reflash every time it does this. Unplugging the PROC0 HSF 4-pin connector gets it to fail hard (due to inability to read the tachometer) and stay failed so the SBE update can proceed. Ugly as this is, it's easier than trying to figure out what systemd thinks it's doing.

SBE_MASTER_VERSION_DOWNLEVEL

If you see the following message reported in the console, then the SBE update process did not work as expected:

16.74709|Error reported by sbe (0x2200) PLID 0x90000008
16.74823|  SBE Image Version Miscompare with Master Target
16.74824|  ModuleId   0x0d SBE_MASTER_VERSION_COMPARE
16.74825|  ReasonCode 0x2215 SBE_MASTER_VERSION_DOWNLEVEL
16.74826|  UserData1  Master Target HUID : 0x0000000000050000
16.74826|  UserData2  Master Target Loop Index : 0x0000000000000000

The machine needs to be reset to finish the update procedure:

root@talos:~# obmcutil chassisoff
root@talos:~# systemctl stop xyz.openbmc_project.State.Host.service
root@talos:~# systemctl start xyz.openbmc_project.State.Host.service
root@talos:~# obmcutil poweron

The update should now complete as expected.

A bug report is open to track this issue.

internal compiler error: Killed

Building the Hostboot source code requires a large amount of RAM. If your machine runs out, you may see an error similar to the following:

powerpc64le-buildroot-linux-gnu-g++.br_real: internal compiler error: Killed (program cc1plus)

To continue you have a few options:

• Reduce the number of parallel jobs being run by appending -j<num> to you build command line

op-build -j4

• Increase the swap space (not recommended)
• Install additional RAM

Bugs

Firmware issues should be reported preferably upstream. If they are specific to Raptor CS products, please report them on their bug tracker.

Building the OpenBMC firmware

Downloading the sources

Raptor CS maintains a public Git repository containing the complete source code for the firmware. To download the source code:

$ git clone -b raptor-v1.07 https://git.raptorcs.com/git/talos-openbmc

Building the firmware

Ensure that all needed support packages are installed. See the README.md for information on needed packages.

The firmware can then be built using the following commands:

$ cd talos-openbmc
$ export TEMPLATECONF=meta-openbmc-machines/meta-openpower/meta-rcs/meta-talos/conf
$ . openbmc-env
$ bitbake obmc-phosphor-image

The resulting firmware image can then be found in the tmp/deploy/images/talos/ directory.

Note: If mboxd fails to build, you may need to patch mboxd.bb.
Note: If building newer versions of the firmware, TEMPLATECONF has changed to TEMPLATECONF=meta-rcs/meta-talos/conf. This should be set before running . open-env. If not, do a git clean and start over with the new TEMPLATECONF.

Installing the firmware

Warning: If you are attempting to upgrade a Talos system from firmware 1.06 or earlier to the 04-16-2019 beta branch, you must follow the directions at Talos_II/Firmware/Public_Beta. Failure to do so may result in a non-booting BMC.

Once firmware has been built, the resulting image-kernel and image-rofs binaries must be copied to /run/initramfs/ on the BMC:

$ scp tmp/deploy/images/talos/image-rofs tmp/deploy/images/talos/image-kernel root@$TALOS_BMC_ADDR:/run/initramfs/

Once the images have been transferred, reboot the BMC. The new firmware files will be detected and automatically applied.

root@talos:~# reboot

The reboot may take some time. Once complete, you will be able to log back in via SSH.

Recovering from failed firmware updates

See Debricking the BMC.