Difference between revisions of "Compiling Firmware"
(→Grabbing the sources: Convert to wiki syntax and templatize the version) |
(→BMC Recovery procedure via U-Boot: Now that dragon_pilot has successfully done a recovery, add the missing steps that were figured out today.) |
||
| Line 160: | Line 160: | ||
In the event of a failure updating the BMC, but with a functioning u-boot, you can still recover by using u-boot to manually bootstrap the BMC over the network or serial line. | In the event of a failure updating the BMC, but with a functioning u-boot, you can still recover by using u-boot to manually bootstrap the BMC over the network or serial line. | ||
| − | * Prepare a TFTP server, and place image-bmc in the root. (TODO: elaborate on how to set this up) | + | * Prepare a TFTP server, and place image-bmc, image-rofs, and image-kernel in the root. (TODO: elaborate on how to set this up) |
* Connect a serial console to the [[Talos_II/Building_FAQ#BMC_serial_port_J7701|BMC serial port]] (J7701, serial port bracket required) and set to 115200 8n1, disable RTS/CTS (hardware flow control). | * Connect a serial console to the [[Talos_II/Building_FAQ#BMC_serial_port_J7701|BMC serial port]] (J7701, serial port bracket required) and set to 115200 8n1, disable RTS/CTS (hardware flow control). | ||
| Line 166: | Line 166: | ||
* Run <code>dhcp x.x.x.x:image-bmc</code>, replacing the IP address of your TFTP server. This will load a copy of the stock boot image into RAM. | * Run <code>dhcp x.x.x.x:image-bmc</code>, replacing the IP address of your TFTP server. This will load a copy of the stock boot image into RAM. | ||
* Run <code>bootm 83080000</code>. This will prepare and boot off of the loaded virtual image. | * Run <code>bootm 83080000</code>. This will prepare and boot off of the loaded virtual image. | ||
| − | * | + | * If your rofs partition is not functional, you will be dropped into the systemd emergency shell at this point. Try both the password you set as well as the default 0penBmc, it may be one or the other depending on the state of the rwfs partition. If it boots up properly, the problem is probably in your kernel partition and you can retry flashing your image-kernel using the normal procedure. (The rest of these instructions are for the systemd emergency shell.) |
| − | + | * mount -t tmpfs none /tmp | |
| − | * (TODO: | + | * run "udhcpc" to get an IP address. (TODO: verify that this is the actual command that you run. Do you have to specify the network interface too?) |
| − | * (TODO: | + | * cd /tmp |
| + | * tftp -g -r image-rofs x.x.x.x | ||
| + | * tftp -g -r image-kernel x.x.x.x | ||
| + | * IMPORTANT: Use md5sum, sha1sum, or sha256sum to verify successful transfer of image-rofs and image-kernel! tftp is a very barebones protocol and relies on transport layer checksumming, which is optional and not always available in UDP! | ||
| + | * Verify that the output of 'cat /sys/class/mtd/mtd3/name' is "kernel" and the output of 'cat /sys/class/mtd/mtd4/name' is "rofs". We will be flashing mtd devices directly in the next step and this is the last chance to verify that they will be flashed to the correct partition. | ||
| + | * flashcp -v image-kernel /dev/mtd3 | ||
| + | * flashcp -v image-rofs /dev/mtd4 | ||
| + | * (TODO: Describe how to reset rwfs in case it was damaged as well?) | ||
| + | * After the flash is complete, you can run restart the BMC and it should boot successfully. | ||
* (TODO: Discussion of using Kermit to upload the image without network access) | * (TODO: Discussion of using Kermit to upload the image without network access) | ||
| + | * (TODO: Load recovery images over USB?) | ||
* (TODO: Discussion of u-boot memory map) | * (TODO: Discussion of u-boot memory map) | ||
| − | |||
=== Troubleshooting === | === Troubleshooting === | ||
TODO | TODO | ||
Revision as of 23:32, 11 June 2018
The following steps can be used to compile and update the firmware on Talos™ II-based solutions. It's maintained by both Raptor CS and community members.
Requirements
- At least 25GB of free hard drive space
- 16GB of free RAM
Operating System
The build system (op-build) has been primarily tested using Debian stretch. If you are on a different operating system such as Fedora 28, a Debian chroot should be used:
sudo yum install debootstrap dpkg sudo debootstrap stretch debian-chroot http://httpredir.debian.org/debian sudo mount -t proc none debian-chroot/proc/ sudo mount -o bind /sys/ debian-chroot/sys/ sudo mount -o bind /dev/shm/ debian-chroot/dev/shm/
Enter the chroot and install the needed packages:
sudo chroot debian-chroot/
apt-get install software-properties-common locales
# Packages needed for PNOR builds
apt-get install cscope ctags libz-dev libexpat-dev \
python texinfo \
build-essential g++ git bison flex unzip \
libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc \
wget bc
# Packages needed for OpenBMC builds
apt-get install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat
Create a chroot user:
useradd -m build-user -s /bin/bash su build-user cd
You can now use the chroot to build the firmware.
To enter the chroot in the future, you can run the following from a regular terminal:
sudo chroot debian-chroot/ su build-user cd
Building the PNOR Firmware
Grabbing the sources
Raptor CS maintains a public git repository containing the complete source code for the firmware. To download the source code:
git clone --recursive https://scm.raptorcs.com/scm/git/talos-op-build
Building the firmware
Before building the firmware, all needed support packages must be installed. Please see the README.md file for directions on installing the needed packages.
Once the packages are installed, the firmware can be build using the following commands:
cd talos-op-build . op-build-env op-build talos_defconfig op-build
To rebuild an individual package (such as hostboot) and recreate the pnor image, the following can be run:
op-build hostboot-rebuild openpower-pnor-rebuild
Updating the firmware
Copy the firmware to the BMC
scp ./output/images/talos.pnor root@<talos-openbmc>:/tmp/
At this point, you should connect two SSH sessions to OpenBMC.
In the first session, run the following to display the console during bootup:
ssh -p 2200 root@<talos-openbmc>
The console log will be useful in debugging any issues with the firmware that could occur.
In the second BMC session, ensure the system is off by running obmcutil. You should see the following:
ssh root@<talos-openbmc> root@talos:~# obmcutil state CurrentBMCState : xyz.openbmc_project.State.BMC.BMCState.Ready CurrentPowerState : xyz.openbmc_project.State.Chassis.PowerState.Off CurrentHostState : xyz.openbmc_project.State.Host.HostState.Off
The CurrentHostState must be Off before continuing with the procedure. If the CurrentHostState is not Off, please turn off the machine:
obmcutil chassisoff
Once off, perform the update:
pflash -E -p /tmp/talos.pnor
Start the machine:
obmcutil poweron
Note: the machine may reboot multiple times after the initial flash.
Troubleshooting
SBE_MASTER_VERSION_DOWNLEVEL
If you see the following message reported in the console, then the SBE update process did not work as expected:
16.74709|Error reported by sbe (0x2200) PLID 0x90000008 16.74823| SBE Image Version Miscompare with Master Target 16.74824| ModuleId 0x0d SBE_MASTER_VERSION_COMPARE 16.74825| ReasonCode 0x2215 SBE_MASTER_VERSION_DOWNLEVEL 16.74826| UserData1 Master Target HUID : 0x0000000000050000 16.74826| UserData2 Master Target Loop Index : 0x0000000000000000
The machine needs to be reset to finish the update proceedure using the following:
obmcutil chassisoff systemctl stop xyz.openbmc_project.State.Host.service systemctl start xyz.openbmc_project.State.Host.service obmcutil poweron
The update should now complete as expected.
A bug report is open[1] to track this issue.
internal compiler error: Killed
Building the hostboot source code requires a large amount of ram. If your machine runs out, you may see an error similar ot the following:
powerpc64le-buildroot-linux-gnu-g++.br_real: internal compiler error: Killed (program cc1plus)
To continue you have a few options:
- Reduce the number of parallel jobs being run by appending -j<num> to you build command line
op-build -j4
- Increase the swap space
- Install additional RAM
Building the OpenBMC firmware
Grabbing the sources
Raptor CS maintains a public git repository containing the complete source code for the firmware. To download the source code and check out the tag:
git clone https://git.raptorcs.com/git/talos-openbmc cd talos-openbmc git checkout raptor-v1.07
Building the firmware
Before building the firmware, all needed support packages must be installed. Please see the README.md file for directions on installing the needed packages.
Once the packages are installed, the firmware can be build using the following commands:
cd talos-openbmc export TEMPLATECONF=meta-openbmc-machines/meta-openpower/meta-rcs/meta-talos/conf . openbmc-env bitbake obmc-phosphor-image
The resulting firmware can be found in the tmp/deploy/images/talos/ directory.
Updating the firmware
Once firmware has been built, the resulting kernel and rofs binaries need to be copied over to the /run/initramfs/
scp tmp/deploy/images/talos/image-rofs tmp/deploy/images/talos/image-kernel root@<talos-openbmc>:/run/initramfs/
Once the images have been transferred, reboot the BMC:
root@<talos-openbmc> reboot
OpenBMC may take a while to reboot. Once complete, you will be able to log back in via ssh.
BMC Recovery procedure via U-Boot
WIP -- these instructions are still being tested, use at your own risk!
In the event of a failure updating the BMC, but with a functioning u-boot, you can still recover by using u-boot to manually bootstrap the BMC over the network or serial line.
- Prepare a TFTP server, and place image-bmc, image-rofs, and image-kernel in the root. (TODO: elaborate on how to set this up)
- Connect a serial console to the BMC serial port (J7701, serial port bracket required) and set to 115200 8n1, disable RTS/CTS (hardware flow control).
- Disconnect and reconnect power to the machine to force a BMC restart. Press a key to interrupt auto-boot when prompted.
- Run
dhcp x.x.x.x:image-bmc, replacing the IP address of your TFTP server. This will load a copy of the stock boot image into RAM. - Run
bootm 83080000. This will prepare and boot off of the loaded virtual image. - If your rofs partition is not functional, you will be dropped into the systemd emergency shell at this point. Try both the password you set as well as the default 0penBmc, it may be one or the other depending on the state of the rwfs partition. If it boots up properly, the problem is probably in your kernel partition and you can retry flashing your image-kernel using the normal procedure. (The rest of these instructions are for the systemd emergency shell.)
- mount -t tmpfs none /tmp
- run "udhcpc" to get an IP address. (TODO: verify that this is the actual command that you run. Do you have to specify the network interface too?)
- cd /tmp
- tftp -g -r image-rofs x.x.x.x
- tftp -g -r image-kernel x.x.x.x
- IMPORTANT: Use md5sum, sha1sum, or sha256sum to verify successful transfer of image-rofs and image-kernel! tftp is a very barebones protocol and relies on transport layer checksumming, which is optional and not always available in UDP!
- Verify that the output of 'cat /sys/class/mtd/mtd3/name' is "kernel" and the output of 'cat /sys/class/mtd/mtd4/name' is "rofs". We will be flashing mtd devices directly in the next step and this is the last chance to verify that they will be flashed to the correct partition.
- flashcp -v image-kernel /dev/mtd3
- flashcp -v image-rofs /dev/mtd4
- (TODO: Describe how to reset rwfs in case it was damaged as well?)
- After the flash is complete, you can run restart the BMC and it should boot successfully.
- (TODO: Discussion of using Kermit to upload the image without network access)
- (TODO: Load recovery images over USB?)
- (TODO: Discussion of u-boot memory map)
Troubleshooting
TODO
