Difference between revisions of "From QEMU to BigEndian FreeBSD"
m |
m |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
This tutorial takes the reader from the bare minimum to launch a virtual machine with QEMU to a full installation of big-endian FreeBSD. The Raptor POWER9 host machine is running little-endian Debian. The short version is for experienced users just needing specifics, while the long version examines each parameter and shows the effect on the virtual machine and installation. Specific details in the short version are linked to explanations in the long version. | This tutorial takes the reader from the bare minimum to launch a virtual machine with QEMU to a full installation of big-endian FreeBSD. The Raptor POWER9 host machine is running little-endian Debian. The short version is for experienced users just needing specifics, while the long version examines each parameter and shows the effect on the virtual machine and installation. Specific details in the short version are linked to explanations in the long version. | ||
| − | Although this documentation is intended to provide information to public users and be freely available, it is released and licensed as required by the GPL v2, per the licensing of the QEMU documentation. Since knowledge was derived from the QEMU documentation, this is therefore derivative works. Please be aware this supercedes the Creative Commons deed under which this | + | Although this documentation is intended to provide information to public users and be freely available, it is released and licensed as required by the GPL v2, per the licensing of the QEMU documentation. Since knowledge was derived from the QEMU documentation, this is therefore derivative works and also licensed GPL v2. Please be aware this supercedes the Creative Commons deed under which this wiki is published. |
=== The short version === | === The short version === | ||
| + | The goal is to rely on the default configuration options, thereby requiring as few custom options as possible. Further configuration options are possible but beyond the scope of this tutorial. | ||
Make sure [[#qemu-system-ppc64]] and [[#qemu-img]] are installed. Use apt install if they are not. | Make sure [[#qemu-system-ppc64]] and [[#qemu-img]] are installed. Use apt install if they are not. | ||
| + | |||
| + | Create a raw disk file to be the installation destination (in this case, a 120GB file named "fbsd.raw"): | ||
| + | |||
| + | <pre> | ||
| + | # qemu-img create fbsd.raw 120G | ||
| + | Formatting 'fbsd.raw', fmt=raw size=128849018880 | ||
| + | # ls -sl | ||
| + | total 924700# ls -sl | ||
| + | 2199132 -rw-r--r-- 1 root root 128849018880 Jun 30 13:19 fbsd.raw | ||
| + | 924696 -rw-r--r-- 1 root root 952018944 Jun 25 05:04 FreeBSD-14.3-RELEASE-powerpc-powerpc64-disc1.iso | ||
| + | </pre> | ||
Download the most recent version of big-endian FreeBSD from | Download the most recent version of big-endian FreeBSD from | ||
| Line 11: | Line 23: | ||
https://download.freebsd.org/releases/powerpc/powerpc64/ISO-IMAGES/ | https://download.freebsd.org/releases/powerpc/powerpc64/ISO-IMAGES/ | ||
| − | Using the example of FreeBSD 14.3 as the iso-based installer, fbsd.raw as the destination [[#storage]] device, four threads per [[#CPU]] core with four cores, 8GB of [[#RAM]], e1000 [[#NIC]], no IPv6, and 2222 as the ssh port into the guest: | + | Using the example of FreeBSD 14.3 as the iso-based installer, fbsd.raw as the destination [[#storage]] device, four threads per [[#CPU]] core with four cores, 8GB of [[#RAM]], e1000 for the [[#NIC]], no IPv6, and 2222 as the ssh port into the guest: |
<pre> | <pre> | ||
| − | # qemu-system-ppc64 --enable-kvm -M pseries,cap-nested-hv=true,cap-ibs=workaround -cpu host -smp cores=4,threads=4,sockets=1 -m 8G -nographic -drive | + | # qemu-system-ppc64 --enable-kvm -M pseries,cap-nested-hv=true,cap-ibs=workaround -cpu host -smp cores=4,threads=4,sockets=1 -m 8G -nographic -drive file=FreeBSD-14.3-RELEASE-powerpc-powerpc64-disc1.iso,media=cdrom,if=virtio,id=drive1 -nic user,ipv6=off,model=e1000,mac=52:54:98:76:54:32,hostfwd=tcp::2222-:22 fbsd.raw |
</pre> | </pre> | ||
| − | Choose MBR for disk partition type. Use DHCP to configure IPv4, and do not configure IPv6. Choose any additional components as needed during the installation process. For this example we named the machine "BEFreeBSD." Add a user and add the user to group wheel (for this we created "user1"). | + | Choose MBR for disk partition type. Use DHCP to configure IPv4, and do not configure IPv6. Choose any additional components as needed during the installation process, including the package management system. For this example we named the machine "BEFreeBSD." Add a user and add the user to group wheel (for this we created "user1"). |
While in the installer, choose "Yes" to enter the shell and modify /etc/fstab to use /dev/vtbd0 instead of /dev/vtbd1 (if necessary): | While in the installer, choose "Yes" to enter the shell and modify /etc/fstab to use /dev/vtbd0 instead of /dev/vtbd1 (if necessary): | ||
| Line 31: | Line 43: | ||
<pre> | <pre> | ||
| − | # qemu-system-ppc64 --enable-kvm -M pseries,cap-nested-hv=true,cap-ibs=workaround -cpu host -smp cores=4,threads=4,sockets=1 -m 8G -nographic | + | # qemu-system-ppc64 --enable-kvm -M pseries,cap-nested-hv=true,cap-ibs=workaround -cpu host -smp cores=4,threads=4,sockets=1 -m 8G -nographic -nic user,ipv6=off,model=e1000,mac=52:54:98:76:54:32,hostfwd=tcp::2222-:22 fbsd.raw |
</pre> | </pre> | ||
| Line 189: | Line 201: | ||
| − | With CDROM | + | With CDROM: |
<pre> | <pre> | ||
Ready! | Ready! | ||
| − | 0 > dev / | + | 0 > dev / ls |
| − | |||
7e644530 : / | 7e644530 : / | ||
7e644b40 : |-- ibm,persistent-memory | 7e644b40 : |-- ibm,persistent-memory | ||
| Line 209: | Line 220: | ||
7e648c58 : | +-- PowerPC,POWER9@c | 7e648c58 : | +-- PowerPC,POWER9@c | ||
7e649738 : |-- pci@800000020000000 | 7e649738 : |-- pci@800000020000000 | ||
| − | 7e64bd18 : | |-- | + | 7e64bd18 : | |-- ethernet@0 |
| − | + | 7e64c280 : | |-- vga@1 | |
| − | + | 7e64c798 : | |-- usb-xhci@2 | |
| − | + | 7e691d68 : | | |-- usb-keyboard@1 | |
| − | + | 7e692828 : | | +-- usb-mouse@2 | |
| − | + | 7e64cdc8 : | +-- scsi@3 <--- added device | |
| − | + | 7e64d428 : |-- vdevice | |
| − | + | 7e64d6a8 : | |-- vty@71000000 | |
| − | + | 7e64d8c8 : | |-- nvram@71000001 | |
| − | + | 7e64db38 : | +-- v-scsi@71000002 | |
| − | + | 7e673d00 : | +-- disk | |
| − | + | 7e64dff8 : |-- memory@0 | |
| − | + | 7e64e1f0 : |-- interrupt-controller | |
| − | + | 7e651330 : |-- aliases | |
| − | + | 7e651438 : |-- options | |
| − | + | 7e6514f8 : |-- openprom | |
| − | + | 7e651600 : |-- packages | |
| − | + | 7e6516c8 : | |-- deblocker | |
| − | + | 7e652168 : | |-- disk-label | |
| − | + | 7e656808 : | |-- fat-files | |
| − | + | 7e658f80 : | |-- rom-files | |
| − | + | 7e6599a0 : | |-- ext2-files | |
| − | + | 7e65c418 : | |-- obp-tftp | |
| − | + | 7e65c7e8 : | +-- iso-9660 | |
| − | + | 7e661230 : +-- mmu ok | |
| − | |||
</pre> | </pre> | ||
| − | |||
==== NIC ==== | ==== NIC ==== | ||
| Line 307: | Line 316: | ||
</pre> | </pre> | ||
'''Solution:''' | '''Solution:''' | ||
| − | There is a mismatch in the device tree configuration and the boot configuration. The following will get FreeBSD up and running | + | There is a mismatch in the device tree configuration and the boot configuration. This can occur when the installation is done with a -device but the reboot is done with a file. FreeBSD and QEMU don't always agree on the order of the devices, and FreeBSD is looking for the hard drive where it was, not where it is. The following will get FreeBSD up and running from a file: |
<pre> | <pre> | ||
| Line 313: | Line 322: | ||
List of GEOM managed disk devices: | List of GEOM managed disk devices: | ||
| − | + | ufsid/6860fe636e2c3598 da0s2b da0s2a da0s2 da0s1 da0 cd0 | |
| − | mountroot> ufs:/dev/ | + | mountroot> ufs:/dev/da0s2 |
</pre> | </pre> | ||
| − | and then edit /etc/fstab: | + | to specify that da0s2 is a ufs file system, and then after booting edit /etc/fstab: |
<pre> | <pre> | ||
# cat /etc/fstab | # cat /etc/fstab | ||
# Device Mountpoint FStype Options Dump Pass# | # Device Mountpoint FStype Options Dump Pass# | ||
| − | /dev/ | + | /dev/da0s2a / ufs rw 1 1 |
| − | /dev/ | + | /dev/da0s2b none swap sw 0 0 |
</pre> | </pre> | ||
Latest revision as of 15:16, 30 June 2025
This tutorial takes the reader from the bare minimum to launch a virtual machine with QEMU to a full installation of big-endian FreeBSD. The Raptor POWER9 host machine is running little-endian Debian. The short version is for experienced users just needing specifics, while the long version examines each parameter and shows the effect on the virtual machine and installation. Specific details in the short version are linked to explanations in the long version.
Although this documentation is intended to provide information to public users and be freely available, it is released and licensed as required by the GPL v2, per the licensing of the QEMU documentation. Since knowledge was derived from the QEMU documentation, this is therefore derivative works and also licensed GPL v2. Please be aware this supercedes the Creative Commons deed under which this wiki is published.
The short version
The goal is to rely on the default configuration options, thereby requiring as few custom options as possible. Further configuration options are possible but beyond the scope of this tutorial.
Make sure #qemu-system-ppc64 and #qemu-img are installed. Use apt install if they are not.
Create a raw disk file to be the installation destination (in this case, a 120GB file named "fbsd.raw"):
# qemu-img create fbsd.raw 120G Formatting 'fbsd.raw', fmt=raw size=128849018880 # ls -sl total 924700# ls -sl 2199132 -rw-r--r-- 1 root root 128849018880 Jun 30 13:19 fbsd.raw 924696 -rw-r--r-- 1 root root 952018944 Jun 25 05:04 FreeBSD-14.3-RELEASE-powerpc-powerpc64-disc1.iso
Download the most recent version of big-endian FreeBSD from
https://download.freebsd.org/releases/powerpc/powerpc64/ISO-IMAGES/
Using the example of FreeBSD 14.3 as the iso-based installer, fbsd.raw as the destination #storage device, four threads per #CPU core with four cores, 8GB of #RAM, e1000 for the #NIC, no IPv6, and 2222 as the ssh port into the guest:
# qemu-system-ppc64 --enable-kvm -M pseries,cap-nested-hv=true,cap-ibs=workaround -cpu host -smp cores=4,threads=4,sockets=1 -m 8G -nographic -drive file=FreeBSD-14.3-RELEASE-powerpc-powerpc64-disc1.iso,media=cdrom,if=virtio,id=drive1 -nic user,ipv6=off,model=e1000,mac=52:54:98:76:54:32,hostfwd=tcp::2222-:22 fbsd.raw
Choose MBR for disk partition type. Use DHCP to configure IPv4, and do not configure IPv6. Choose any additional components as needed during the installation process, including the package management system. For this example we named the machine "BEFreeBSD." Add a user and add the user to group wheel (for this we created "user1").
While in the installer, choose "Yes" to enter the shell and modify /etc/fstab to use /dev/vtbd0 instead of /dev/vtbd1 (if necessary):
# cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/vtbd0s2a / ufs rw 1 1 /dev/vtbd0s2b none swap sw 0 0
Either remove remote login by root at this time or do so after rebooting. Shutdown the VM and relaunch it with
# qemu-system-ppc64 --enable-kvm -M pseries,cap-nested-hv=true,cap-ibs=workaround -cpu host -smp cores=4,threads=4,sockets=1 -m 8G -nographic -nic user,ipv6=off,model=e1000,mac=52:54:98:76:54:32,hostfwd=tcp::2222-:22 fbsd.raw
to remove the CD installer. Log in with the user ("user1") that was added to the wheel group. Check the configuration for any unexpected values.
% dmesg | grep CPU
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
xicp0: Handling CPUs 0-15
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
% dmesg | grep memory
real memory = 8544366592 (8148 MB)
avail memory = 8216293376 (7835 MB)
% uname -a
FreeBSD BEFreeBSD 14.3-RELEASE FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC powerpc
% ifconfig -a
em0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=48525bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether 52:54:98:76:54:32
inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Log in locally with the created user ("user1") to check SSH:
# ssh -p 2222 user1@127.0.0.1 (user1@127.0.0.1) Password for user1@BEFreeBSD: Last login: Sun Jun 29 09:02:48 2025 from xx FreeBSD 14.3-RELEASE (GENERIC) releng/14.3-n271432-8c9ce319fef7 Welcome to FreeBSD! ...
Check that su works:
% su root Password: #
Log in remotely to make sure external ssh access is available. Configure automatic restart of VM with reboot of host machine.
The long version
This tutorial draws from the following web sites:
https://www.qemu.org/docs/master/system/introduction.html
https://www.qemu.org/docs/master/system/invocation.html
https://www.qemu.org/docs/master/tools/qemu-img.html
https://www.qemu.org/docs/master/system/ppc/powernv.html
qemu-system-ppc64
qemu-system-ppc64 is the QEMU executable used to provide a virtual machine on Raptor's hosts. To determine if you are running on a physical or virtual host, execute
# facter | grep virtual is_virtual => false virtual => physical
This is real host. A virtual one would look like
# facter | grep virtual is_virtual => true virtual => kvm
QEMU needs several parameters to describe the
CPU
RAM
No devices added (default SLOF boot):
0 > dev / ok 0 > ls 7e644530 : / 7e644b40 : |-- ibm,persistent-memory 7e644da0 : |-- hypervisor 7e644f00 : |-- chosen 7e645310 : |-- rtas 7e646638 : |-- event-sources 7e6467f0 : | |-- hot-plug-events 7e646948 : | +-- epow-events 7e646ae8 : |-- cpus 7e646db0 : | |-- PowerPC,POWER9@0 7e6477e8 : | |-- PowerPC,POWER9@4 7e648220 : | |-- PowerPC,POWER9@8 7e648c58 : | +-- PowerPC,POWER9@c 7e649738 : |-- pci@800000020000000 7e64bd18 : | |-- vga@0 7e64c230 : | +-- usb-xhci@1 7e68f7f8 : | |-- usb-keyboard@1 7e6902b8 : | +-- usb-mouse@2 7e64c908 : |-- vdevice 7e64cb88 : | |-- vty@71000000 7e64cda8 : | |-- nvram@71000001 7e64d018 : | |-- l-lan@71000002 7e64d3e8 : | +-- v-scsi@71000003 7e673ba8 : | +-- disk 7e64d8a8 : |-- memory@0 7e64daa0 : |-- interrupt-controller 7e650be0 : |-- aliases 7e650ce8 : |-- options 7e650da8 : |-- openprom 7e650eb0 : |-- packages 7e650f78 : | |-- deblocker 7e651a18 : | |-- disk-label 7e6560b8 : | |-- fat-files 7e658830 : | |-- rom-files 7e659250 : | |-- ext2-files 7e65bcc8 : | |-- obp-tftp 7e65c098 : | +-- iso-9660 7e660ae0 : +-- mmu ok 0 >
qemu-img
Create a raw disk file to be the installation destination (in this case, 120GB):
# qemu-img create fbsd.raw 120G
Formatting 'fbsd.raw', fmt=raw size=128849018880
# ls -sl
total 924700
4 -rw-r--r-- 1 root root 128849018880 Jun 28 13:31 fbsd.raw
924696 -rw-r--r-- 1 root root 952018944 Jun 25 05:04 FreeBSD-14.3-RELEASE-powerpc-powerpc64-disc1.iso
Storage
With CDROM:
Ready! 0 > dev / ls 7e644530 : / 7e644b40 : |-- ibm,persistent-memory 7e644da0 : |-- hypervisor 7e644f00 : |-- chosen 7e645310 : |-- rtas 7e646638 : |-- event-sources 7e6467f0 : | |-- hot-plug-events 7e646948 : | +-- epow-events 7e646ae8 : |-- cpus 7e646db0 : | |-- PowerPC,POWER9@0 7e6477e8 : | |-- PowerPC,POWER9@4 7e648220 : | |-- PowerPC,POWER9@8 7e648c58 : | +-- PowerPC,POWER9@c 7e649738 : |-- pci@800000020000000 7e64bd18 : | |-- ethernet@0 7e64c280 : | |-- vga@1 7e64c798 : | |-- usb-xhci@2 7e691d68 : | | |-- usb-keyboard@1 7e692828 : | | +-- usb-mouse@2 7e64cdc8 : | +-- scsi@3 <--- added device 7e64d428 : |-- vdevice 7e64d6a8 : | |-- vty@71000000 7e64d8c8 : | |-- nvram@71000001 7e64db38 : | +-- v-scsi@71000002 7e673d00 : | +-- disk 7e64dff8 : |-- memory@0 7e64e1f0 : |-- interrupt-controller 7e651330 : |-- aliases 7e651438 : |-- options 7e6514f8 : |-- openprom 7e651600 : |-- packages 7e6516c8 : | |-- deblocker 7e652168 : | |-- disk-label 7e656808 : | |-- fat-files 7e658f80 : | |-- rom-files 7e6599a0 : | |-- ext2-files 7e65c418 : | |-- obp-tftp 7e65c7e8 : | +-- iso-9660 7e661230 : +-- mmu ok
NIC
With the network card:
0 > dev / ok 0 > ls 7e644530 : / 7e644b40 : |-- ibm,persistent-memory 7e644da0 : |-- hypervisor 7e644f00 : |-- chosen 7e645310 : |-- rtas 7e646638 : |-- event-sources 7e6467f0 : | |-- hot-plug-events 7e646948 : | +-- epow-events 7e646ae8 : |-- cpus 7e646db0 : | |-- PowerPC,POWER9@0 7e6477e8 : | |-- PowerPC,POWER9@4 7e648220 : | |-- PowerPC,POWER9@8 7e648c58 : | +-- PowerPC,POWER9@c 7e649738 : |-- pci@800000020000000 7e64bd18 : | |-- ethernet@0 <-- Added device 7e64c280 : | |-- vga@1 7e64c798 : | |-- usb-xhci@2 7e6933f8 : | | |-- usb-keyboard@1 7e693eb8 : | | +-- usb-mouse@2 7e64cdc8 : | |-- scsi@3 7e64d380 : | +-- scsi@4 7e64d9e0 : |-- vdevice 7e64dc60 : | |-- vty@71000000 7e64de80 : | |-- nvram@71000001 7e64e0f0 : | +-- v-scsi@71000002 7e6742b8 : | +-- disk 7e64e5b0 : |-- memory@0 7e64e7a8 : |-- interrupt-controller 7e6518e8 : |-- aliases 7e6519f0 : |-- options 7e651ab0 : |-- openprom 7e651bb8 : |-- packages 7e651c80 : | |-- deblocker 7e652720 : | |-- disk-label 7e656dc0 : | |-- fat-files 7e659538 : | |-- rom-files 7e659f58 : | |-- ext2-files 7e65c9d0 : | |-- obp-tftp 7e65cda0 : | +-- iso-9660 7e6617e8 : +-- mmu ok
% uname -a FreeBSD BEFreeBSD 14.3-RELEASE FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC powerpc
Post-boot blues
Mountroot fails
mountroot: waiting for device /dev/vtbd1s2a... Mounting from ufs:/dev/vtbd1s2a failed with error 19. Loader variables: vfs.root.mountfrom=ufs:/dev/vtbd1s2a vfs.root.mountfrom.options=rw
Solution: There is a mismatch in the device tree configuration and the boot configuration. This can occur when the installation is done with a -device but the reboot is done with a file. FreeBSD and QEMU don't always agree on the order of the devices, and FreeBSD is looking for the hard drive where it was, not where it is. The following will get FreeBSD up and running from a file:
mountroot> ? List of GEOM managed disk devices: ufsid/6860fe636e2c3598 da0s2b da0s2a da0s2 da0s1 da0 cd0 mountroot> ufs:/dev/da0s2
to specify that da0s2 is a ufs file system, and then after booting edit /etc/fstab:
# cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/da0s2a / ufs rw 1 1 /dev/da0s2b none swap sw 0 0
Some other errors
QEMU's output is somewhere else.
At the end of the output from QEMU, further output is suppressed and the last line is
char device redirected to /dev/pts/N (label monitor-base)
Solution: Output has now been redirected to /dev/pts/N where N is a number correlated to the pseudo-tty in /dev/pts/. You can attach to it with
# screen /dev/pts/N 115200 8 N 1
or add -nographic to the command line.
Duplicate/conflicting serial devices
qemu-system-ppc64: Duplicate ID 'serial0' for chardev qemu-system-ppc64: could not connect serial device to character backend 'vc:80Cx24C'
Solution: Most likely you are attempting to add a chardev with the id=serial0 parameter. vc is the default i/o and is named "serial0." Use -nodefaults and provide your own chardev (on the command line) or use the default serial0.
No host
qemu-system-ppc64: unable to find CPU model 'host'
Solution: The command line is missing --enable-kvm.
No discs
Trying to load: from: disk ... E3405: No such device Trying to load: from: /vdevice/v-scsi@71000003/disk@8200000000000000 ... No medium !
Solution: No virtio has been specified in the command line. Add one.
Can not use IDE on pSeries VMs
qemu-system-ppc64: -drive file=FreeBSD-14.3-RELEASE-powerpc-powerpc64-disc1.iso,if=ide,index=1,media=cdrom: machine type does not support if=ide,bus=0,unit=1 qemu-system-ppc64: -drive file=fbsd.raw,if=ide,index=0,format=raw: machine type does not support if=ide,bus=0,unit=0
Solution: Add a virtio.
