Endianness
All POWER CPU's from POWER3 onward support both big-endian (BE) and little-endian (LE) modes, though for practical reasons (including firmware support), most ppc64le Linux distros require POWER8 or higher, while ppc64 Linux distros frequently support older POWER CPU's.[1] This is in contrast to x86, which supports only little-endian mode.
Contents
Operating System Support
Some distros only support BE; others only support LE; others support both. See the Operating System Compatibility List to check support in your preferred distro, or to find a distro that supports your preferred endianness.
Security
BE mode has a security advantage over LE mode: there exist certain classes of memory-safety bugs (e.g. accessing a variable with the wrong data size) that typically cause an immediate crash (or otherwise obviously-wrong behavior) in BE mode, but instead manifest as silent security vulnerabilities in LE mode. Theoretically, it is likely that a sanitizer could achieve similar security benefits in LE mode, but such a sanitizer would have a performance impact that usage of BE mode would not incur. (Also, no such sanitizer is currently known to exist.)
Performance
Networking-heavy code may be slightly faster in BE mode, because many network protocols serialize data as BE.
LE mode may be slightly faster than BE mode for code that can be optimized by deliberately accessing variables with a smaller-than-usual data size. Note that this class of optimization is a memory-safety bug if implemented incorrectly; see the Security section for information on downsides of this performance advantage.
Debugging
Since human cultures tend to use BE when dealing with numbers, memory dumps of BE mode tend to be more intuitively readable by humans than memory dumps of LE mode. This can make debugging easier in BE mode.
Porting Buggy Software
With rare exceptions (e.g. library thunking), correctly written software will work in both BE and LE modes without issues. However, if software was only subject to QA testing on one endianness mode, there is a nontrivial chance that it was written to rely on assumptions that only are accurate for that mode. In practice, this means that software that was exclusively tested on x86 is likely to exhibit bugs in BE mode, while software that was exclusively tested on Linux on POWER7 and earlier, AIX, and old Macs is likely to exhibit bugs in LE mode.
The Hardware Compatibility List may be helpful in checking which endianness modes your preferred hardware's drivers support, or finding hardware whose drivers support your preferred endianness mode. For checking other software support, the Ports List may be helpful.
Library Thunking
Some emulators (e.g. Box86) utilize library thunking to allow a foreign-architecture executable to link to native-architecture shared libraries, enabling a large speed-up compared to emulating the libraries. While it is theoretically possible to do this when the emulated and native architecture are different endianness, doing so is impractical because 100% of data structures passed between the executable and the libraries will need to be manually endian-swapped. In contrast, it is relatively easy when the emulated and native architecture are the same endianness, since only a small fraction of data structures will need manual translation. In practice, this means that if you are emulating x86 software, you probably will want to run in LE mode.
ABI
Endianness is independent of ELFv1 vs ELFv2. However, there exist some OS distros that, on 64-bit POWER, only support ELFv1 for BE and ELFv2 for LE.
Virtual Machines
Virtual machine guests (in KVM-HV) can run in either endianness, regardless of the host OS's endianness. This can be a good approach if you need to run a specific software package that doesn't support your host OS's endianness.
