Difference between revisions of "POWER9"

From RCS Wiki
Jump to navigation Jump to search
(add sections, expand module writeup)
(→‎Steppings: KVM-PR probably works fine on DD2.1)
 
(43 intermediate revisions by 10 users not shown)
Line 17: Line 17:
 
|data8 = January 2018
 
|data8 = January 2018
 
|label9 = Production stepping(s)
 
|label9 = Production stepping(s)
−
|data9 = DD2.2
+
|data9 = (Nimbus) DD2.2, DD2.3
 
|label10 = [[POWER8E|← POWER8E]]
 
|label10 = [[POWER8E|← POWER8E]]
−
|data10 = POWER10 →  
+
|data10 = [[POWER10|POWER10 →]]
 
}}
 
}}
  
−
POWER9 is IBM's most recent POWER compatible server and workstation CPU ([[PowerISA|POWER ISA]] v3.0B).  Built on a 14nm process, each CPU package can contain up to 24 [[SMT4|SMT4]] cores or 12 [[SMT8|SMT8]] cores.  Each pair of [[SMT4|SMT4]] cores, or singleton [[SMT8|SMT8]] core, comprises a slice; each slice in turn contains 512kB L2 cache and 10MB L3 cache. Raptor Computing Systems' 4- and 8-core processors provide unpaired cores, such that one [[SMT4|SMT4]] core per slice is fused off.  This allows each of the [[SMT4|SMT4]] cores to utilize the full cache of the slice exclusively, increasing performance for these [[Single_Thread|ST]]-focused processors.
+
POWER9 is IBM's most recent POWER compatible server and workstation CPU ([[Power ISA|POWER ISA]] v3.0B).  Built on a 14nm process, each CPU package can contain up to 24 [[SMT4|SMT4]] cores or 12 [[SMT8|SMT8]] cores.  Each pair of [[SMT4|SMT4]] cores, or singleton [[SMT8|SMT8]] core, comprises a slice (some IBM documentation appears to call slices "chiplets"); each slice in turn contains 512kB L2 cache and 10MB L3 cache.
 +
 
 +
==Pairing==
 +
 
 +
Most POWER9 processors share each slice's L2 and L3 cache between cores within the slice.  However, [[Sforza#Configurations|some Sforza processors]] provide unpaired cores, such that one [[SMT4|SMT4]] core per slice is fused off.  This allows each of the [[SMT4|SMT4]] cores to utilize the full cache of the slice exclusively, increasing performance for these [[Single_Thread|ST]]-focused processors. There are also some [[Speculative_Execution_Vulnerabilities_of_2018#Attack_surface_reduction|security implications]] to whether cores are paired or unpaired.
 +
 
 +
==Process==
 +
POWER9 is fabricated using the GlobalFoundries 14HP (High Performance) process. This is distinct from the GlobalFoundries 14LPP (Low Power) process used by other GF 14nm customers, and is believed to be an IBM-specific process using ex-IBM Microelectronics intellectual property. The process is also used for the CPUs in IBM's z14 mainframes.<ref>Schor, David. [https://fuse.wikichip.org/news/956/globalfoundries-14hp-process-a-marriage-of-two-technologies/ GlobalFoundries 14HP process, a marriage of two technologies]. Wikichip Fuse.</ref>
  
 
== Chips ==
 
== Chips ==
  
−
{| class="wikitable"
+
There are three known silicon masks of POWER9:
−
|+ POWER9 Chips
+
* Nimbus (POWER9 [[Scale Out]])
−
!  !! PowerNV !! PowerVM
+
* Cumulus (POWER9 [[Scale Up]])
−
|-
+
* Axone (POWER9&prime; ("POWER9 Prime"), aka POWER9 with Advanced I/O)
−
! Scale Out
+
 
−
| Nimbus || ?<ref group="note">The presentation by Jeff Stuecheli makes it clear that these chips will exist, but the codename for them is currently unknown.</ref>
+
Nimbus is the &#8220;[[Scale Out|scale out]]&#8221; variant and uses direct-attach DDR4 memory. Cumulus is the &#8220;[[Scale Up|scale up]]&#8221; version and uses [[Centaur]] memory buffers, allowing larger amounts of memory to be attached to a system.
−
|-
+
 
−
! Scale Up
+
Chips can be fused as [[SMT4]] or [[SMT8]] during manufacturing. The [[SMT8]] variant essentially fuses each pair of cores into one &#8220;core&#8221;, halving the core count while doubling the number of threads per core. [[SMT4]] variants are intended for [[PowerNV]] platforms running Linux, and [[SMT8]] variants are intended for use with IBM's PowerVM hypervisor which can run Linux, AIX or IBM i.<ref>Stuecheli, Jeff. POWER9. Presentation for [https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/AIX+Virtual+User+Group+-+USA AIX VUG]. ([https://public.dhe.ibm.com/systems/power/community/aix/Central-VUG-Replays/2017-01-26_IBM_POWER9.wmv video download], [https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/61ad9cf2-c6a3-4d2c-b779-61ff0266d32a/page/1cb956e8-4160-4bea-a956-e51490c2b920/attachment/56cea2a9-a574-4fbb-8b2c-675432367250/media/POWER9-VUG.pdf slides], [[User:Torpcoms/Timemark/POWER9|timemarks]])</ref>
−
| || Cumulus
+
 
−
|}
+
=== Steppings ===
−
<references group="note"/>
 
  
−
[[PowerNV|PowerNV]] systems use [[SMT4|SMT4]] packages exclusively, and these PowerNV SMT4 chips are called Nimbus.
+
Several revisions of the Nimbus mask have been issued:
  
−
PowerVM systems, in contrast, use SMT8 chips, and are intended to run Linux, AIX, or IBM i under IBM's PowerVM hypervisor. SMT8 chips are planned to be made in both [[Scale Out|Scale Out]] (direct-attach RAM) and [[Scale Up|Scale Up]] (centaur-buffered RAM) configurations.<ref>Stuecheli, Jeff. POWER9. Presentation for [https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/AIX+Virtual+User+Group+-+USA AIX VUG]. ([https://public.dhe.ibm.com/systems/power/community/aix/Central-VUG-Replays/2017-01-26_IBM_POWER9.wmv video download], [https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/61ad9cf2-c6a3-4d2c-b779-61ff0266d32a/page/1cb956e8-4160-4bea-a956-e51490c2b920/attachment/56cea2a9-a574-4fbb-8b2c-675432367250/media/POWER9-VUG.pdf slides], [[User:Torpcoms/Timemark/POWER9|timemarks]])</ref>
+
* '''DD 2.1''' was the final preproduction revision before GA. It has errata preventing the use of hardware virtualization (HVM, e.g. KVM-HV), but DD 2.1 Sforza can be used in e.g. the [[Talos II]] if this functionality is not needed. It is [https://twitter.com/biolizard89/status/1567474064448430080 likely (but unconfirmed)] that paravirtualization (PV, e.g. KVM-PR) will work fine on DD2.1; test reports welcome.
 +
* '''DD 2.2''' is the first GA revision of Nimbus. DD 2.2 Sforza is sold by RCS as simply "POWER9", implying version 1.
 +
* '''DD 2.3''' is an updated revision of Nimbus, adding [[Power ISA/Privilege States#Ultravisor State|Ultravisor]] functionality, Hardware watchpoint support, and improved Meltdown and Spectre mitigations. DD 2.3 parts are sold by RCS as "POWER9 v2".
  
 
== Modules ==
 
== Modules ==
Line 46: Line 54:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ POWER9 Modules
 
|+ POWER9 Modules
−
! Chip !!Module
+
! Chip
 +
! Module
 +
! Memory Channels
 +
! XBUS Lanes
 +
! PCIe Lanes
 +
! OpenCAPI Lanes
 +
! Socket
 
|-
 
|-
 
!rowspan="3"|Nimbus
 
!rowspan="3"|Nimbus
−
| [[Sforza|Sforza]]
+
| [[Sforza]]
 +
| 4
 +
| 1
 +
| 48
 +
| 0
 +
| LGA 2601
 
|-
 
|-
−
| [[Monza|Monza]]
+
| [[Monza]]
 +
| 8
 +
| 1
 +
| 34
 +
| 48
 +
| LGA 3899
 
|-
 
|-
−
| [[LaGrange|LaGrange]]
+
| [[LaGrange]]
 +
| 8
 +
| 2
 +
| 42
 +
| 16
 +
| LGA 3899
 
|-
 
|-
 
! Cumulus
 
! Cumulus
−
|
+
| (unknown)
 +
| (memory attached via [[Centaur|Centaurs]])
 +
| (unknown)
 +
| (unknown)
 +
| (unknown)
 +
| ?
 +
|-
 +
! Axone
 +
| (unknown)
 +
| (memory attached via OMI)
 +
| Up to 3
 +
| Up to 48
 +
| Up to 48
 +
| ?
 
|}
 
|}
  
−
Nimbus chips are available in three different modules: Sforza, Monza, and LaGrange.  Each module exposes different I/O functionality to the host platform, allowing purpose-built systems to be constructed in addition to more general-purpose computers. [[Sforza|Sforza]] is the most flexible of these packages, providing PCIe 4.0 lanes as the main I/O resource, and is what [[Talos_II|Talos™ II]] uses for maximal similarity to existing desktop, workstation, and server systems. [[Monza|Monza]] modules offer the most OpenCAPI/NVLink bandwidth and are used in IBM's AC922 (Witherspoon) systems, such as those used by the Sierra and Summit supercomputers. [[LaGrange|LaGrange]] modules offer increased XBus bandwidth between processor sockets and are used by the Google/Rackspace Zaius motherboard used in the Barreleye G2 system.<ref>Gangidi, Adi [https://blog.rackspace.com/zaius-barreleye-g2-server-development-update-2 Zaius/Barreleye G2 Server Development Update]. 2017-11-13</ref>
+
'''''XBUS''' is used for inter-processor communication on dual-socket system''
 +
 
 +
=== Nimbus ===
 +
 
 +
Nimbus chips are available in three different modules: [[Sforza]], [[Monza]], and [[LaGrange]].  Each module uses the same silicon mask but is packaged differently, exposing different I/O functionality to the host platform, allowing purpose-built systems to be constructed in addition to more general-purpose computers.
 +
 
 +
'''[[Sforza|Sforza]]''' is the most flexible of these packages, providing PCIe 4.0 lanes as the main I/O resource, and is what [[Talos_II|Talos™ II]] uses for maximal similarity to existing desktop, workstation, and server systems.
 +
 
 +
'''[[Monza|Monza]]''' modules offer the most OpenCAPI/NVLink bandwidth and are used in IBM's AC922 (Witherspoon) systems, such as those used by the Sierra and Summit supercomputers.
 +
 
 +
'''[[LaGrange|LaGrange]]''' modules offer increased XBus bandwidth between processor sockets and are used by the Google/Rackspace Zaius motherboard used in the Barreleye G2 system.<ref>Gangidi, Adi [https://blog.rackspace.com/zaius-barreleye-g2-server-development-update-2 Zaius/Barreleye G2 Server Development Update]. 2017-11-13</ref>
 +
 
 +
Part numbers for different POWER9 Sforza SKUs can be found on page 58 of the [[:File:POWER9 Sforza DS v16 23JUL2018 pub.pdf|datasheet]]. These part numbers are printed on the surface of the CPU module and can be used to determine the type of the CPU.
 +
 
 +
=== Cumulus ===
 +
 
 +
Little is known about Cumulus chips at this time; as Scale Up chips, they will trade some I/O bandwidth for support for more than two sockets.<ref>Morgan, Timothy Prickett. [https://www.nextplatform.com/2017/12/05/power9-to-the-people/ POWER9 to the People]. 2017-12-05</ref>
 +
 
 +
=== Axone ===
  
−
Little is known about Cumulus chips now; as Scale Up chips, they will trade some peripherals bandwidth for communication between more than 2 sockets.<ref>Morgan, Timothy Prickett. [https://www.nextplatform.com/2017/12/05/power9-to-the-people/ POWER9 to the People]. 2017-12-05</ref>
+
Branded POWER9&prime; ("POWER9 Prime"), also known as POWER9 with Advanced I/O. Newly announced in August 2019. Uses serial memory attachment via OMI, an evolution from the [[Centaur]].
  
 
== References ==
 
== References ==
Line 69: Line 129:
 
== Resources ==
 
== Resources ==
  
 +
* '''[[:Category:Documentation|POWER9 CPU and Platform Documentation]]'''
 +
* [[POWER9 Hardware Compatibility List]]
 
* [[:File:POWER9-Features-and-Specifications.pdf|Basic POWER9 overview presentation]]
 
* [[:File:POWER9-Features-and-Specifications.pdf|Basic POWER9 overview presentation]]
 
* [[:File:PowerISA_public.v3.0B.pdf|Power ISA version 3.0B]] - implemented by POWER9
 
* [[:File:PowerISA_public.v3.0B.pdf|Power ISA version 3.0B]] - implemented by POWER9
Line 76: Line 138:
 
* [https://en.wikipedia.org/wiki/POWER9 POWER9 English Wikipedia page]
 
* [https://en.wikipedia.org/wiki/POWER9 POWER9 English Wikipedia page]
 
* [https://en.wikichip.org/wiki/ibm/microarchitectures/power9 POWER9 wikichip page]
 
* [https://en.wikichip.org/wiki/ibm/microarchitectures/power9 POWER9 wikichip page]
 +
[[Category:POWER9|*]]

Latest revision as of 06:41, 8 September 2022

Processor Information
POWER9
Power ISA 3.0B
Process node 14nm
Maximum slices 24
Maximum cores 12 SMT8 / 24 SMT4
L2 cache / slice 512kB
L3 cache / slice 10MB
Production availability January 2018
Production stepping(s) (Nimbus) DD2.2, DD2.3
← POWER8E POWER10 →

POWER9 is IBM's most recent POWER compatible server and workstation CPU (POWER ISA v3.0B). Built on a 14nm process, each CPU package can contain up to 24 SMT4 cores or 12 SMT8 cores. Each pair of SMT4 cores, or singleton SMT8 core, comprises a slice (some IBM documentation appears to call slices "chiplets"); each slice in turn contains 512kB L2 cache and 10MB L3 cache.

Pairing

Most POWER9 processors share each slice's L2 and L3 cache between cores within the slice. However, some Sforza processors provide unpaired cores, such that one SMT4 core per slice is fused off. This allows each of the SMT4 cores to utilize the full cache of the slice exclusively, increasing performance for these ST-focused processors. There are also some security implications to whether cores are paired or unpaired.

Process

POWER9 is fabricated using the GlobalFoundries 14HP (High Performance) process. This is distinct from the GlobalFoundries 14LPP (Low Power) process used by other GF 14nm customers, and is believed to be an IBM-specific process using ex-IBM Microelectronics intellectual property. The process is also used for the CPUs in IBM's z14 mainframes.[1]

Chips

There are three known silicon masks of POWER9:

  • Nimbus (POWER9 Scale Out)
  • Cumulus (POWER9 Scale Up)
  • Axone (POWER9′ ("POWER9 Prime"), aka POWER9 with Advanced I/O)

Nimbus is the “scale out” variant and uses direct-attach DDR4 memory. Cumulus is the “scale up” version and uses Centaur memory buffers, allowing larger amounts of memory to be attached to a system.

Chips can be fused as SMT4 or SMT8 during manufacturing. The SMT8 variant essentially fuses each pair of cores into one “core”, halving the core count while doubling the number of threads per core. SMT4 variants are intended for PowerNV platforms running Linux, and SMT8 variants are intended for use with IBM's PowerVM hypervisor which can run Linux, AIX or IBM i.[2]

Steppings

Several revisions of the Nimbus mask have been issued:

  • DD 2.1 was the final preproduction revision before GA. It has errata preventing the use of hardware virtualization (HVM, e.g. KVM-HV), but DD 2.1 Sforza can be used in e.g. the Talos II if this functionality is not needed. It is likely (but unconfirmed) that paravirtualization (PV, e.g. KVM-PR) will work fine on DD2.1; test reports welcome.
  • DD 2.2 is the first GA revision of Nimbus. DD 2.2 Sforza is sold by RCS as simply "POWER9", implying version 1.
  • DD 2.3 is an updated revision of Nimbus, adding Ultravisor functionality, Hardware watchpoint support, and improved Meltdown and Spectre mitigations. DD 2.3 parts are sold by RCS as "POWER9 v2".

Modules

POWER9 Modules
Chip Module Memory Channels XBUS Lanes PCIe Lanes OpenCAPI Lanes Socket
Nimbus Sforza 4 1 48 0 LGA 2601
Monza 8 1 34 48 LGA 3899
LaGrange 8 2 42 16 LGA 3899
Cumulus (unknown) (memory attached via Centaurs) (unknown) (unknown) (unknown) ?
Axone (unknown) (memory attached via OMI) Up to 3 Up to 48 Up to 48 ?

XBUS is used for inter-processor communication on dual-socket system

Nimbus

Nimbus chips are available in three different modules: Sforza, Monza, and LaGrange. Each module uses the same silicon mask but is packaged differently, exposing different I/O functionality to the host platform, allowing purpose-built systems to be constructed in addition to more general-purpose computers.

Sforza is the most flexible of these packages, providing PCIe 4.0 lanes as the main I/O resource, and is what Talos™ II uses for maximal similarity to existing desktop, workstation, and server systems.

Monza modules offer the most OpenCAPI/NVLink bandwidth and are used in IBM's AC922 (Witherspoon) systems, such as those used by the Sierra and Summit supercomputers.

LaGrange modules offer increased XBus bandwidth between processor sockets and are used by the Google/Rackspace Zaius motherboard used in the Barreleye G2 system.[3]

Part numbers for different POWER9 Sforza SKUs can be found on page 58 of the datasheet. These part numbers are printed on the surface of the CPU module and can be used to determine the type of the CPU.

Cumulus

Little is known about Cumulus chips at this time; as Scale Up chips, they will trade some I/O bandwidth for support for more than two sockets.[4]

Axone

Branded POWER9′ ("POWER9 Prime"), also known as POWER9 with Advanced I/O. Newly announced in August 2019. Uses serial memory attachment via OMI, an evolution from the Centaur.

References

  1. ↑ Schor, David. GlobalFoundries 14HP process, a marriage of two technologies. Wikichip Fuse.
  2. ↑ Stuecheli, Jeff. POWER9. Presentation for AIX VUG. (video download, slides, timemarks)
  3. ↑ Gangidi, Adi Zaius/Barreleye G2 Server Development Update. 2017-11-13
  4. ↑ Morgan, Timothy Prickett. POWER9 to the People. 2017-12-05

Resources

External Links

Retrieved from "https://wiki.raptorcs.com/w/index.php?title=POWER9&oldid=3914"