Einleitung

Hardware and Configurations of the Cluster

Hardware-Übersicht 2015

The cluster has three sections:

MPI section for MPI intense applications

MEM section for applications that need a lot of memory

ACC section for applications that use accelerators

The whole system is located at the HPC building (L5|08) on campus Lichtwiese and consists of two building stages (the later phase with one extension). Phase I has been operational since fall 2013. Phase II is in operation since February 2015 and has been extended end of 2015.

Hardware of Phase I (operating since 2013)

780 compute nodes and 4 login nodes

  • Processors: Overall ~261 TFlop/s computing power (DP-exactness, peak – theorectical)
    • realistically achieved approximately 216 TFlop/s computing power with linpack
  • Accelerators: Overall ~168 TFlop/s computing power (DP-exactness, peak – theorectical)
    • realistically achieved approximately 119 TFlop/s computing power with linpack
  • Memory: overall ~32 TByte main memory
  • The computing nodes are subdivided into 19 islands:
    • 1 x MPI island with 162 nodes (2592 cores, overall 5184 GByte main memory)
    • 2 x MPI island with each 32 nodes (512 cores and 2048 GByte main memory per island
    • 15 x MPI island with each 32 nodes (512 cores and 1024 GByte main memory per island)
    • 1 x ACC island with 44 nodes (ACC-G) and 26 nodes (ACC-M), 4 nodes (MEM)

Hardware of Phase II (operating since 2015)

632 compute nodes and 8 login nodes

  • Processors: Overall ~512 TFlop/s computing power (DP-exactness, peak – theorectical)
    • realistically achieved approximately 460 TFlop/s computing power with linpack
  • Accelerators: Overall 11.54 TFlop/s computing power (DP-exactness, peak – theorectical)
  • Memory: overall ~44 TByte main memory
  • The computing nodes are subdivided into 18 islands:
    • 1 x MPI island with 84 nodes (2016 cores, overall 5376 GByte main memory)
    • 16 x MPI island with each 32 nodes (768 cores and 2048 GByte main memory per island)
    • 1 x ACC island with 32 nodes (ACC-N) – 3x with accelerators and 29x without

Storage Phase I+II (overall operational since 2015)

  • HOME file system: 500 TByte, 5 GB/s read and 4 GB/s write
  • PROJECTS & SCRATCH file systems: 1.5 PByte, 40 GB/s read and write

Usage of computing nodes

  • Each node can be used as a single node (you can use 16, 24, 60 or 64 cores)
  • You can use several nodes with similar data communication (MPI) with InfiniBand:
    • Each island by itself (for example for MPI 32 or up to 161 cores)
    • Cross-islands with limited bandwidth (on request)