Einleitung

Hardware and Configurations of the Cluster

Hardware Übersicht Compute-Sektionen

The cluster consists of three sections:

MPI section for MPI intense applications

MEM section for applications that need a lot of memory

ACC section for applications that use accelerators

The whole system is located at the HPC building (L5|08) on campus Lichtwiese and consists of two building stages (the later phase with one extension).

Phase I has been operational since fall 2013 and has been decommissioned since 2020-04-27.

Phase II is in operation since February 2015 and has been extended end of 2015.

Hardware of Phase II (operating since 2015)

632 compute nodes and 8 login nodes

  • Processors: Overall ~512 TFlop/s computing power (Double Precision, peak – theoretical)
    • realistically achieved approximately 460 TFlop/s computing power with Linpack bnchmark
  • Accelerators: Overall 11.54 TFlop/s computing power (Double Precision, peak – theorectical)
  • Memory: overall ~44 TByte main memory
  • The computing nodes are subdivided into 18 islands:
    • 1 x MPI island with 84 nodes (2016 cores, overall 5376 GByte main memory)
    • 16 x MPI island with each 32 nodes (768 cores and 2048 GByte main memory per island)
    • 1 x ACC island with 32 nodes (ACC-N) – 3x with accelerators and 29x without

Hardware of Phase II (operating since 2015)

632 compute nodes and 8 login nodes

  • Processors: Overall ~512 TFlop/s computing power (Double Precision, peak – theoretical)
    • realistically achieved approximately 460 TFlop/s computing power with Linpack bnchmark
  • Accelerators: Overall 11.54 TFlop/s computing power (Double Precision, peak – theorectical)
  • Memory: overall ~44 TByte main memory
  • The computing nodes are subdivided into 18 islands:
    • 1 x MPI island with 84 nodes (2016 cores, overall 5376 GByte main memory)
    • 16 x MPI island with each 32 nodes (768 cores and 2048 GByte main memory per island)
    • 1 x ACC island with 32 nodes (ACC-N) – 3x with accelerators and 29x without

Filesystems / Storage

Technology

The Lichtenberg storage subsystem is based on IBM's Spectrum Scale (formerly General Parallel File System).

2019

The new system provides 3 TByte of storage, built on SSDs (Solid State Disks) for metadata and ~1000 of the fastest magnetic hard disks available.

All file system's content is actively (re-) balanced over all available disks. Unlike before, there is thus no performance difference any longer between eg. /work/scratch and /home, respectively.

2015 – 2019

In the first storage system, each file system was dedicated a varying number of disks, resulting in some file systems being smaller and less performant than others.

  • total: 2 PByte
  • HOME file system: 500 TByte, 5 GB/s read and 4 GB/s write
  • PROJECTS & SCRATCH file systems: 1.5 PByte, 40 GB/s read and write

Filesystems / Storage

Technology

The Lichtenberg storage subsystem is based on IBM's Spectrum Scale (formerly General Parallel File System).

2019

The new system provides 3 TByte of storage, built on SSDs (Solid State Disks) for metadata and ~1000 of the fastest magnetic hard disks available.

All file system's content is actively (re-) balanced over all available disks. Unlike before, there is thus no performance difference any longer between eg. /work/scratch and /home, respectively.

2015 – 2019

In the first storage system, each file system was dedicated a varying number of disks, resulting in some file systems being smaller and less performant than others.

  • total: 2 PByte
  • HOME file system: 500 TByte, 5 GB/s read and 4 GB/s write
  • PROJECTS & SCRATCH file systems: 1.5 PByte, 40 GB/s read and write

Usage of computing nodes

  • Each node can be used as-is, ie. “single node” (you can use 24 cores on most or 60 cores on some nodes), with either one large or several smaller tasks
  • You can use several nodes concurrently by process communication (MPI) via InfiniBand:
    • Each island by itself (for example 32 or up to 161 nodes in one MPI job)
    • Across islands with reduced inter-island bandwidth, ie. between nodes of distinct islands (on request)