High Performance Computing Systems

Onyx Onyx
Onyx is a Cray XC40/50 system. It has 4,806 standard compute nodes, 8 large-memory compute nodes, 32 GPU compute nodes, 32 Knights Landing (Phi) compute nodes, and 64 Machine Learning Accelerator (MLA) multi-GPGPU nodes (a total of 4,942 compute nodes or 217,128 compute cores). It is rated at 6.06 peak PFLOPS.

Node Configuration
Login Standard Large-Memory KNL GPU 2-MLA 10-MLA
Total Nodes 12 4,806 8 32 32 60 4
Processor Intel E5-2699v4 Broadwell Intel E5-2699v4 Broadwell Intel E5-2699v4 Broadwell Intel 7230 Knights Landing Intel E5-2699v4 Broadwell Intel 6148 Skylake Intel 6148 Skylake
Processor Speed 2.8 GHz 2.8 GHz 2.8 GHz 1.3 GHz 2.8 GHz 2.4 GHz 2.4 GHz
Sockets / Node 2 2 2 1 1 2 2
Cores / Node 44 44 44 64 22 40 40
Total CPU Cores 528 211,464 352 2,048 704 2,400 160
Usable Memory / Node 247 GB 121 GB 1 TB 90 GB 247 GB 172 GB 735 GB
Accelerators / Node None None None None 1 2 10
Accelerator n/a n/a n/a n/a NVIDIA P100 PCIe 3 NVIDIA V100 SXM 2 NVIDIA V100 PCIe 3
Memory / Accelerator n/a n/a n/a n/a 16 GB 16 GB 32 GB
Storage on Node None None None None None None None
Interconnect Ethernet Cray Aries Cray Aries Cray Aries Cray Aries InfiniBand InfiniBand
Operating System SLES CLE CLE CLE CLE CentOS CentOS

File Systems on Onyx
Path Formatted Capacity File System Type Storage Type User Quota Minimum File Retention
/p/home ($HOME) 897 TB Lustre HDD 100 GB None
/p/work ($WORKDIR) 13 PB Lustre HDD None 30 Days
/p/cwfs ($CENTER) 3.3 PB GPFS HDD 100 TB 120 Days
/p/app ($PROJECTS_HOME) 897 TB Lustre HDD None None

Carpenter Carpenter
Carpenter is an HPE Cray EX4000 system located at the ERDC DSRC. It has 1,440 standard compute nodes, 4 large-memory nodes, and 8 GPU nodes (a total of 278,272 compute cores). It has 518 TB of memory and is rated at 9.348 peak PFLOPS.

Node Configuration
Login Standard Large-Memory Visualization
Total Nodes 10 1,440 4 8
Processor AMD 9654 Genoa AMD 9654 Genoa AMD 9654 Genoa AMD 7713 Milan
Processor Speed 2.1 GHz 2.1 GHz 2.1 GHz 2.0 GHz
Sockets / Node 2 2 2 2
Cores / Node 192 192 192 128
Total CPU Cores 1,920 276,480 768 1,024
Usable Memory / Node 8 GB 349 GB 2.973 TB 467 GB
Accelerators / Node None None None 1
Accelerator n/a n/a n/a NVIDIA A40 PCIe 4
Memory / Accelerator n/a n/a n/a 45 GB
Storage on Node 1.3 TB NVMe SSD None 8.8 TB NVMe SSD None
Interconnect Ethernet HPE Slingshot HPE Slingshot HPE Slingshot
Operating System SLES 15 SLES 15 SLES 15 SLES 15

File Systems on Carpenter
Path Formatted Capacity File System Type Storage Type User Quota Minimum File Retention
/p/home ($HOME) 919 TB Lustre HDD 100 GB None
/p/work ($WORKDIR) 3.8 PB Lustre NVMe SSD 50 TB 21 Days
/p/cwfs ($CENTER) 3.3 PB GPFS HDD 100 TB 120 Days
/p/app ($PROJECTS_HOME) 460 TB Lustre HDD None None