Hardware Specifications

Avi Faculty Research Cluster Specifications

  • 1136 total computing cores and 3616 GiB (3.5TiB) RAM
  • 142 compute nodes (1136 cores total). Each node is a Dell PowerEdge R410 server with two quad-core Intel(R) Xeon(R) X5550 processors @ 2.67GHz
  • Most compute nodes have 24 GiB of RAM.  A few “high memory” nodes with 128 GiB of RAM are provided for special programs that require large amounts of memory on a single node
  • One head node running the SLURM resource manager, a Dell PowerEdge R310 server with 6 Intel(R) Xeon(R) E5-2407 processors @ 2.20GHz and 32 gigabytes of RAM.   An identical backup node automatically takes over in the event of a head node failure
  • A primary IO node, a Dell PowerEdge R710 server, with two quad-core Intel(R) Xeon(R) E5520 processors @ 2.27GHz, 48 GiB of system memory and seven Dell PowerVault MD1000 3Gb/s SAS attached expansion units, serving nine shared RAID 60 and RAID 10 partitions of approximately 7 terabytes each over NFSv4
  • One high-speed I/O node, a Dell PowerEdge R720xd with two six-core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz and 32 gigabytes of RAM, serving a single 10 terabyte RAID 6 partition over NFSv4
  • All compute and I/O nodes are linked by Qlogic DDR InfiniBand (16Gb/s) and gigabit Ethernet networks

Mortimer Faculty Research Cluster Specifications

  • 1914 computing cores and 7488 GiB (7.3TiB) RAM
  • 28 standard compute nodes, each with 16 cores and 48 GiB RAM.  Each node is a Dell PowerEdge R420 server with two 8-core Intel(R) Xeon(R) E5-2450 v2 processors @ 2.50GHz
  • 55 standard compute nodes, each with 24 cores and 64 GiB RAM.  Each node is a Dell PowerEdge R430 server with two 12-core Intel(R) Xeon(R) E5-2680 v3 processors @ 2.50GHz
  • 4 high-memory compute nodes, each with 24 cores and 256 GiB RAM.  Each node is a Dell PowerEdge R630 with 2 12-core Intel(R) Xeon(R) E5-2680 v3 processors @ 2.50GHz
  • 1 high-memory compute node with 16 cores, 768 GiB RAM, and a local 17TiB RAID.  Dell PowerEdge R720xd with four 8-core Intel(R) Xeon(R) E5-2650 v2 processors @ 2.60GHz
  • 1 high-memory compute node with 24 cores, 768 GiB RAM, and a local 1TiB RAID.  Dell PoweerEdge R720xd with four 12-core Intel(R) Xeon(R) E5-2680 v3 processors @ 2.50GHz
  • 1 GPU node.  Dell PowerEdge C4130 with two 10-core Intel(R) Xeon(R) E5-2660 v3 processors @ 2.60GHz , 128 GiB 2133 MT/s RAM, and 2 NVIDIA Tesla K80 accelerators, offering a total of 9,984 CUDA cores.
  • 1 visualization node.  Dell PowerEdge R415 with 16 cores, 64 GiB RAM and two 8-core AMD Opteron(tm) 4386 Processors @ 3.1 Ghz.
  • One head node running the SLURM resource manager.  Dell PowerEdge R415 server with one 4-core AMD Opteron(tm) Processor 4133 processor @ 2.8Ghz and 16GiB  of RAM.   An identical backup node automatically takes over in the event of a head node failure
  • 10 high-speed I/O nodes.  Dell PowerEdge R720xd serving a single 19 TiB RAID over NFSv4 with write speeds up to 800 MiB/sec from compute nodes over the Infiniband network
  • 7 high-speed, high-capacity I/O nodes, each a Dell PowerEdge R720xd serving a single 37 TiB RAID over NFSv4 with write speeds up to 800 MiB/sec from compute nodes over the Infiniband network
  • All compute and I/O nodes are linked by Mellanox FDR Infiniband (56Gb/s) and gigabit Ethernet networks

Avi and Mortimer Common Specifications

  • All nodes currently run the latest CentOS Linux 6
  • Hundreds of open source packages installed via the pkgsrc package manager
  • Many commercial software packages (mostly licensed to individual research groups or colleges)
  • Intel compiler suite available to all users

Peregrine Educational Cluster Specifications

  • 8 compute nodes (96 cores total). Each node is a Dell PowerEdge R415 rack-mount server with two six-core AMD Opteron 4180 2.6GHz processors and 32 GB of system memory
  • One head node, a Dell PowerEdge R415 server, with one 6-core AMD Opteron processor and 16 GB of system memory
  • The head node houses a 5 Terabyte RAID5 array utilizing the advanced ZFS filesystem, available to all compute nodes via NFS
  • All nodes are connected by a dedicated gigabit Ethernet network interface
  • Jobs are scheduled using the SLURM resource manager
  • In addition, Peregrine is a submit node and manager for the UWM HTCondor grid, which provides access to idle cores on lab PCs and other machines around campus for use in embarrassingly parallel computing
  • All nodes run the FreeBSD operating system
  • Each compute node is preloaded with over 700 open source applications and libraries via the FreeBSD Ports package manager, including compilers such as Clang and GCC, many other languages like Java, Octave (Matlab (r) compatible), Perl, Python, R, etc., and hundreds of scientific applications and libraries including BLAS, LAPACK, NCBI BLAST, Qiime, Trinity, Falcon, and many more.

Meadows High Throughput Computing Grid

Thanks to a donation of servers and data center space from the UWM’s LIGO gravity wave research group, the Meadows grid has a base of over 400 dedicated processors available 24/7 for long-running HTCondor jobs.

Unlike many grids which employ the “Bring Your Own Binary” (BYOB) model, most of the Meadows compute hosts are preloaded with over a thousand open source applications and libraries (the same software packages as Peregrine’s compute nodes, described above).

  • Hundreds of dedicated computing cores in the UWM Physics data center
  • Hundreds of additional computing cores available through virtual machines running on lab machines around campus (available when the machines are not in use)
  • Jobs scheduled via HTCondor
  • Hosts run FreeBSD and are preloaded with over 700 open source applications and libraries (mostly the same software as Peregrine’s compute nodes, described above).
  • Unused processors on UW Madison’s CHTC grid may be utilized when the UWM grid is full (these are Bring Your Own Binary)

SciProg Development and Education Server

  • Dell PowerEdge R420 server
  • Dual Intel(R) Xeon(R) CPU E5-2450 CPUs for a total of 16 hyper-threaded cores (32 threads)
  • 64 GB RAM
  • 5.3 TB RAID storage with advanced ZFS filesystem
  • FreeBSD operating system
  • Preloaded with over a thousand open source applications and libraries (the same software packages as Peregrine’s compute nodes, described above).

Additional Notes

Mortimer I/O nodes are fully independent NFS RAID servers, operating in “embarrassingly parallel” fashion.  This configuration was strategically chosen over a parallel filesystem in order to maximize throughput for individual jobs and aggregate throughput for the cluster.  It also minimizes maintenance effort and the urgency of server failures.  The cost of this configuration is the need to manually load-balance I/O-intensive users, which is minimal where there are not many I/O-intensive jobs.