M.F. Guest

Contents

• ABSTRACT
• 1. INTRODUCTION
• 2. The SPEC BENCHMARKS
• 3. THE DL_POLY BENCHMARK
• 4. SUMMARY
• 5. REFERENCES

ABSTRACT

This report compares the performance of a number of different computer systems using the DL_POLY software. The comparison involves twenty seven computers, including scientific workstations from IBM, Sun, Hewlett Packard, Digital and Silicon Graphics, and Pentium-based PCs. The benchmark suite consists of a set of six typical MD simulations detailed below.

1. INTRODUCTION

Workstations that have been benchmarked, include those from

• IBM, with the Power, Power2 and Power2 Super chip (P2SC), and the recently released Power3 IBM Risc system RS6000-based models. The power3 CPU is available in the dual processor IBM RS/6000 43P Model 260.
• Hewlett Packard with the PA-RISC based HP model 9000 series, including the more recent PA8000 and PA8200 CPUs, the former in the C160 (160 MHz), the latter in the HP PA-9000/V2200 (the Exemplar-V class), PA-9000/C200 (200 MHz) and PA-9000/C240 (236 MHz).
• Compaq/DEC, with the AXP alpha EV5-based PW AU and 8400 series. Also included are several systems housing the more recent A21264 EV6 CPU. These include the DEC Alpha 8400/6-575, a prototype of the XP1000 workstation from Compaq (featuring a 450 MHz EV6 CPU), the 500 MHz A21264-based Compaq DS20 AlphaServer (dual processor) and ES40 (quad processor), and the Compaq GS140 with 525 MHz A21264.
• Silicon Graphics, with the R10k (Power Challenge, Origin200, Origin2000, Octane and the ill-fated O2) and R5k CPUs (Indy and O2 R5k). The R10k CPUs include those rated at 175, 195 and 250 MHz.
• SUN, with the Ultra-SPARC-2 processors (Ultra-2/200, Ultra-2/300, Ultra30/300 and Enterprise HPC4500, 336MHz).

We should stress from the outset that our access to much of the hardware evaluated herein has been at best short lived, and has often involved the temporary loan or donation of machines as part of one of the hardware evaluation exercises run at the Daresbury Laboratory. In many cases these machines were not optimally configured in terms of either memory, or high speed disk, and consideration of the results presented here should be viewed in that light.

Following an introductory evaluation of hardware based on the SPEC Benchmarks (section 3), we present in Sections 4, and 5 results using the DL_POLY simulation program [1].

Note that the present results are taken from a more detailed report on computational chemistry benchmarks; the associated MS powerpoint presentation is also available.

2. The SPEC BENCHMARKS

One of the most useful indicators of CPU performance is provided by the SPEC (``Systems Performance Evaluation Corporation'') benchmarks. This benchmark suite contains non-tuned application-based code to measure processor speed for both integer (SPECint) and floating point (SPECfp) arithmetic. While earlier versions of the suite (e.g. SPECmark89) had certain well-advertised flaws, the more recent offerings, SPECfp95 and SPECint95 have become industry standards in measuring primarily the performance of a system's processor, memory architecture, operating system and compiler.

SPECfp95 is derived from the results of ten floating-point benchmarks compiled with aggressive optimization. It is the geometric mean of ten normalized ratios (one for each floating-point benchmark). SPECint95 is derived from the results of eight integer benchmarks compiled with aggressive optimization. It is the geometric mean of eight normalized ratios (one for each integer benchmark) Note that the level of optimization is not mandated. While highly aggressive optimization is permitted, results derived from benchmarks compiled with conservative optimization (as in SPECbase) can be submitted.

SPECfp95 and SPECint95 results for many of the CPUs discussed in this paper are given in Table 1. It is clear that no single CPU has dominated the SPECfp95 ratings over the recent past, that is until the arrival of the EV6/21264 from Digital. Thus until July 1998, the P2SC/160 CPU in the IBM RS/6000-397 exhibited the most impressive SPECfp95 rating; with a value of 26.6, the P2SC was marginally faster than the HP PA-9000/C240, 1.2 times faster than the HP PA-9000/C200 and SUN Enterprise HPC4500/336, 1.3 times faster than the DEC Alpha 8400/5-625 and SUN Ultra-2/300, and 1.4 times faster than the R10k-based SGI Origin2000/195. This picture has changed quite drastically with the arrival of the EV6. With SPECfp95 ratings of 47.7 (in the 8400/6-575) and 58.7 in the Compaq DS20, the EV6 alpha is seen to be more than twice as fast as the other leading processors of Table 1. The following points should be noted regarding the values specified in this Table;

1. SPEC ratings for the Compaq XP1000 6/450 have been derived by extrapolating values from the Compaq AlphaServer DS20 (500 MHz); the values quoted have not been confirmed, and should be viewed with this in mind.
2. Values quoted for the Cray T3E systems are again estimates based on extrapolations from the corresponding Digital EV5 specifications. No SPEC figures have ever been submitted by Cray/SGI for the T3D or T3E series.

Using the Compaq AlphaServer DS20 value of 58.7 to normalises the SPECfp ratings, we would expect the DS20 and ES40 to be somewhat ahead of the other EV6-based machines (the Compaq Alpha GS140 and Alpha 8400/6-575), given the more optimal memory subsystem involved. These four machines appear far superior to the remainder; based on this performance metric, the DS20 is seen to be 1.95 times the power3-based 200 MHz IBM RS/6000-43P/260 and 2.3 times the HP PA-9000/C240 and 250 MHz R10k-based SGI Origin2000. All other CPUs are projected to be significantly less than half the speed.

While the EV6-based machines from Compaq/DEC are also seen to dominate the SPECint95 ratings, a quite different ordering of the processors from IBM (Power3, Power2SC and Power2) and SGI (both R10k and R8k) is seen compared to the SPECfp95 ratings; both are now slower than those from DEC (EV5-based) and SUN. We also note that the Specint95 ratings suggest that Pentium II/400 is 1.7 times slower than the Compaq AlphaServer DS20, while the SPECfp95 ratings point to the Pentium being a factor of 4.5 times slower.

When considering the present benchmarking results, there are several factors we wish to consider in assessing the usefulness of the SPEC ratings;

i. Do the SPECfp95 values provide a reliable metric for evaluating the capabilities of hardware in computational chemistry? If so, we would expect to find a close mapping of the ratios for the various chemistry benchmarks onto the SPECfp ratios;

ii. Does any particular CPU consistently ``underperform'' based on the SPECfp criteria? - this would manifest itself as the ratios from the chemistry benchmarks falling below the SPECfp ratios. In particular we shall look for indicators of the memory problems of the SGI O2-R10k impacting on the benchmarks.

We will attempt to address these issues below. Finally, we note that A SPEC FAQ describing the SPEC benchmark suite and the SPEC consortium is periodically posted to comp.benchmarks, and can be found on the WWW at

http://www.specbench.org/spec/faq

An excellent summary of the SPEC benchmarks that is periodically updated is available via anonymous ftp from ftp.cs.toronto.edu in the file /pub/spectable More SPEC-related information is available at the SPEC WWW site,

http://www.specbench.org

and at the Performance Database Web site,

http://performance.netlib.org/performance/html/spec.html#specsite.

3. THE DL_POLY BENCHMARK

The benchmark summarised below is designed to reflect the typical range of simulations undertaken by the molecular dynamicist. It includes 6 calculations carried out using the DL_POLY molecular dynamics code, and includes the following functionality;

• Benchmark 1: Simulation of a sodium-potassium disilicate glass (1080 atoms, 300 time steps);
• Benchmark 2: Simulation of metallic aluminium with Sutton-Chen potential at 300K (256 atoms, 8000 time steps);
• Benchmark 3: Simulation of valinomycin in 1223 water molecules (3837 atoms, 100 time steps);
• Benchmark 4: Dynamic Shell model water structure (768 atoms, 1024 sites, 1000 time steps);
• Benchmark 5: Dynamic Shell model MgCl₂ structure (768 atoms, 1280 sites, 1000 time steps);
• Benchmark 8: Simulation of a model membrane with 2 membrane chains, 202 solute molecules and 2746 solvent molecules (3148 atoms, 1000 time steps).

The data presented in Table 2 is collected under control of the UNIX command time where available, and includes CPU time (both user and system), total elapsed time and Efficiency, measured as CPU versus elapsed. The total user CPU timings of Table 2 refer to the summed user CPU timings over all 6 calculations of the benchmark. Note that in contrast to the QC benchmark, little I/O is performed by the DL_POLY calculations, so that efficiency should always be high assuming the benchmarks were conducted on a dedicated resource.

The total CPU timings of Table 2 suggest that the Digital/Compaq Alpha CPU and, to a lesser extent, the SGI 250 MHz R10k, are dominant. The Compaq AlphaServer GS140 is the optimum CPU (13.9 minutes), slightly faster than the AlphaServer DS20 and DS40 (14.3 and 14.5 minutes respectively) and the Compaq XP1000 6/450 (15.6 minutes).The EV6-based GS140 outperforms the EV5-based DEC Alpha 8400/5-625 (19.8 mins.) and the Alpha PW/600AU (20.2 mins.) by a factor of 1.45, and the SGI Origin2000/250 (21.6 mins.) by a factor of 1.55. These are followed by the SGI Octane/250 (24.5 mins.), the DEC Alpha PW/433AU (28.1 mins.), SGI Origin2000/195 (29.9 mins.) and SGI PChall-R10k/195 (33.1 mins). The leading 11 CPUs are from either Digital/Compaq or Silicon Graphics. Note that the incorporation of DL-POLY into the benchmark suite came after the availability of the Alpha 8400/6-575.

When considering the performance of the CPUs from SUN, IBM and Hewlett Packard, we would note the following:

• The treatment of dynamic memory within Version 2.11 of DL_POLY relies on usage of the f90 compiler. This explains the exceedingly poor performance of DL_POLY on the SUN HPC4500/336, for SUN's current f90 compiler typically generates extremely inefficient code, running only half the speed of that generated using f77.
• All CPUs from IBM perform badly on this benchmark. The power3 RS/6000-43P Model 260 is seen to be 2.6 times slower than the Compaq AlphaServer GS140, the P2SC SP2/120Thin 4.9 times slower, and the IBM RS/6000-59H a factor of 7.7 times slower. These factors are much higher (by a factor of two) than those suggested by the SPECfp95 ratings (1.5, 2.7 and 4.3 respectively).
• The PentiumII/400 performs broadly in line with expectations, a factor of 3.2 times slower than the Compaq XP1000 6/450.
• The performance of the Cray T3E is also disappointing. Just why the T3E/1200 is outperformed by the DEC Alpha PW/600AU by a factor of 2.1, and the T3E/900 by the DEC Alpha PW/433AU by a factor of 1.8, remains unclear.

4. SUMMARY

Based on the published SPECfp95 ratings, and normalising with respect to the Compaq AlphaServer DS20 value of 58.7, we would expect (see section 1) the Alpha DS20 and ES40 to be somewhat ahead of the other EV6-based machines (the Compaq Alpha GS140 and Alpha 8400/6-575), with these four machines appear far superior to the remainder. The DS20 is seen to be 1.95 times the power3-based 200 MHz IBM RS/6000-43P/260 and 2.3 times the HP PA-9000/C240 and 250 MHz R10k-based SGI Origin2000. All other CPUs are projected to be significantly less than half the speed.

While the EV6-based machines from Compaq/DEC are also seen to dominate the SPECint95 ratings, a quite different ordering of the processors from IBM (Power3, Power2SC and Power2) and SGI (both R10k and R8k) is seen compared to the SPECfp95 ratings; both are now slower than those from DEC (EV5-based) and SUN. We also note that the Specint95 ratings suggest that Pentium II/400 is 1.7 times slower than the Compaq AlphaServer DS20, while the SPECfp95 ratings point to the Pentium being a factor of 4.5 times slower.

When analysing the results, we wish to consider based on the present evaluation exercise, (i) do the SPECfp95 values provide a reliable metric for evaluating the capabilities of hardware in computational chemistry? If so, we would expect to find a close mapping of the ratios for the various benchmarks onto the SPECfp95 ratios, (ii) does any particular CPU consistently ``underperform'' based on the SPECfp criteria? - this would manifest itself as the ratios from the benchmarks falling below the SPECfp ratios, and (iii) do the ``simple'' Matrix and Chemistry Kernel benchmarks lead to the same conclusions as the GAMESS-UK and DL_POLY benchmarks? To these ends an approximate Performance Index (PI) has been devised for each machine, based on an average value of the Matrix-97, Chemistry Kernels and DL_POLY benchmarks. A full discussion is to be found in the paper on computational chemistry benchmarks. Here we present the conclusions only.

• While the EV6-based Compaq AlphaServer GS140, DS20 and ES40 are clearly the optimum CPUs based on the benchmarks conducted in this report, this superiority is not as pronounced as might be expected from just a consideration of the SPECfp95 rankings. It would certainly appear that our suggested SPECfp rating for the Compaq XP1000 is an underestimate, for the 450 MHz CPU achieves a PI of 102% (ratios of 88%, 122% and 97% for the Matrix-97, Chemistry Kernel and DL_POLY benchmarks), against the projected SPECfp95 ratio of 78%.
• Following the Compaq GS140, DS20 and ES40, the optimum CPUs based on the computational chemistry benchmarks conducted in both this and the full report, with the associated PI, would appear in order to be the Compaq/DEC Alpha 8400/6-575 and Compaq Alpha GS140 (92%), Compaq XP1000 (90%), IBM RS/6000-43P (77%) , HP PA-9000/C240 (63%), SGI Origin2000/250 (61%), HP PA-9000/V2200 (59%), the DEC Alpha PW600AU (58%) and DEC Alpha 8400/5-625, IBM RS/6000-397 and SGI Octane/250 (55%), and the SUN HPC4500/336 (52%). The apparent strength of the IBM RS/6000-43P must be considered in light of the DL_POLY molecular dynamics benchmarks; inclusion of these figures would have lowered the PI significantly (from 77 to 68%). All the above machines exhibit PI's significantly greater than the corresponding SPECfp95 ratios.
• The results suggest that while the expected superiority of the AlphaServer DS20, ES40 and GS140 is shown by both GAMESS-UK and DL_POLY benchmarks, with relative performance figures broadly in line with the SPECfp-based rankings, this superiority is less marked in the MATRIX-97 benchmark, and is certainly not evident in the chemistry kernel benchmark.
• In the chemistry kernel benchmark all machines, with the possible exception of the SUN Ultra, show higher performance ratios relative to the DS20 and ES40 than expected based on their SPECfp95 rating. This effect is often quite marked; the IBM RS/6000-43P, SGI Origin2000/250, SGI Octane/250, DEC Alpha PW/600AU, IBM RS/6000-397 and P2SC/160 exhibit performance ratios ranging from 67% to 59%, to be compared with the corresponding the SPECfp95 ratios that range from 51% to 36%. Clearly the DS20/ES40 is underperforming in these four benchmarks. The impact of the improved memory bandwidth of the Compaq DS20 and ES40 compared to the DEC Alpha 8400/6-575 is particularly noticeable in the JACOBI benchmark.

In terms of relative speed, we find that the PI values and all the chemistry benchmarks are broadly in line with the SPECfp predictions, the only notable exceptions being summarised below:

• The good performance of the EV5-based PWAU series from Digital, with both the DEC Alpha PW/433AU and 600AU often outperforming other CPUs with higher SPECfp95 ratings.
• The memory bandwidth problems associated with the SGI O2/R10k-SC, accounting for its poor relative SPECfp95 value (26%), are not shown by either the Matrix-89 or chemistry kernels (with the exception of Jacobi). They are more evident in the GAMESS-UK benchmarks, where it performs in far inferior fashion to the SGI Octane/175.
• The relatively poor performance of the HP PA-9000/C160 on the DL_POLY benchmarks; this is caused in part by the need to drop the higher optimisation levels used by the FORTRAN compiler on key routines (due to the generation of incorrect code). The machine is outperformed by SGI Origin200, SGI Octane/175, DEC Alpha 500/5-400 and SGI PChall-R10k/195, all of which have lower SPECfp95 ratings.
• The excellent performance of the HP PA-9000/V2200 on the matrix kernels, outperforming the PA-9000/C240.
• The poor performance of the SUN Ultra-2/300 and SUN Ultra30/300 on the chemistry kernels. Also of note is the poor showing of the SUN HPC4500/336 on the DL_POLY benchmark, caused by the current status of SUN's f90 compiler. The UltraSPARC systems from SUN do appear however to be performing broadly in line with the SPECfp95 ratings.
• The cost-effective performance of the PentiumII; the 400 MHz CPU exhibits a PI value of 33, with its performance in all benchmarks exceeding expectations based solely on SPECfp95 figures. The Pentium II/266 would appear to be outperforming the Pentium 233 MMX by a factor of 2-3.

5. REFERENCES

1

DL_POLY is a parallel molecular dynamics simulation package developed at Daresbury Laboratory by W. Smith and T.R. Forester under the auspices of the Engineering and Physical Sciences Research Council (EPSRC) for the EPSRC's Collaborative Computational Project for the Computer Simulation of Condensed Phases (CCP5) and the Molecular Simulation Group (MSG) at Daresbury Laboratory. The package is the property of the Central Laboratory of the Research Councils.

2

In theory the O2-R10k should have outperformed the corresponding Indigo2; with better memory bandwidth, superior I/O and more tightly coupled integration it should have done well. However SGI made a design decision which has seriously impaired the performance of the O2 in some application areas. It took about 3 months for this "flaw" to be fully identified. Until December 1996, SGI were claiming that the O2 R10k would perform in the region of 10, 12 or even 15 SPECfp95. Indeed on some benchmarks it does indeed achieve performance that matches these figures. However, the O2 has a Unified Memory Architecture which uses main system memory as memory for the graphics display and operations. Despite the impressive bandwidth figures for the O2, it does seem that the O2 memory architecture severely impedes the performance of the R10k processor, particularly when compared with the Octane, Indigo2 and Origin systems. This is shown by the SPEC comparisons of Table 1; we suspect that the two main factors limiting performance in the memory subsystem are the main memory speed and the CRIME chip.

The CRIME chip, which acts as the memory interface between the memory and the three drains on it - the CPU (800 MByte/second), I/O engine (500 MByte/sec) and the monitor display (700 MByte/second) - is probably the main bottleneck. This chip was designed to work as a built in memory controller, but the design was biased toward the R5k; it can't work directly with the R10k because the R5k expects 32 byte cache refills while the R10k wants to have 64 or 128 byte refills. Therefore SGI supply a custom ASIC with the R10k daughter board. This interfaces the R10k's level 2 cache with the CRIME chip. Performance problems are caused by the ASIC having to break each 128 byte cache refill operation into 4, 32 byte refills. The net impact of this effect is that the O2 R10k will only work well with problems that fit into the L2 cache (1 MByte). Not surprisingly, the memory intensive SPECfp95 figures are badly affected, although the impact on less memory intensive applications is not so severe. It should be noted that this type of incident is very rare; chips often fail to deliver but not system architectures designed for existing chips.

3

see the following recently opened web page to obtain Pentium Pro Optimized BLAS and FFTs for Intel Linux: http://www.cs.utk.edu/ ghenry/distrib