A New Approach to Computer Performance Measurement

What is HINT?

HINT measures the capability of any digital computer. With it, you can predict and understand how well a computer will run a complete application. . . yet HINT takes very little time or effort to use. It produces a fair comparison for

• any architecture
• any precision
• any memory size

and it scales to any speed. HINT stands for Hierarchical INTegration. It produces a measure of work done based on the quality of the answer, not the activity of getting that answer. Speed can then be measured in QUality Improvement Per Second, or QUIPS.

Background

Computers are often compared with inaccurate and misleading metrics like "megahertz rates" or "MIPS" that have no scientific foundation and don't predict real performance on real applications. Benchmarks like SPEC, LINPACK, and the NAS Parallel Benchmarks have to be revised frequently as computers become faster and have more memory. Some "bench marks" have tens of thousands of lines of source program, making it more expensive to benchmark than to simply convert your application!

The SLALOM benchmark introduced by Ames Lab in 1990 solved many of the problems with measuring performance by scaling problem size but restricting run time and output precision. HINT makes the breakthrough of rigorously measuring the knowledge produced by a computer as a step function of how long it runs. This reveals the range from "burst speed" for very small problems to large problems that must use mass storage; in other words, it makes the speed of each memory regime visible for the first time. Since the timed portion of the program fits on two pages, it is very easy to port and use . . . yet it captures the salient features of real application. It's not just a "kernel" benchmark.

Reading HINT Graphs

This graph shows a typical HINT result for a workstation. The form of the HINT curve reveals machine "personality." The farther left the curve starts, the better the latency of the system. Note how the 64-bit performance has several dropoffs, marking the end of primary cache, secondary cache, and main memory. This workstation has good performance over a broad time scale. It has enough main memory to show high speed for tasks in the several-second range. After about three seconds, the computer runs out of main memory and performance falls off with paging to the hard disk.

Things to look for in HINT graphs:

• A jittery curve indicates a machine distracted by interrupts, perhaps from daemons or other users.
• A curve of narrow width indicates a special-purpose computer.
• If the curve drops drastically when it falls out of cache, the computer depends on cache for performance and might be surprisingly slow on large simulations.
• If it has ample memory but fades fast (like the integer line above), then it ran out of precision and should be tested with another data type.
• If the curve for integers is much better than for floating-point, it may indicate the architecture is optimized for word processing or financial tasks, not scientific computing.

Where HINT Comes From

A Single Number: "Net QUIPS"

HINT consolidates precision, amount of memory, speed of memory, and arithmetic speed into a single figure of merit. That figure is the area under the HINT graph (logarithmic integral), called Net QUIPS. The more memory and precision, the more the graph stays at a high high plateau. If a computer has

• insufficient memory speed
• uneven instruction speeds
• long pipeline startup
• a memory hog operation system

or other design shortcomings, some of the HINT graph will fall and the Net QUIPS will be reduced.

Vendor, Hardware		No. of PE's	Net MQUIPS, data type	Operating System		Compiler and Command Options
Intel Paragon		1840	633. fp	SUNMOS		icc -0-4 -knoieee Mvect
		512	249.	"		"
		64	46.2	"		"
		32	25.7	"		"		"
		16	13.5	"		"
		8	7.07	"		"
		4	3.76	"		"
		2	2.03	"		"
		1	1.22	"		"
Intel Paragon		32	12.6 fp	OSF/1 1.0.4		cc -03 -knoieee
nCUBE 2S		256	35.8 fp	IRIX 4.0.5+ Vertex 3.2		ncc -02 -ncube2s
		128	18.4 fp	"		"
		64	9.42 fp	"		"
		32	4.84 fp	"		"
		16	2.49 fp	"		"
		8	1.29 fp	"		"
		4	0.67 fp	"		"
		2	0.36 fp	"		"
		1	0.26 fp	"		"
nCUBE 2		128	12.6 fp	IRIX 4.0.5+ Vertex 3.2		ncc
		64	7.81 fp	"		"
		32	4.00 fp	"		"
		16	2.06 fp	"		"
		8	1.07 fp	"		"
		4	0.57 fp	"		"
		2	0.33 fp	"		"
		1	0.20 fp	"		"
SGI Challenge L R 4400/150		8	17.5 fp	IRIX 5.2		cc v3.18 -03 -sopt
		4	10.2 fp	"		"
		1	4.62 fp	"		"
MasPar MP-1		16384	16.5 fp	ULTRIX 4.3		mpl
MasPar MP-2		4096	15.7 fp	ULTRIX 4.3		mpl
HP 712/80i		1	3.48 fp	HP-UX 9.05		gcc v2.5.8 -03
DEC 3000/300L		1	3.39 fp	OSF/1 1/3		cc -03
SGI Indy SC R4000/100		1	2.70 fp	IRIX 5.2		cc v3.18 -03 -sopt
Sun SPARC 10		1	2.34 fp	SunOS 5.3		gcc v2.5.8 -03
IBM PC Pentium		1	2.09 fp	MS DOS 5.0		gcc 2.5.7 -03
SGI Indy PC R4000/100		1	1.86 fp	IRIX 5.2		cc v3.18 -03
DEC 5000/240		1	1.31 fp	ULTRIX 4.3		cc -03
SGI Indigo R3000/33		1	0.97 fp	IRIX 5.2		cc v3.18 -03
IBM PC		1	0.82 fp	MS DOS 5.0		gcc 2.5.7 -03
COMPAQ		1	0.38 fp	MS DOS 5.0		gcc 2.5.7 -03

The table shows some Net QUIPS numbers. All were run by Quinn snell at Ames Lab in November 1994, except T3D and CM-5 data which were provided by Paul Hinker of Los Alamos National Laboratories in October 1994.

Getting HINT Via Internet

The HINT directory is located within /pub. It is organized into subdirectories containing the current version of HINT for a variety of architectures. A technical paper describing it is in the doc subdirectory. Also posted is a table of machines and their respective Net QUIPS rating, and the HINT data file for that rating.

The ftp site does not allow uploading. Therefore, any data files that you wish to post and any questions you have should be mailed to hint@scl.ameslab.gov.

Using HINT

HINT is designed to be very easy to use. It takes only minutes for conventional computers, including the time to modify it for a particular system. Even for exotic computers needing software restructuring, HINT can usually be converted and run in less than an hour.

The HINT files are self-managing, and include instructions for use. You can access a completed HINT run from the database or lead the programs onto your computer and execute them. The program produces data that can be graphed or printed out and compared with other computers.

Keeping it Honest

As with other Ames Lab performance analysis efforts, our reporting standards are high and we will not accept or post anonymous data. Submitters are held accountable for their results. Ames Laboratory is an unbiased entity, without commercial ties to any computer vendor. HINT data should never be accepted directly from a vendor without confirmation by the Scalable Computing Laboratory. We welcome your participation in creating and maintaining the HINT database, which we expect to grow to be a national resource.

Contact:	John Gustafson		gus@scl.ameslab.gov
Alternate:	Quinn Snell	+1-515-294-6751	snell@scl.ameslab.gov

Pages prepared by Maria E. Blanco.