The Standard Performance Evaluation Corporation is an American non-profit corporation that aims to "produce, establish, maintain and endorse a standardized set" of performance benchmarks for computers. SPEC was founded in 1988. SPEC benchmarks are widely used to evaluate the performance of computer systems; the test results are published on the SPEC website. SPEC evolved into an umbrella organization encompassing four diverse groups; Graphics and Workstation Performance Group, the High Performance Group, the Open Systems Group and the newest, the Research Group.
Structure
The Open Systems Group (OSG)
The High-Performance Group (HPG)
The Graphics and Workstation Performance Group (GWPG)
SPEC Research Group (RG)
Membership
Membership in SPEC is open to any interested company or entity that is willing to commit to SPEC's standards. It allows:
Participation in benchmark development
Participation in review of results
Complimentary software based on group participation
The list of members is available on SPEC's membership page;.
Membership Levels
Sustaining Membership requires dues payment and typically includes hardware or software companies.
SPEC "Associates" pay a reduced fee and typically include Universities.
SPEC "Supporting Contributors" are invited to participate in development of a single benchmark, and do not pay fees.
SPEC Benchmark Suites
The benchmarks aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation to a full system with Java EE, database, disk, and network. The SPEC CPU suites test CPU performance by measuring the run time of several programs such as the compilerGCC, the chemistry program gamess, and the weather program WRF. The various tasks are equally weighted; no attempt is made to weight them based on their perceived importance. An overall score is based on a geometric mean.
Cloud
Measuring and comparing the provisioning, compute, storage, and network resources of IaaS cloud platforms.
SPEC Cloud IaaS 2018
SPEC Cloud IaaS 2016
CPU
Measuring and comparing combined performance of CPU, memory and compiler.
SPEC CPU2006 contains two suites:
* CINT2006 - testing integer arithmetic, with programs such as compilers, interpreters, word processors, chess programs etc.
* The SPECspeed 2017 Integer and SPECspeed 2017 Floating Point suites are used for comparing time for a computer to complete single tasks.
* The SPECrate 2017 Integer and SPECrate 2017 Floating Point suites measure the throughput or work per unit of time.
Handheld
Graphics and Workstation Performance
Measuring performance of an OpenGL 3D graphics system, tested with various rendering tasks from several popular 3D-intensive real applications on a given system.
SPEC Chauffeur WDK Tool. Designed to simplify the development of workloads for measuring both energy efficiency and performance.
PTDaemon. The SPEC PTDaemon software is used to control power analyzers in benchmarks which contain a power measurement component.
Benchmark Search Program
SPEC CPUv6, The CPU Search Program seeks to encourage those outside of SPEC to assist them in locating applications that could be used in the next CPU-intensive benchmark suite, currently designated as SPEC CPUv6. Obsoleted now.
Retired Benchmarks Have No Successor
SPEC SDM91
SPECsip_infrastructure2011 - the benchmark is still available for purchase but no additional result submissions are being accepted and support is no longer offered.
Retired Benchmarks No Longer Documented
SPECapcSM for Lightwave 3D 9.6, performance evaluation software for systems running NewTek LightWave 3D v9.6 software.
SPEC 2001
SPEC CPU89
Portability
SPEC benchmarks are written in a portable programming language, and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks. SPEC has rules that attempt to limit such optimizations.
Licensing
In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars. This pay-for-license model might seem to be in violation of the GPL as the benchmarks include software such as GCC that is licensed by the GPL. However, the GPL does not require software to be distributed for free, only that recipients be allowed to redistribute any GPLed software that they receive; the license agreement for SPEC specifically exempts items that are under "licenses that require free distribution", and the files themselves are placed in a separate part of the overall software package.
Culture
SPEC attempts to create an environment where arguments are settled by appeal to notions of technical credibility, representativeness, or the "level playing field". SPEC representatives are typically engineers with expertise in the areas being benchmarked. Benchmarks include "run rules", which describe the conditions of measurement and documentation requirements. Results that are published on SPEC's website undergo a peer review by members' performance engineers.
