ROSE Compiler Framework/SPEC CPU 2006 benchmark

spec cpu 2006
Assuming the source package is under /usr/casc/overture/ROSE/svn/spec_cpu2006_v1.1

installation
./install.sh -d /home/yourAccount/opt/spec_cpu2006

files and directories
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench Subdirectories
 * build data  Docs  exe  run  Spec  src

benchmark summary
12+17= 29 benchmarks

12 Integer benchmarks: http://www.spec.org/cpu2006/CINT2006/
 * 400.perlbench 	C  	 Programming Language
 * Derived from Perl V5.8.7. The workload includes SpamAssassin, MHonArc (an email indexer), and specdiff (SPEC's tool that checks benchmark outputs).


 * 401.bzip2 	C 	Compression
 * Julian Seward's bzip2 version 1.0.3, modified to do most work in memory, rather than doing I/O.


 * 403.gcc 	C 	C Compiler
 * Based on gcc Version 3.2, generates code for Opteron.


 * 429.mcf 	C 	Combinatorial Optimization
 * Vehicle scheduling. Uses a network simplex algorithm (which is also used in commercial products) to schedule public transport.


 * 445.gobmk 	C 	Artificial Intelligence: Go
 * Plays the game of Go, a simply described but deeply complex game.


 * 456.hmmer 	C 	Search Gene Sequence
 * Protein sequence analysis using profile hidden Markov models (profile HMMs)


 * 458.sjeng 	C 	Artificial Intelligence: chess
 * A highly-ranked chess program that also plays several chess variants.


 * 462.libquantum 	C 	Physics / Quantum Computing
 * Simulates a quantum computer, running Shor's polynomial-time factorization algorithm.


 * 464.h264ref 	C 	Video Compression
 * A reference implementation of H.264/AVC, encodes a videostream using 2 parameter sets. The H.264/AVC standard is expected to replace MPEG2


 * 471.omnetpp 	C++ 	Discrete Event Simulation
 * Uses the OMNet++ discrete event simulator to model a large Ethernet campus network.


 * 473.astar 	C++ 	Path-finding Algorithms
 * Pathfinding library for 2D maps, including the well known A* algorithm.


 * 483.xalancbmk 	C++ 	XML Processing
 * A modified version of Xalan-C++, which transforms XML documents to other document types.

17 Floating point benchmarks: http://www.spec.org/cpu2006/CFP2006/ Fortran 	Biochemistry / Molecular Dynamics 	Molecular dynamics, i.e. simulate Newtonian equations of motion for hundreds to millions of particles. The test case simulates protein Lysozyme in a solution. Fortran 	Physics / General Relativity 	Solves the Einstein evolution equations using a staggered-leapfrog numerical method Fortran 	Structural Mechanics 	Finite element code for linear and nonlinear 3D structural applications. Uses the SPOOLES solver library. Fortran 	Weather 	Weather modeling from scales of meters to thousands of kilometers. The test case is from a 30km area over 2 days.
 * 410.bwaves 	Fortran  	Fluid Dynamics  	 Computes 3D transonic transient laminar viscous flow.
 * 416.gamess 	Fortran 	Quantum Chemistry. 	 Gamess implements a wide range of quantum chemical computations. For the SPEC workload, self-consistent field calculations are performed using the Restricted Hartree Fock method, Restricted open-shell Hartree-Fock, and Multi-Configuration Self-Consistent Field
 * 433.milc 	C 	Physics / Quantum Chromodynamics 	 A gauge field generating program for lattice gauge theory programs with dynamical quarks.
 * 434.zeusmp 	Fortran 	Physics / CFD 	ZEUS-MP is a computational fluid dynamics code developed at the Laboratory for Computational Astrophysics (NCSA, University of Illinois at Urbana-Champaign) for the simulation of astrophysical phenomena.
 * 435.gromacs 	C,
 * 436.cactusADM 	C,
 * 437.leslie3d 	Fortran 	Fluid Dynamics 	Computational Fluid Dynamics (CFD) using Large-Eddy Simulations with Linear-Eddy Model in 3D. Uses the MacCormack Predictor-Corrector time integration scheme.
 * 444.namd 	C++ 	Biology / Molecular Dynamics 	Simulates large biomolecular systems. The test case has 92,224 atoms of apolipoprotein A-I.
 * 447.dealII 	C++ 	Finite Element Analysis 	deal.II is a C++ program library targeted at adaptive finite elements and error estimation. The testcase solves a Helmholtz-type equation with non-constant coefficients.
 * 450.soplex 	C++ 	Linear Programming, Optimization 	Solves a linear program using a simplex algorithm and sparse linear algebra. Test cases include railroad planning and military airlift models.
 * 453.povray 	C++ 	Image Ray-tracing 	Image rendering. The testcase is a 1280x1024 anti-aliased image of a landscape with some abstract objects with textures using a Perlin noise function.
 * 454.calculix 	C,
 * 459.GemsFDTD 	Fortran 	Computational Electromagnetics 	Solves the Maxwell equations in 3D using the finite-difference time-domain (FDTD) method.
 * 465.tonto 	Fortran 	Quantum Chemistry 	An open source quantum chemistry package, using an object-oriented design in Fortran 95. The test case places a constraint on a molecular Hartree-Fock wavefunction calculation to better match experimental X-ray diffraction data.
 * 470.lbm 	C 	Fluid Dynamics 	Implements the "Lattice-Boltzmann Method" to simulate incompressible fluids in 3D
 * 481.wrf 	C,
 * 482.sphinx3 	C 	Speech recognition 	A widely-known speech recognition system from Carnegie Mellon University

language summary
C : 14 benchmarks/ 12 real ones excluding specrand 400.perlbench    	 Programming Language 401.bzip2           Compression 403.gcc             C Compiler 429.mcf             Combinatorial Optimization 433.milc            Physics / Quantum Chromodynamics 445.gobmk           Artificial Intelligence: Go    456.hmmer            Search Gene Sequence 458.sjeng           Artificial Intelligence: chess 462.libquantum      Physics / Quantum Computing 464.h264ref         Video Compression 470.lbm     	 Fluid Dynamics 482.sphinx3         Speech recognition 998.specrand 999.specrand C++: 7 benchmarks : namd dealII soplex povray omnetpp astar xalancbmk 444.namd 447.dealII 450.soplex 453.povray 471.omnetpp 473.astar 483.xalancbmk Fortran: 6 benchmarks: bwaves gamess zeusmp leslie3d GemsFDTD tonto

410.bwaves 416.gamess 434.zeusmp 437.leslie3d 459.GemsFDTD 465.tonto

Mixed C and Fortran: 4: gromacs cactusADM  calculix wrf 435.gromacs 436.cactusADM 454.calculix 481.wrf

configuration
. ./shrc
 * enviornment Source the shrc or cshrc in /home/youAccount/opt/spec_cpu2006 to set up your environment for the benchmark.
 * configuration prepare config/tux268-gcc42.cfg  set compilers and flags  based on existing examples there

Table summary
 * output_format = asc, pdf, Screen, html

use
http://www.spec.org/cpu2006/Docs/runspec.html#section3.1actions

bash-3.00$ which runspec /home/youAccount/opt/spec_cpu2006/bin/runspec bash-3.00$ runspec --config=tux268-gcc42.cfg --action=build --tune=base bzip2
 * build only:

clean up first
 * runspec --config=rose-linux64-gcc41 --tune=base --size=test --iterations=1 --noreportable --action=clobber all

runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2
 * Run with test data set default is --size=ref : run with reference data set

runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2

runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable int  // total 12
 * run all integer or fp benchmarks     12 total integer

400.perlbench     --     3.78           -- S 401.bzip2          --     7.52           -- S 403.gcc            --     1.42           -- S 429.mcf            --     5.37           -- S 445.gobmk          --    20.0            -- S

456.hmmer         --     5.35           -- S 458.sjeng          --     5.00           -- S 462.libquantum     --     0.0951         -- S 464.h264ref        --    21.7            -- S 471.omnetpp        --     0.545          -- S

473.astar         --    10.7            -- S 483.xalancbmk      --     0.115          -- S

Est. SPECint_base2006                  -- Est. SPECint2006                                                  Not Run

runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable fp  // total 17

410.bwaves        --    12.4            -- S 416.gamess         --     0.510          -- S 433.milc           --     8.71           -- S 434.zeusmp         --    25.2            -- S 435.gromacs        --     1.74           -- S

436.cactusADM     --     4.66           -- S 437.leslie3d       --    32.7            -- S 444.namd           --    15.6            -- S 447.dealII         --    22.6            -- S 450.soplex         --     0.0263         -- S

453.povray        --     0.874          -- S 454.calculix       --     0.0546         -- S 459.GemsFDTD       --     4.52           -- S 465.tonto          --     1.35           -- S 470.lbm            --    13.6            -- S

481.wrf           --     5.66           -- S 482.sphinx3        --     2.26           -- S

Est. SPECfp_base2006                   -- Est. SPECfp2006                                                   Not Run

test gcc/g++/gfortran first then build only

cleanup/clobber
clean up: --action=clean or clobber then identityTranslator

runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable gobmk

data set
Reportable runs ensure that your binaries can produce correct results with the test and train workloads (this run rule requirement is new as of CPU2006 V1.0) and then run the ref workload 3 times for the actual measurements.

runspec --config=tux268-rose-gcc42.cfg --tune=base --size=ref --iterations=1 --noreportable perlbench

data sizes
 * test: smallest data set
 * train: test the ability to run more than one size in a single invocation
 * ref (default): run 3 times for reportable results

selective run
runspec --action=clobber int fp runspec --config=tux268-intel.cfg --tune=base --size=test --iterations=1 --noreportable bzip2 runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2 runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2

bash-3.00$ pwd /home/yourAccount/opt/spec_cpu2006 bash-3.00$ runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable fp

modified source run
Changed /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench/src/spec_config.h

to have #include 

Have to add a line in the config file (eg. config/tux268-rose-gcc42.cfg) to avoid corrupted source warning

strict_rundir_verify = 0

debug
pwd

/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench/src

/home/yourAccount/opt/roseLatest/bin/identityTranslator -c -o sv.o -DSPEC_CPU -DNDEBUG -DPERL_CORE  -O2 -DSPEC_CPU_LINUX_IA32        sv.c

Adding argv[3] = sv.o to p_objectFileNameList Using strcmp: This was not a valid string (buffer = define returned) Using strcmp: This was not a valid string (buffer = define returned) Error: Unknown cppIndentifier = identityTranslator: ../../../../rose/src/frontend/SageIII/rose_attributes_list.C:1676: bool ROSEAttributesList::isCppDirective(const std::string&, PreprocessingInfo::DirectiveType&, std::string&): Assertion `false' failed. Aborted

results verification
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/401.bzip2/run/run_base_test_i386-m32-gcc42-nn.0001


 * .mis

bash-3.00$ cat dryer.jpg.out.mis 0008: Compressed data 1434910 bytes in length Compressed data 1434894 bytes in length ^ 0013: Compressed data 1127895 bytes in length Compressed data 1127995 bytes in length ^ 0018: Compressed data 1067335 bytes in length Compressed data 1067427 bytes in length ^

run manually
each benchmark has a command line file: run_base_test_amd64-m64-gcc41-nn.0000/speccmds.cmd

sphinx_livepretend_base.amd64-m64-gcc41-nn ctlfile. args.an4

403.gcc ~/opt/spec_cpu2006/benchspec/CPU2006/403.gcc/run/run_base_test_amd64-m64-gcc41-nn.0000] [yourAccount@tux322]cat speccmds.cmd -C /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/403.gcc/run/run_base_test_amd64-m64-gcc41-nn.0000 -o cccp.out -e cccp.err ../run_base_test_amd64-m64-gcc41-nn.0000/gcc_base.amd64-m64-gcc41-nn cccp.i -o cccp.s


 * 1) ignore -C etc. focus on the final part!

gdb -args ../run_base_test_amd64-m64-gcc41-nn.0000/gcc_base.amd64-m64-gcc41-nn cccp.i -o cccp.s

formal text
We use ROSE's identityTranslator. Like any other ROSE-based translators, it is designed to automatically detect input source file types and invoke the right EDG capability.

We do test ROSE on a subset of SPEC CPU2006 as part of our daily regression test. ROSE can fully support 10 out 12 spec C benchmarks (compiling them and having verified execution results). The 10 C benchmars are perlbench, bzip2, mcf, milc, gobmk, hmmer, sjeng, libquantum, lbm, and sphinx3. gcc and h264ref have some minor issues (mostly related to ROSE's handling of macro calls) but we are working on this.

The way we compile SPEC using ROSE is: 1) install (via make install) ROSE to a directory, like ROSE_INS (specified via configure --prefix=ROSE_INS).

2) set up the environment for using the installed ROSE, identityTranslator is located under ROSE_INS/bin PATH=$ROSE_INS/bin:$PATH LD_LIBRARY_PATH=$ROSE_INS/lib:$LD_LIBRARY_PATH export PATH LD_LIBRARY_PATH

3) preparing an SPEC configuration file (e.g. spec_installation_path/config/rose.cfg) to use ROSE A set of relevant options in rose.cfg are:

ignore_errors = no output_format = asc, Screen reportable = 0 CC = identityTranslator CXX = identityTranslator FC = identityTranslator COPTIMIZE = -O2 --edg:no_warnings CXXOPTIMIZE = -O2 --edg:no_warnings FOPTIMIZE = -O2 --edg:no_warnings
 * 1) We want to the test to abort on errors and report immediately
 * 1) we want have ascii and table-based output (Screen) for results
 * 1) The result is not intended for official reports to the SPEC organization
 * 1) compilers to compile benchmarks
 * 1) compilation options: turn off ROSE ’s EDG frontend warnings
 * 2) since we are not interesting in fixing the benchmarks

4) finally you can test ROSE on spec

cd $SPEC_CPU2006_INS . ./shrc runspec --config=rose.cfg --tune=base --size=test --iterations=1 --noreportable fp int --action=clobber
 * 1) set up environment to run spec
 * 1) clean up previous object/executable files for all floating point and integer benchmarks

runspec --config=$SPEC_CPU2006_CONFIG --tune=base --size=test --iterations=1 --noreportable \ perlbench bzip2 mcf milc gobmk hmmer sjeng libquantum lbm sphinx3

12 C benchmarks
10/12 passed now

Passing set: runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable perlbench bzip2 mcf milc gobmk hmmer sjeng libquantum lbm sphinx3

perlbench: pass !!! compilation -D_GNUG__ problem bool problem bzip2:    pass !!! line continuation for two empty line problem: compilation but wrong results gcc :               noncompilable rose_x  pending on a bug 347?, Unparse_ExprStmt::unparseEnumDeclStmt SgInitializedName should be moved to located node test case test2009_05.c mcf :     pass!!! milc:     pass!!!

gobmk :   pass!!! file path issue include/liberty.h vs. engine/liberty.h  rose_out.c engine/rose_out.c hmmer :    pass!!! EDG frontend failure: implicit C function decl sjeng :   pass!!! libquantum:pass!!! h264ref:            noncompilable rose_x  recursive macro replace after macro expansion:

lbm :      pass!!! sphinx3:   pass!!! __FILE__ macro expansion problem. file name vs. full path+filename

runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable sphinx3

/home/yourAccount/opt/spec_cpu2006/bin/specinvoke -d /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482.sp hinx3/run/run_base_test_i386-m32-gcc42-nn.0000 -e speccmds.err -o speccmds.stdout -f speccmds.cmd -C /home/yourAccount/opt/spec_cpu2006/bin/specinvoke -E -d /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482 .sphinx3/run/run_base_test_i386-m32-gcc42-nn.0000 -c 1 -e compare.err -o compare.stdout -f compare.c md

/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482.sphinx3/run/run_base_test_i386-m32-gcc42-nn.0 000/an4.log.mis Error: 1x482.sphinx3
 * Miscompare of an4.log; for details see

7 C++ benchmarks
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable namd dealII soplex povray omnetpp astar xalancbmk