Results::Help

A foreword

This page tries to give a formal description of how results shown on this website are built. In a word, all viewed results for a given benchmarks are relatives. This is useful for consistancy within few implied mathematical operations (like sums).

Following example should make things clear. Compairing two compilers on two benchmarks; values in columns express the benchmark results :

Benchmark	Compiler 1	Compiler 2
A	10	12
B	1000	998
Total	1010	1010

Tend to shows that both compilers give same performance on selected benchmarks, despite compiler gives a gain 20% execution speed on first benchmark, with a minor performance leak on benchmark B.

If we choose to express each result of a given benchmarks relatively, we obtain :

Benchmark	Compiler 1	Compiler 2
A	1.0	1.2
B	1.002	1.0
Total	2.002	2.2

Now, we can notice the best overall performances of compiler 2.

Following chapters give details on this process.

Terms

Term	CompilerBenchmarks' definition
Host	Represents a physical host.
Benchmark	Particular benchmark test (Dhrystone, Whetstone or whatever)
Compiler & flags	A particular version of a compiler running on a single host with defined flags
Results	Number of distinct benchmarks' result values. Each result refer to the relative performance of the benchmark using a particular context (hard- and software configuration).

Results in database

Raw results' values comes from .txt files (generated by benchmark package) and are stored in database and are shown as relative result values. Duplicates are tested according many fields (host's name, date & time, compiler, compiler version, OS, optimisation flags, etc.) and related results are not inserted in the database.

You can browse posted results here.

Interpreting results

A good starting point is the comp.benchmarks FAQ

Raw results

To the CompilerBenchs' package, each raw benchmark B (Drysthone, Whetstone...) gives a single result R for a on a given C context :

$?$

Basically benchmark's context can be see as a N dimensions' array (of strings or number). Context includes informations like host's and benchmark's names, date & time, compiler, compiler version, OS, optimisation flags, and so on, and a real positive value (the result, R) gives the raw performance of the benchmark B according to that context. For convience I'll reference results (.txt files) from benchmark package as raw results or as raw benchmark.

All raw results match the law "Bigger is better".

Relative results

To provide more flexibily (hopefuly), CompilerBenchmarks uses a particular viewing mechanism based on benchmark relatives results. For a raw benchmark B, two entries E(n) and R(n) are created in database DB (only if informations don't exist yet) :

$?$

So each entry in database presents a result relatively to the miminum raw value for a given benchmark (not context). Database holds entries, and minima gets a performance weight of 1.0.

Relative values are computed hourly, and results aren't shown before.

Results' selection

Portal allows you to select the entries P you'd want to take, by setting the common part S of their contexts.

$?$

What's displayed

The previous results are used. Then, a function D may be applied on them to get a 2D-array that could be easily displayed (using a chart) according to a specific c context item :

$?$

Finally, the average values of (relative) results that hold c in their contexts are shown by the portal, as well as the number of DB entries matching C.

Page updated on 2007-09-26
1 guest(s).