This is a set of Forth application benchmarks, in contrast to the
small benchmarks that are usually used for benchmarking Forth systems.

Most of these applications (except cross and vmgen, which are not run
by default) should be portable between Forth systems, but a test with
Gforth (several versions), bigForth 2.1.6, iforth 2.1.2541, and vfxlin
4.05 Alpha 8 [build 0207] showed the following systems to be working:

Benchmark Systems
benchgc            gforth iforth
brainless          gforth
brew               gforth
cd16sim   bigforth gforth iforth vfxlin
fcp       bigforth gforth
lexex              gforth        vfxlin

That's with most systems out of the box, but a little more effort for
Gforth (otherwise gforth-0.7.0 would not have worked for fcp, and the
output would contain warnings).  It may be easy for knowledgeable
users to enable options in the systems to make them work with more
benchmarks, though.


HOW TO RUN THE BENCHMARKS

There is a bash script "run" that makes it easy.  cd into the
appbench-1.0 directory, then say, e.g.:

BENCH="cd16sim lexex" FORTH=vfxlin ./run

and after a while you will see output like:

[0.500 0.500 0.512 ] cd16sim
[4.040 4.048 4.052 ] lexex

These are the user time from three runs of the benchmarks; the results
for one benchmark are sorted to make it easier to see or compute the
median.  The standard output of the benchmarks is suppressed.  To make
it easier to see what's going on, there is also another script "test"
that performs only one run and shows the standard output.  Use it,
e.g., like this:

BENCH="cd16sim lexex" FORTH=vfxlin ./test

In my testing I used the following variants for the FORTH variable:

FORTH="bigforth -d 16M -e "
FORTH="gforth-fast -m 16M ../setup-gforth.fs -e " #default
FORTH="iforth"
FORTH="vfxlin"

The benchmark names possible in BENCH are:

BENCH="benchgc brainless brew cd16sim fcp lexex cross vmgen"

The default is all but cross and vmgen.

You can change the number of runs by setting RUNS:

RUNS=5 BENCH="cd16sim lexex" FORTH=vfxlin ./run

The default number of RUNS is 3.

You can also change the command used for timing by setting TIME, but
that command should produce only a single number on stderr as output.
The default is TIME, and the TIMEFORMAT="%U" per default to report
only user time.


DOWNLOADING AND INSTALLING

You can download this package from
http://www.complang.tuwien.ac.at/forth/appbench.zip

Just unpack it anywhere, then cd into the appbench-1.0 directory and
benchmark away.


ABOUT THE BENCHMARKS

Benchmark       Author          Purpose, Remarks
bench-gc 1.0    Anton Ertl      Garbage Collector
brainless 0.0.2 David Kuehling  Chess, different results for 32-bit and 64-bit systems
brew 0.2.0      Robert Epprecht Evolutionary playground, incorrect on 64-bit
cd16sim v11     Brad Eckert     CPU emulator
cross 0.7.x     Bernd Paysan    Forth cross compiler, Gforth only, short run, unstable
fcp 1.31-64     Ian Osgood      Chess
lexex           Gerry Jackson   Scanner Generator
vmgen 0.7.x     Anton Ertl      Interpreter generator, Gforth-only, short run

If you want to see how the benchmarks are invoked on the Forth level,
look into the file benchstrings.


ACKNOWLEDGMENTS

Thanks to all the authors of the applications that serve here as
benchmark for making them available and portable.


FEEDBACK

If you have any feedback (problem reports, fixes, new benchmarks
etc.), you can contact me by email at
anton@mips.complang.tuwien.ac.at.

Anton Ertl


