-----
 
FAQ
Benchmarks
Credits
Documentation
Download
Hemlock
Home
Install
News
Platforms
Ports
Projects
Search
Support
 Benchmarks: measuring CMUCL performance
There are lies, damned lies and benchmarks.
-- after Benjamin Disraeli

Here are the results of some performance benchmarking tests run on a number of different Common Lisp implementations. The tests run include

  • the well-known Gabriel benchmarks
  • hashtable exercising
  •  READ-LINE exercising
  •  mathematical functions: factorial, fibonnaci, ackermann's number
  •  some bignum intensive code from Bruno Haible
  • various aspects of CLOS: time taken to compile and execute DEFCLASS forms which create a class hierarchy, instance creation time, time taken to compile and execute DEFMETHOD forms, execution of method combinations, both with and without :after methods.
  •  various operations on arrays, bitvectors and strings.

Except for the CLOS code, timings do not include compilation time. The garbage collector is run before each test to try to make the timings more repeatable. Remember that the only real benchmark is your application: this code is only representative of real-life programs to a limited extent. For certain targets, we assume that the times reported by GET-INTERNAL-RUN-TIME and GET-INTERNAL-REAL-TIME are accurate.

The code for these performance benchmarks can be downloaded from www.chez.com/emarsden/downloads.

Here is an example of the type of results you can obtain, for x86 and SPARC platforms. To interpret these numbers: the number in the Reference column is a number of seconds taken to execute that test. The other numbers are the relative performance of that implementation, so for example a number of 1.3 means 30% slower than the reference implementation. A number of -1 means that the corresponding test was not run on that implementation, either because it does not support the given feature (certain types of method combination for CLISP, for instance), or because of implementation limits (heap size limit on LispWorks Personal, for example).

,---- Pentium III@1.1GHz, 512MB RAM ---
|
| Benchmark                 Reference   18d   18dsm  CLISP   SBCL  Poplog
| -----------------------------------------------------------------------
| BOYER                          1.50   0.73   0.74   5.61   1.01   3.03
| BROWSE                         0.61   0.74   0.74   2.54   1.26   2.85
| DDerviv                        0.66   0.52   0.53   2.14   1.15   1.62
| Deriv                          1.29   0.50   0.48   2.05   1.10   1.26
| DESTRUCTIVE                    1.05   0.91   0.95   3.17   1.13   3.38
| DIV2-TEST-1                    1.09   0.51   0.51   1.66   1.07   1.29
| DIV2-TEST-2                    0.54   0.52   0.61   1.69   1.09   1.31
| FFT                            0.19   1.00   1.00  37.00   1.21  91.21
| FRPOLY/FIXNUM                  0.54   0.87   0.89   6.85   1.09   3.00
| FRPOLY/BIGNUM                  1.30   0.73   0.74   2.44   1.08   1.39
| FRPOLY/FLOAT                   0.54   0.76   0.78   4.41   1.06   2.11
| PUZZLE                         0.68   0.99   0.99  25.09   9.35  27.93
| TAK                            0.79   0.97   0.97   9.08   1.25   3.10
| CTAK                           0.73   1.00   0.97   6.49   1.05   4.84
| TRTAK                          0.79   0.97   0.97   9.13   1.25   3.11
| TAKL                           1.37   1.01   1.00   7.70   1.27   4.02
| STAK                           0.96   1.02   1.02   7.07   1.08   1.50
| FPRINT                         1.21   0.92   0.82   1.06   2.28   4.17
| TRAVERSE                       2.00   0.94   0.99  11.45   1.50   5.42
| TRIANGLE                       1.70   1.01   1.01  17.58   1.28  23.19
| RICHARDS                       0.53   1.00   0.98  21.26   1.00   6.47
| FACTORIAL                      0.37   0.54   0.57   0.81   1.14   0.62
| FIB                            0.33   1.00   1.00   5.30   1.06   1.18
| RATIOS                         2.24   0.96   0.96   0.12   0.96   1.81
| BIGNUM/ELEM-100-1000           1.06   0.92   0.97   0.08   0.97   2.27
| BIGNUM/ELEM-1000-100           4.99   0.99   0.99   0.05   0.93   2.14
| BIGNUM/ELEM-10000-1            5.19   1.01   1.00   0.05   0.92  12.90
| BIGNUM/PARI-100-10             1.21   0.99   1.02   0.02   0.95   0.04
| BIGNUM/PARI-200-5             14.30   0.99   1.00   0.01   0.92   0.01
| HASH-STRINGS                   1.15   0.84   0.69   3.25   0.99  30.89
| HASH-INTEGERS                  0.44   0.80   0.66   2.75   1.27   6.95
| BOEHM-GC                       6.41   0.47   0.46   4.10   1.12   1.53
| DEFLATE                        1.16   1.00   1.02   6.34   1.52   5.53
| CLOS/defclass                  1.75   0.90   0.92   0.53   2.53   0.06
| CLOS/defmethod                 8.73   0.77   0.71   0.02   2.17   0.00
| CLOS/instantiate               6.42   0.57   0.54   0.96   0.99   3.57
| CLOS/methodcalls               8.17   0.42   0.44   1.01   0.90   3.93
| CLOS/method+after             10.76   0.84   0.77   0.47   2.42   2.01
| CLOS/complex-methods           0.29   1.07   1.14  -1.00   6.17  -1.00
| 1D-ARRAYS                      2.44   0.81   0.79   5.68   1.39  15.75
| 2D-ARRAYS                     11.28   0.99   0.89   3.45   4.39   6.04
| BITVECTORS                     2.01   0.82   0.81   0.60   0.98  -1.00
| FILL-STRINGS                   6.83   0.88   0.82   1.16   3.77  27.38
| fill-strings/adjustable       14.18   0.99   0.87   1.36   1.73 118.49
| BENCH-STRING-CONCAT            5.79   1.88   1.62  -1.00   2.75  -1.00
|
| Reference implementation: CMU Common Lisp 18c
| Impl   18d: CMU Common Lisp 18d-pre, level-1 built 2002-04-08 on melbourne
| Impl 18dsm: CMU Common Lisp 18d-pre-small, level-1 built 2002-04-08 on sydney
| Impl CLISP: CLISP 2.27.2 (released 2001-10-05) (built 3222523548) (memory 3222523674)
| Impl SBCL : SBCL 0.7.1.22 (from debian testing)
| Impl Poplog: Sussex Poplog Common Lisp 2.0
| Linux melbourne 2.4.17 #2 SMP Wed Feb 6 11:35:03 CET 2002 i686 unknown
|  (declaim (optimize (speed 3) (space 0) (safety 1) (debug 0) (compilation-speed 0)))
`----


,---- UltraSPARCIIe at 500MHz, 640MB RAM, SunOS 5.8 ---
|
| Benchmark                 Reference  CMU C  CLISP
| -----------------------------------------------------
| BOYER                          3.98   0.91   8.03
| BROWSE                         1.72   0.91   2.85
| DDerviv                        2.02   0.75   3.21
| Deriv                          3.63   0.81   3.13
| DESTRUCTIVE                    3.11   1.01   4.18
| DIV2-TEST-1                    2.19   0.83   3.92
| DIV2-TEST-2                    1.12   0.82   3.85
| FFT                            0.74   1.03  28.86
| FRPOLY/FIXNUM                  1.87   1.01   7.89
| FRPOLY/BIGNUM                  4.59   1.29   3.07
| FRPOLY/FLOAT                   1.65   0.96   5.68
| PUZZLE                         2.07   0.95  30.62
| CTAK                           2.74   1.01   9.04
| TAK                            1.84   1.00  14.08
| RTAK                           1.84   1.01  13.95
| TAKL                           3.37   1.01  11.63
| STAK                           2.32   1.01   8.87
| FPRINT                         4.17   1.02   1.12
| TRAVERSE                       5.84   0.99  13.74
| TRIANGLE                       5.53   0.86  15.57
| CASCOR                        10.53   0.73  52.81
| RICHARDS                       2.35   0.94  22.46
| FACTORIAL                      1.46   1.48   2.88
| FIB                            0.94   0.99   6.71
| BIGNUM/ELEM-100-1000           2.80   1.24   0.28
| BIGNUM/ELEM-1000-100          10.14   1.19   0.44
| BIGNUM/ELEM-10000-1           11.38   1.35   0.41
| BIGNUM/PARI-100-10             2.76   1.15   0.09
| BIGNUM/PARI-200-5             27.19   1.06   0.05
| READ-LINE                      3.39   1.06   1.19
| HASH-STRINGS                   5.42   1.20   2.19
| HASH-INTEGERS                  1.61   0.76   2.00
| BOEHM-GC                      19.97   0.76   4.14
| CLOS/defclass                  4.78   1.01   0.81
| CLOS/defmethod                27.61   0.89   0.03
| CLOS/instantiate              20.93   0.85   1.28
| CLOS/methodcalls              23.62   1.08   1.94
| CLOS/method+after             33.70   1.07   0.78
| CLOS/complex-methods           1.41   0.92  -1.00
| 1D-ARRAYS                     10.77   0.92   3.51
| 2D-ARRAYS                     56.66   1.40   2.61
| BITVECTORS                     5.35   0.86   0.42
| FILL-STRINGS                  18.88   1.07   0.97
| fill-strings/adjustable       45.09   1.46   1.41
| BENCH-STRING-CONCAT           48.10   0.90  -1.00
|
| Reference implementation: CMU Common Lisp 18c, Built 2000-11-27
| Impl CMU C: CMU Common Lisp 18d-pre, level-1 built 2001-12-12 on liszt
| Impl CLISP: CLISP 2.27.2 (released 2001-10-05) (built on moustacho)
| SunOS eagles 5.8 Generic_108528-10 sun4u sparc SUNW,Sun-Blade-100
`----

Further information

Printable version of this page
CMUCLon

Last modified 2015-10-17 by <webmaster@cmucl.cons.org>
Copyright © 1999-2010 CMUCL Project
Validate links, HTML, stylesheet.