Index of /cmucl/cmucl-build
# or so...
## Recompiling CMUCL is not an exact science... ##
## [Peter Van Eynde, September 1998] ##
# The is the README file of the CMUCL build instructions by Martin
# Cracauer <firstname.lastname@example.org>.
# These build instructions and scripts are placed in the public domain.
# You can use, modify and redistribute them as you like, including
# redistribution in modified form, but these is no warranty of any
# kind and you use them at your own risk.
# To use these build instructions, you need all files from the
# this directory - cmucl-build/
# Single browseable files: http://www.cons.org/cmucl/cmucl-build/
# All in one tarfile: http://www.cons.org/cmucl/cmucl-build.tar.gz
# Before you start, these instructions is just one way to build CMUCL.
# Alternative ways to build CMUCL:
# - the Debian source package has its own self-building infrastructure
# - the tools/ subdirectory of the CMUCL sources contains shellscripts
# to aid in building CMUCL. Douglas T. Crosher uses polished
# versions of these for his own rebuilding. His versions along with
# a README are on
cat << EOF > /dev/null
Basic understanding of the task:
So you made it through the disclaimers, what now? You need to know
what you are going to do. CMUCL is not a normal program that can just
be built from a basic set of tools. Instead, you need an existing
Lisp to compile the new Lisp sources. Only then you are able to use
the C startup code (which compiles halfway like a normal Unix program)
to build a new Lisp image from these files.
- "Existing Lisp" - the Lisp you already have and use as a tool to
compile the new one.
- "to-be-build sources" - the CMUCL source you are going to build a
new Lisp system from
- "C startup code" - the small C program that pushes the CMUCL system
into start position on a Unix system. This C program cannot run any
of the Lisp code by itself. It is just used to put the Lisp
mechanism into place and then let it loose.
The tasks to rebuild CMUCL:
1) You have to build the startup code (the "lisp" Unix binary). To
do so, you use your existing Lisp to generate a header file that
encodes various information from your to-be-build sources into
constants in this header file. They become literal values in both
C and Lisp code.
2) You use your existing Lisp to compile each of the *.lisp files from
the to-be-build sources. You may have to work around bootstrapping
issues when a new file depends on other new files (that are of
course not loaded in your existing Lisp).
3) From your existing Lisp, you generate a bare-bone basic corefile
that is compatible with your newly generated startup code. This is
"kernel.core", which contains just enough of CMUCL to load more
compiled files and write corefiles by itself. The trick here is
that this corefile does not contain any complicated stuff
-especially not the CMUCL compiler- that would in the way of
4) This new kernel.core should be compatible with your new C startup
code (the 'lisp' Unix binary). You use the combination of both to
load in all the other Lisp files (you already compiled them before,
but did not load them into kernel.core) and write a new corefile.
This new corefile is a full Lisp, with compiler. However, as often
with big Lisp systems, you should recompile from that environment
again to make sure all defintions are consistent over the whole
compilation process and no definitions from your original existing
Lisp are used. So...
5) ... you use that new Lisp core with it's up-to-date compiler to
recompile all *.lisp files again. This time, you may also compile
optional parts like PCL/CLOS. Again, you write a kernel.core from
this stage that you will use for clean bootstrapping.
6) Like before, you use kernel.core to load everything you just
(re)compiled and a final core, this time containing the Compiler
(in "polished" form) and if desired optional parts like PCL/CLOS.
7) Paranoid folks like to recompile just again, to be sure that the
new system has no artifact from the old running Lisp. These
instructions also build the very-optional parts like CLX at this
stage. If you want to do that, activate the for loop near the
8) Finally, I polish the image once more with my to-be-compiled.lisp,
which fixes some symbol and package definition artifacts and makes
sure a newly started Lisp with start in package "COMMON-LISP-USER*
Finally, in directory lisp/, you have the './lisp' startup code and
lisp.clore, your final image.
'./lisp -core lisp.clcore' should run that new Lisp.
Doing this thing:
This looks like a shellscript, but you should use it as such only
when you know it successfully runs through.
As long as you are unsure whether it runs through, cut and
paste the non-comment likes from here and drop them into a running
interactive shell (Bourne shell compatible required, no csh).
Before you start:
- These instructions assume that you start from src/lisp/
- These instructions assume that the files that accompany these
instructions are reachable from src/lisp as ../../crabuild/
The directory structure should look like this:
$BASE/src/[other cmucl dirs]
$BASE/fasl-crabuild/lisp # will maybe created
$BASE/fasl-crabuild/tools # will maybe created
$BASE is just to make the point clear, you don't need to set it as an
environment variable. All paths are releative, so you can have as
many of the environments as you like.
- In src/lisp/, symlink the right Config.* file to the name 'Config'
- Adjust path settings in env.lisp and env-simple-raw.lisp if needed
(shouldn't be if you direcorty layout looks like above).
Make some adjustments in the env-* files in crabuild/:
The structure of the env things is like this:
- env-simple.lisp is automatically generated by the crabuild:buildenv
script. You edit env-simple-raw.lisp, the script makes
env-simple.lisp from it. DO NOT EDIT env-simple.lisp, it will be
- env-simple.lisp is meant to contain as much as it can, but it must
be readable by a lisp with just kernel.core. That means, no reader
macros, no (if ...) and nothing else that uses the compiler must be
- Things that must use features not available in kernel.core go into
env.lisp, which reads env-simple.lisp, so there's no need to add
things to both files.
- Of special importance in env*.lisp are '*features*' settings. For
each combination of existing Lisp and to-be-compiled sources, make
sure that in all steps in your first try you use exactly the same
set of '*features*'. That means, for your existing lisp and for all
compiled Lisps, make sure that *features* is the same, no additions,
no omissions. Start the Lisp interactivly and print *features* to
see what you existing Lisp has. Then add or remove features as
needed in env.lisp and env-simple-raw.lisp for the to-be-compiled
- Optional: also see the crabuild:buildenv script and the resulting
env-simple.lisp. The crabuild script is meant to automatically
generate *feature* settings that for sure must be there when
compiling normal sources in non-crosscompiling situations.
Typically, it looks at the platform OS kind, OS version, libc
version and object format and adds *features* for them. If you set
features right, it should add nothing or symbols you already have.
You may have to edit it when you changed sourcefiles to that they do
different things on various *features* or when you compile older
sources than this script is aware of.
- Very optional: at earliest when compiling with the *same* set of
*features* works, experiment with adding or removing *features* for
the to-be-created Lisp. One symbol at a time is highly recommended.
- NOT OPTIONAL: If you never rebuilt CMUCL with itself before, you
should start with an existing Lisp and new CMUCL sources from the
same date and release branch. Recompiling newer sources on older
binaries often causes bootstrapping problems. Use 'cvs -q update
-dP -D 20000524' (or such) for current sources (current binaries
should indicate the date of their sources at startup), for releases
use an appropriate CVS tag. Make sure you don't mix up the HEAD and
RELEASE_* or RELENG_* tags and branches.
- The file CompileCL.howto is Paul's original document, from which
this one is derived. People frequenty get this wrong. Thanks,
- Bootstrap files: if you compile newer sources that your existing
Lisp is, you usually need bootstrap files. Manually inspect the
files crabuild/boot.lisp and load whatever you find appropriate.
Loading bootstrap files that are not needed is usually harmless, but
not e.g. when moving the static space. You don't need any bootstrap
files when your existing Lisp and to-be-build sources are from the
same date and branch.
- The file to-be-compiled.lisp is a file that gets loaded by before
writing the final image, kind of a final polish. Hooks for cleanups
of the image go here. See the file's comments for more
documentation. Usually, you don't need to set this. At least it
doesn't influence your chances of getting Lisp built.
Sorry for the interruption, the explanation was quite long, so here
are the important things you have to check again:
- You have symlinked boot.lisp, created an empty one or one from
concatenating several boot files?
- You created or symlinked src/lisp/Config?
- You reviewed env-simple-raw.lisp and env.lisp for your needs?
- Unless you know what you do, you have a running Lisp and sources
from the same branch and from the same date? Othewise, have you
enabled bootstrap files you migh need in boot.lisp
- You know you have to start the steps below from the directory
OK, here we go. Enter these command in a bource-shell compatible
shell (sh, bash, zsh etc., but not csh or tcsh). You may also just
start this README like a shellscript
'sh ../../crabuild/README 2>&1 | tee log'
and trace its steps later.
# End of NOTES
# be paranoid
# believe me, you don't want these coredumps
ulimit -c 0 || true
# Utility functions. If you execute this file by cut/paste, drop
# these into your interactive shell (must be a Bourne shell clone), so
# that they are available for later shell command lines.
lisp -noinit -eval '(load "../../crabuild/env")' "$@" -eval "(quit)"
./lisp -noinit -eval '(load "../../crabuild/env")' -core "$@" -eval "(quit)"
# Here we go:
if ! [ -f Config ] ; then
echo Cannot find Config. You either forgot to symlink it or 1>&2
echo you are not in the src/lisp directory 1>&2
# generate env-simple.lisp
../../crabuild/buildenv > ../../crabuild/env-simple.lisp
# look for the age of the sources we are going to build, for
../../crabuild/sourcedate > version.tmp
# Remove old fasl files and set up stupid Makefile quirks
test -n "$NOCLEAN" || ../../crabuild/rmcompiled src
if ! test -n "$DO_CLEAN" ; then
# create dummy file
echo 'Map file for lisp version 1' > lisp.nm
# build internals.h
li -eval '(load "crabuild:build-int_h")'
# compile C startup code
li -eval '(load "crabuild:build-int_h")'
# The next step ompiles everything except PCL and builds kernel.core.
# kernel.core does not contain all the newly compiled files, just
# those that are needed to build a new full core from them. This step
# is running in the old Lisp, which can compile the files, but cannot
# load them to build a core. Instead, a small core is being built
# that contains just as much of the newly compiled files as is need to
# load all others and write a real core.
li -eval '(load "crabuild:build-the-subsystems")'
# compile the file containing last fixups, we need it later
# li -eval '(progn (compile-file "../../crabuild/to-be-compiled") (quit))'
# Here we start the new(!) C startup binary (lisp/lisp) that we
# compiled with gmake. We start it with kernel.core, where the
# interal structures are expected to match. This is the first
# critical step, it often fails. See the troubleshooting section at
# the end of this document.
# If the new small Lisp runs, it is able to load all the compiled
# files (compiled from the old Lisp, this Lisp world does not and can
# not compile them itself, this lisp does not have a compiler) and
# build a real lisp.core from them, with compiler and everything, but
# still no PCL.
cat << EOF | ./lisp -noinit -core kernel.core
(pushnew :no-pcl *features*)
test -n "$SHORT" && exit 0
# Move core files around
mv lisp.core tmp.core
mv kernel.core kernel.core.old
# Now rebuild the new compiler with itself.
# This time pcl is compiled.
# If you compile a newer version of the sources than your binary is,
# do it several times by increasing the "1".
for i in 1 ; do
# FIXME - pcl build when doing even number of runs. Do it uneven times.
# remove old compiled files
# Same procedure as before. Use an old image to compile but not
# load all files. Build just a basic loading-capable kernel.core.
li2 tmp.core -eval '(load "../../crabuild/build-the-subsystems")'
# Now use loading-capable core to build a new core.
cat << EOF | ./lisp -noinit -core kernel.core
mv lisp.core tmp.core
# Doing some polish to the final image
./lisp -noinit -core tmp.core -eval '(progn
(load "/home/cracauer/usr_x86-linux/fasl/cralib.lisp") ; do not remove this!
(compile-file "../../crabuild/to-be-compiled" :load t)
(save-lisp "tmp2.clcore" :purify nil))'
./lisp -noinit -core tmp2.clcore -eval '(progn
(save-lisp "lisp.clcore" :purify t))'
# now maybe recompile lesser-know subsystems and load them
# ./lisp -noinit -core lisp.clcore -eval '(load "../../crabuild/env")' -eval '(load "crabuild:build-some-subsystems")'
# li2 lisp.clcore -eval '(load "crabuild:build-full-core")'
# Fixme - probably need polishing again
cat << EOF > /dev/null
So things went wrong, what to do?
- Review *features* as indicated at the beginning of this document.
Make sure they match after loading crabuild/env.lisp for:
- your existing lisp
- ./lisp -core kernel.lisp
- ./lisp -core tmp.core
Features are dropped and added in rather obscrure ways in the build
process. Use env.lisp and env-simple.lisp to adjust them.
- Did you understand the boot.lisp stuff from above? Make sure you
load the ones you need and in case of further trouble none that you
- Make sure env-simple-raw.lisp stays simple. If is used by a Lisp that
has no compiler and no reader macros.
- no #+ or #-
- no #|...|# comments
- no (if ...) ; requires compiler
- no function creation, including no #'(lambda ...)
- Usually, you cannot not load crabuild:boot.lisp from a kernel.core
lisp, this must be done in env.lisp, not env-simple.lisp
- If you still fail, especially with things like "gc lossage",
signal 11/segfault or "object is of the wrong type", nuke the whole
tree you used to build CMUCL and restart. Nuke everything: the
checked-out sources, the target dir, the C code compile dir,
everything, everything, everything. Restart by checking out a fresh
src/ tree. Don't ask, just do it. Yes, you removed everything that
was build, you are sure, blah, blah, blah. Shut up, sit down, nuke
the damn thing and restart. This is what I do and since you're
reading these instructions and seem to follow them, you should do so
too. With the same head-shaking when it works afterwards although
you was so sure you had it the old thing clean, it is just the
feeling I have when it happens to me. And it does happen, did I
mention that rebuilding CMUCL is not an exact science?
You usually should get crosscompilers from the cmucl mailing list or
from the dtc sources in the experimental/ directory of the CMUCL
However, these build instructions contain slightly edited versions of
various crosscompiler by other people (mostly dtc).
cross-simple.lisp, cross.lisp and cross-loadbackend.lisp. My editing
is done so that they work for non-experimental sources and/or don't
require special features.
Basic usage instructions:
- Uncomment the right block in the middle of the file for the kind of
crosscompiler you want, i.e. "Linux to Linux".
- Go to src/lisp, as usual
- Just load env-simple.lisp, env.lisp and then cross.lisp
- It uses the p86/ subdir when present. p86 is the cannonical name
for "highly experimental", if you use it, you should also you
crosscompiler you find at the same place. My copies here are
usually hacked so that they do not require p86 stuff that is not in
the normal CVS tree.
- It will compile the compiler in fasl-cross1/, load it and use it to
compile the world in fasl-cross2/. It also creates
fasl-cross2/lisp/kernel.core, which you are suppoed to be able to do
a worldload with.
- You may have to compile your lisp startup code again, the cross
compiler script will tell you if so.
UPDATE: README.cross is what I currently use, it is a
README-shellscript like this file. Try it, it has proven paths.
Cross fingers (hehe) and do
In certain circumstances you don't have to go through the whole of the
recompiling effort. Examples:
- If you made changes to C code that shifts the positions of symbols, so
that something like this happens:
WARNING: startup-code version (98) different from core version (97).
You may lose big.
interrupt_handle_now: No handler for signal 11?
- You changed relativly harmless Lisp files that don't affect other
files (for example, function body changes or variable settings only)
In these cases, there is a script 'crabuild/redo' that recompiles
changed C files, changed Lisp files and builds the cores anew. You
could usually do ever shorter, but this script already runs for less
than a minute on a modern machine (14 seconds on Gigahertz Pentium 3).
I don't get enough feedback on these instructions. Send them to