Index of /cmucl/cmucl-build

[ICO]NameLast modifiedSizeDescription

[DIR]Parent Directory  -  
[TXT]00_README08-Jan-2014 01:50 19K 
[TXT]00_README.cross08-Jan-2014 01:50 2.5K 
[TXT]CompileCL.howto19-Jul-2000 17:39 5.8K 
[   ]FEATURES04-Oct-1998 13:18 1.3K 
[   ]TODO15-Aug-2000 15:15 346  
[TXT]boot-a.lisp18-Sep-2000 09:24 2.6K 
[TXT]boot-cpl.lisp18-Sep-2000 09:24 2.9K 
[   ]boot-define-symbol-macro.lisp18-Sep-2000 09:24 353  
[TXT]boot-stream-n-nconditions.lisp18-Sep-2000 09:24 2.6K 
[   ]boot.lisp22-Jan-2003 19:09 551  
[   ]boot2.lisp02-Nov-2002 03:01 252  
[   ]build-core.lisp21-May-2003 23:17 197  
[   ]build-full-core.lisp20-Jul-2000 16:22 105  
[   ]build-int_h.lisp19-Jul-2000 17:39 276  
[   ]build-some-subsystems.lisp30-Nov-2001 00:05 401  
[   ]build-the-subsystems.lisp27-Jul-2001 21:54 426  
[   ]buildenv15-Aug-2000 15:05 308  
[   ]cmucl-bootstrap-streams.lisp03-Oct-1998 23:30 1.7K 
[   ]cmucl-x86-crosscompile.lisp19-Jul-2000 17:39 4.4K 
[   ]cross-loadbackend.lisp14-Jul-2000 12:58 1.9K 
[TXT]cross-simple.lisp19-Jul-2000 12:25 2.0K 
[   ]cross.lisp10-Sep-2003 17:45 5.4K 
[   ]echodirs19-Jul-2000 17:39 88  
[TXT]env-simple-raw.lisp08-Oct-2003 14:41 1.2K 
[   ]env.lisp10-Sep-2003 19:56 476  
[   ]fixworld.lisp20-Jul-2000 16:22 463  
[   ]genfeatures07-Oct-1998 09:08 275  
[   ]install-cmucl01-May-2003 18:24 714  
[   ]killlisp19-Jul-2000 12:25 82  
[   ]makedir15-Aug-2001 22:27 297  
[   ]plainfeatures14-Jul-2000 08:23 60  
[   ]redo01-May-2003 18:24 691  
[TXT]rmcompiled14-May-2002 20:26 907  
[   ]runcross21-May-2003 23:38 732  
[   ]shutup.lisp14-Oct-1998 17:19 84  
[   ]sourcedate01-May-2003 18:24 503  
[   ]to-be-compiled.lisp21-May-2003 23:17 1.5K 

#! /bin/sh
#         or so...

#######################################################################
##                                                                   ##
## Recompiling CMUCL is not an exact science...                      ##
##                                 [Peter Van Eynde, September 1998] ##
##                                                                   ##
#######################################################################

#
# The is the README file of the CMUCL build instructions by Martin
# Cracauer <cracauer@cons.org>.

# These build instructions and scripts are placed in the public domain.
# You can use, modify and redistribute them as you like, including
# redistribution in modified form, but these is no warranty of any
# kind and you use them at your own risk.

# To use these build instructions, you need all files from the
# this directory - cmucl-build/
# Single browseable files: http://www.cons.org/cmucl/cmucl-build/
# All in one tarfile:      http://www.cons.org/cmucl/cmucl-build.tar.gz
#
# Before you start, these instructions is just one way to build CMUCL.
# Alternative ways to build CMUCL:
# - the Debian source package has its own self-building infrastructure
# - the tools/ subdirectory of the CMUCL sources contains shellscripts
#   to aid in building CMUCL.  Douglas T. Crosher uses polished
#   versions of these for his own rebuilding.  His versions along with
#   a README are on 
#http://www2.cons.org:8000/ftp-area/cmucl/experimental/cmucl-dtc.src.tgz   
#http://www2.cons.org:8000/ftp-area/cmucl/experimental/cmucl-dtc.src.thread.tgz

cat << EOF > /dev/null

Basic understanding of the task:
================================

So you made it through the disclaimers, what now? You need to know
what you are going to do.  CMUCL is not a normal program that can just
be built from a basic set of tools.  Instead, you need an existing
Lisp to compile the new Lisp sources.  Only then you are able to use
the C startup code (which compiles halfway like a normal Unix program)
to build a new Lisp image from these files.

Some terminology:

- "Existing Lisp" - the Lisp you already have and use as a tool to
  compile the new one.

- "to-be-build sources" - the CMUCL source you are going to build a
  new Lisp system from

- "C startup code" - the small C program that pushes the CMUCL system
  into start position on a Unix system.  This C program cannot run any
  of the Lisp code by itself.  It is just used to put the Lisp
  mechanism into place and then let it loose.

The tasks to rebuild CMUCL:

1) You have to build the startup code (the "lisp" Unix binary).  To
   do so, you use your existing Lisp to generate a header file that
   encodes various information from your to-be-build sources into
   constants in this header file.  They become literal values in both
   C and Lisp code.

2) You use your existing Lisp to compile each of the *.lisp files from
   the to-be-build sources.  You may have to work around bootstrapping
   issues when a new file depends on other new files (that are of
   course not loaded in your existing Lisp).

3) From your existing Lisp, you generate a bare-bone basic corefile
   that is compatible with your newly generated startup code.  This is
   "kernel.core", which contains just enough of CMUCL to load more
   compiled files and write corefiles by itself.  The trick here is
   that this corefile does not contain any complicated stuff
   -especially not the CMUCL compiler- that would in the way of
   bootstrapping.

4) This new kernel.core should be compatible with your new C startup
   code (the 'lisp' Unix binary).  You use the combination of both to
   load in all the other Lisp files (you already compiled them before,
   but did not load them into kernel.core) and write a new corefile.
   This new corefile is a full Lisp, with compiler.  However, as often
   with big Lisp systems, you should recompile from that environment
   again to make sure all defintions are consistent over the whole
   compilation process and no definitions from your original existing
   Lisp are used.  So...

5) ... you use that new Lisp core with it's up-to-date compiler to
   recompile all *.lisp files again.  This time, you may also compile
   optional parts like PCL/CLOS.  Again, you write a kernel.core from
   this stage that you will use for clean bootstrapping.

6) Like before, you use kernel.core to load everything you just
   (re)compiled and a final core, this time containing the Compiler
   (in "polished" form) and if desired optional parts like PCL/CLOS.

7) Paranoid folks like to recompile just again, to be sure that the
   new system has no artifact from the old running Lisp.  These
   instructions also build the very-optional parts like CLX at this
   stage.  If you want to do that, activate the for loop near the
   end. 

8) Finally, I polish the image once more with my to-be-compiled.lisp,
   which fixes some symbol and package definition artifacts and makes
   sure a newly started Lisp with start in package "COMMON-LISP-USER*

Finally, in directory lisp/, you have the './lisp' startup code and
lisp.clore, your final image.

'./lisp -core lisp.clcore' should run that new Lisp.


Doing this thing:
=================

This looks like a shellscript, but you should use it as such only
when you know it successfully runs through.

As long as you are unsure whether it runs through, cut and
paste the non-comment likes from here and drop them into a running
interactive shell (Bourne shell compatible required, no csh).

Before you start:

- These instructions assume that you start from src/lisp/
- These instructions assume that the files that accompany these
  instructions are reachable from src/lisp as ../../crabuild/

The directory structure should look like this:

  $BASE/crabuild/README
  $BASE/crabuild/rmcompiled
  $BASE/src/lisp/
  $BASE/src/tools/
  $BASE/src/[other cmucl dirs]
  $BASE/fasl-crabuild/lisp     # will maybe created
  $BASE/fasl-crabuild/tools    # will maybe created

$BASE is just to make the point clear, you don't need to set it as an
environment variable.  All paths are releative, so you can have as
many of the environments as you like.

- In src/lisp/, symlink the right Config.* file to the name 'Config'

- Adjust path settings in env.lisp and env-simple-raw.lisp if needed
  (shouldn't be if you direcorty layout looks like above).

Make some adjustments in the env-* files in crabuild/:

  The structure of the env things is like this:
  - env-simple.lisp is automatically generated by the crabuild:buildenv
    script.  You edit env-simple-raw.lisp, the script makes
    env-simple.lisp from it.  DO NOT EDIT env-simple.lisp, it will be
    overwritten. 
  - env-simple.lisp is meant to contain as much as it can, but it must
    be readable by a lisp with just kernel.core.  That means, no reader
    macros, no (if ...) and nothing else that uses the compiler must be
    in env-simple.lisp
  - Things that must use features not available in kernel.core go into
    env.lisp, which reads env-simple.lisp, so there's no need to add
    things to both files.

- Of special importance in env*.lisp are '*features*' settings.  For
  each combination of existing Lisp and to-be-compiled sources, make
  sure that in all steps in your first try you use exactly the same
  set of '*features*'.  That means, for your existing lisp and for all
  compiled Lisps, make sure that *features* is the same, no additions,
  no omissions.  Start the Lisp interactivly and print *features* to
  see what you existing Lisp has.  Then add or remove features as
  needed in env.lisp and env-simple-raw.lisp for the to-be-compiled
  sources. 

- Optional: also see the crabuild:buildenv script and the resulting
  env-simple.lisp.  The crabuild script is meant to automatically
  generate *feature* settings that for sure must be there when
  compiling normal sources in non-crosscompiling situations.
  Typically, it looks at the platform OS kind, OS version, libc
  version and object format and adds *features* for them.  If you set
  features right, it should add nothing or symbols you already have.
  You may have to edit it when you changed sourcefiles to that they do
  different things on various *features* or when you compile older
  sources than this script is aware of.

- Very optional: at earliest when compiling with the *same* set of
  *features* works, experiment with adding or removing *features* for
  the to-be-created Lisp.  One symbol at a time is highly recommended.

- NOT OPTIONAL: If you never rebuilt CMUCL with itself before, you
  should start with an existing Lisp and new CMUCL sources from the
  same date and release branch.  Recompiling newer sources on older
  binaries often causes bootstrapping problems.  Use 'cvs -q update
  -dP -D 20000524' (or such) for current sources (current binaries
  should indicate the date of their sources at startup), for releases
  use an appropriate CVS tag.  Make sure you don't mix up the HEAD and
  RELEASE_* or RELENG_* tags and branches.

- The file CompileCL.howto is Paul's original document, from which
  this one is derived.  People frequenty get this wrong.  Thanks,
  Paul.

- Bootstrap files: if you compile newer sources that your existing
  Lisp is, you usually need bootstrap files.  Manually inspect the
  files crabuild/boot.lisp and load whatever you find appropriate.
  Loading bootstrap files that are not needed is usually harmless, but
  not e.g. when moving the static space.  You don't need any bootstrap
  files when your existing Lisp and to-be-build sources are from the
  same date and branch.

- The file to-be-compiled.lisp is a file that gets loaded by before
  writing the final image, kind of a final polish.  Hooks for cleanups
  of the image go here.  See the file's comments for more
  documentation.  Usually, you don't need to set this.  At least it
  doesn't influence your chances of getting Lisp built.

----------
WAKE UP!!!
----------

Sorry for the interruption, the explanation was quite long, so here
are the important things you have to check again:
- You have symlinked boot.lisp, created an empty one or one from
  concatenating several boot files?
- You created or symlinked src/lisp/Config?
- You reviewed env-simple-raw.lisp and env.lisp for your needs?
- Unless you know what you do, you have a running Lisp and sources
  from the same branch and from the same date? Othewise, have you
  enabled bootstrap files you migh need in boot.lisp
- You know you have to start the steps below from the directory
  src/lisp/ ?

OK, here we go.  Enter these command in a bource-shell compatible
shell (sh, bash, zsh etc., but not csh or tcsh).  You may also just
start this README like a shellscript
  'sh ../../crabuild/README 2>&1 | tee log'
and trace its steps later.

# End of NOTES
EOF

# be paranoid
set -x
set -e
# believe me, you don't want these coredumps
ulimit -c 0 || true

# Utility functions.  If you execute this file by cut/paste, drop
# these into your interactive shell (must be a Bourne shell clone), so
# that they are available for later shell command lines.

li () 
{
  lisp -noinit -eval '(load "../../crabuild/env")' "$@" -eval "(quit)"
}

li2 () 
{
  ./lisp -noinit -eval '(load "../../crabuild/env")' -core "$@" -eval "(quit)"
}

#
# Here we go:
#

if ! [ -f Config ] ; then
	echo Cannot find Config.  You either forgot to symlink it or 1>&2
	echo you are not in the src/lisp directory 1>&2
	exit 1	
fi

#
# generate env-simple.lisp
../../crabuild/buildenv > ../../crabuild/env-simple.lisp
# look for the age of the sources we are going to build, for
# *lisp-implementation-version* 
../../crabuild/sourcedate > version.tmp

# Remove old fasl files and set up stupid Makefile quirks
test -n "$NOCLEAN" || ../../crabuild/rmcompiled src
touch Depends
if ! test -n "$DO_CLEAN" ; then
   gmake clean
   # create dummy file
   echo 'Map file for lisp version 1' > lisp.nm
   # build internals.h
   li -eval '(load "crabuild:build-int_h")'
   # compile C startup code
   touch Depends
else
  li -eval '(load "crabuild:build-int_h")'
fi
gmake depend
gmake all

# The next step ompiles everything except PCL and builds kernel.core.
# kernel.core does not contain all the newly compiled files, just
# those that are needed to build a new full core from them.  This step
# is running in the old Lisp, which can compile the files, but cannot
# load them to build a core.  Instead, a small core is being built
# that contains just as much of the newly compiled files as is need to
# load all others and write a real core.

li -eval '(load "crabuild:build-the-subsystems")'

# compile the file containing last fixups, we need it later
# li -eval '(progn (compile-file "../../crabuild/to-be-compiled") (quit))'

# Here we start the new(!) C startup binary (lisp/lisp) that we
# compiled with gmake.  We start it with kernel.core, where the
# interal structures are expected to match.  This is the first
# critical step, it often fails.  See the troubleshooting section at
# the end of this document.  
#
# If the new small Lisp runs, it is able to load all the compiled
# files (compiled from the old Lisp, this Lisp world does not and can
# not compile them itself, this lisp does not have a compiler) and
# build a real lisp.core from them, with compiler and everything, but
# still no PCL.

cat << EOF | ./lisp -noinit -core kernel.core
(load "../../crabuild/env-simple")
(pushnew :no-pcl *features*)
(load "crabuild:build-core")
`cat version.tmp`
(quit)
EOF

test -n "$SHORT" && exit 0

# Move core files around
mv lisp.core tmp.core
mv kernel.core kernel.core.old

# Now rebuild the new compiler with itself.
# This time pcl is compiled.
#
# If you compile a newer version of the sources than your binary is,
# do it several times by increasing the "1".
for i in 1 ; do

# FIXME - pcl build when doing even number of runs.  Do it uneven times.

	# remove old compiled files
	../../crabuild/rmcompiled src

	# Same procedure as before.  Use an old image to compile but not
	# load all files.  Build just a basic loading-capable kernel.core.
	li2 tmp.core -eval '(load "../../crabuild/build-the-subsystems")'

	# Now use loading-capable core to build a new core.
cat << EOF | ./lisp -noinit -core kernel.core
(load "../../crabuild/env-simple")
(load "crabuild:build-core")
`cat version.tmp`
(quit)
EOF
	mv lisp.core tmp.core
done

# Doing some polish to the final image
./lisp -noinit -core tmp.core -eval '(progn
	(load "/home/cracauer/usr_x86-linux/fasl/cralib.lisp") ; do not remove this!
	(compile-file "../../crabuild/to-be-compiled" :load t)
	(save-lisp "tmp2.clcore" :purify nil))'
./lisp -noinit -core tmp2.clcore -eval '(progn
	(in-package "COMMON-LISP-USER")
	(save-lisp "lisp.clcore" :purify t))'

# now maybe recompile lesser-know subsystems and load them
# ./lisp -noinit -core lisp.clcore -eval '(load "../../crabuild/env")' -eval '(load "crabuild:build-some-subsystems")'
# li2 lisp.clcore -eval '(load "crabuild:build-full-core")'
# Fixme - probably need polishing again

cat << EOF > /dev/null

TROUBLESHOOTING
===============

So things went wrong, what to do?

- Review *features* as indicated at the beginning of this document.
  Make sure they match after loading crabuild/env.lisp for:
  - your existing lisp
  - ./lisp -core kernel.lisp
  - ./lisp -core tmp.core
  Features are dropped and added in rather obscrure ways in the build
  process.  Use env.lisp and env-simple.lisp to adjust them.

- Did you understand the boot.lisp stuff from above?  Make sure you
  load the ones you need and in case of further trouble none that you
  don't need.

- Make sure env-simple-raw.lisp stays simple.  If is used by a Lisp that
  has no compiler and no reader macros.
  - no #+ or #-
  - no #|...|# comments
  - no (if ...) ; requires compiler
  - no function creation, including no #'(lambda ...)
  - Usually, you cannot not load crabuild:boot.lisp from a kernel.core
    lisp, this must be done in env.lisp, not env-simple.lisp

- If you still fail, especially with things like "gc lossage",
  signal 11/segfault or "object is of the wrong type", nuke the whole
  tree you used to build CMUCL and restart.  Nuke everything: the
  checked-out sources, the target dir, the C code compile dir,
  everything, everything, everything.  Restart by checking out a fresh
  src/ tree.  Don't ask, just do it.  Yes, you removed everything that
  was build, you are sure, blah, blah, blah.  Shut up, sit down, nuke
  the damn thing and restart.  This is what I do and since you're
  reading these instructions and seem to follow them, you should do so
  too.  With the same head-shaking when it works afterwards although
  you was so sure you had it the old thing clean, it is just the
  feeling I have when it happens to me.  And it does happen, did I
  mention that rebuilding CMUCL is not an exact science?

CROSSCOMPILING
==============

You usually should get crosscompilers from the cmucl mailing list or
from the dtc sources in the experimental/ directory of the CMUCL
distribution server.

However, these build instructions contain slightly edited versions of
various crosscompiler by other people (mostly dtc).
cross-simple.lisp, cross.lisp and cross-loadbackend.lisp.  My editing
is done so that they work for non-experimental sources and/or don't
require special features.

Basic usage instructions:
- Uncomment the right block in the middle of the file for the kind of
  crosscompiler you want, i.e. "Linux to Linux".
- Go to src/lisp, as usual
- Just load env-simple.lisp, env.lisp and then cross.lisp
- It uses the p86/ subdir when present.  p86 is the cannonical name
  for "highly experimental", if you use it, you should also you
  crosscompiler you find at the same place.  My copies here are
  usually hacked so that they do not require p86 stuff that is not in
  the normal CVS tree.
- It will compile the compiler in fasl-cross1/, load it and use it to
  compile the world in fasl-cross2/.  It also creates
  fasl-cross2/lisp/kernel.core, which you are suppoed to be able to do
  a worldload with.
- You may have to compile your lisp startup code again, the cross
  compiler script will tell you if so.

UPDATE: README.cross is what I currently use, it is a
README-shellscript like this file.  Try it, it has proven paths.
Cross fingers (hehe) and do
../../crabuild/README.cross


JUST RECOMPILING
================

In certain circumstances you don't have to go through the whole of the
recompiling effort.  Examples:

- If you made changes to C code that shifts the positions of symbols, so
  that something like this happens:
  WARNING: startup-code version (98) different from core version (97).
  You may lose big.
  interrupt_handle_now: No handler for signal 11?
- You changed relativly harmless Lisp files that don't affect other
  files (for example, function body changes or variable settings only)

In these cases, there is a script 'crabuild/redo' that recompiles
changed C files, changed Lisp files and builds the cores anew.  You
could usually do ever shorter, but this script already runs for less
than a minute on a modern machine (14 seconds on Gigahertz Pentium 3).

FEEDBACK
========

I don't get enough feedback on these instructions.  Send them to
cracauer@cons.org

Happy Lisping
      Martin

EOF