[Keine deutsche Version dieses Documents verfuegbar]

Martin Cracauer's WWW Pages

Thread Interfaces for Common Lisp

Intentions of this page:



Existing Interfaces

CLIM

System abstract:
CLIM is the Common Lisp Interface Manager, basically a GUI toolkit for Common Lisp. The specification includes a thread interface with basic functionality.

Thread interface abstract:
The CLIM thread specification differs from the other systems presented here in that it is a layer with only basic functionality, intended to be mapped to one of the thread interfaces shown below. See below for more comments of mine.

Reference manual for thread interface:
Appendix C of the Reference manual discusses the thread interface. A postscript version of the CLIM 2.0 specification is somewhere else on the net.

Franz Allegro Common Lisp (Unix version)

System abstract:
This is a commercial implementation of Common Lisp, mainly emacs-based with an optional graphical development environment, running on Unix workstations (a Linux version that is supposed to be free for non-commercial use is announced to be out around Christmas). Franz provides a Windows-based Common Lisp, which is a completly different implementation. For the Windows system, threads has been announced to be added, but I didn't see a specification so far (didn't ask Franz).

Thread interface abstract:
Allegro has a richer thread interface than other environments. Some features I find important are suspend/resume functionality and you can ask a lock to return it's holding thread (which can also be used for a "test-lock"-like functionality). See below for more comments of mine.

Reference manual for thread interface:
Franz gives out demo versions of their Unix version and those include 'clman', a retrival system for reference manual entries. The demo version includes the full documentation. But please be fair, when you request a Demo CD for the manual only, please tell them what you want it for.

You might try to get manual entries here: http://www.tf-ten.switch.ch/cgi-bin/clman
Use these starting points:

  • Franz CLMAN: "about-multiprocessing"
  • Franz CLMAN: "about-the-sheduler"
  • Franz CLMAN: "about-stack-groups"
  • Harlequin Lispworks

    System abstract:
    This is a commercial implementation of Common Lisp and a graphical development environment, running on Unix workstations. A tool is availiable to run the resulting programs under Windows (named Transducer).

    Thread interface abstract:
    Lispworks thread interface includes the basic functionality, plus a "reset" option. Important for me, it lacks a function to ask a lock for it's holder, if any. It has a concept to assign a thread to a certain listener (that is, the standard streams). See below for more comments of mine.

    Reference manual for thread interface:
    The manual of the thread interface is available online at ftp://ftp.cs.rochester.edu/pub/archives/lisp-standards/multitasking/vendor/.

    Lucid Common Lisp/Harlequin Liquid

    System abstract:
    This is a commercial implementation of Common Lisp and a development environment, running on Unix workstations.

    Thread interface abstract:
    Lucid's thread interface is quite similar to that of Allegro, but adds better defined terminal (user I/O) handling and it has no function to ask a thread who its holder is. See below for more comments of mine.

    Reference manual for thread interface:
    The manual of the thread interface is availiable online at ftp://ftp.cs.rochester.edu/pub/archives/lisp-standards/multitasking/vendor/. The postscript file as I loaded it doesn't allow individual page selection in ghostscript. To fix this, change the first line from
    #!
    to
    #!PS-Adobe-3.0

    Symbolics Genera

    System abstract:
    Genera is the operating system of the Lisp Maschines produced by Symbolics. It is availiable in combination with a LispM CPU emulator for DEC Alpha machines running the OSF/1 Unix clone.

    Thread interface abstract:
    Since the whole operating system is written in Lisp, the thread interface of Genera is very rich. It is unclear for me whether this rich functionality is useful in the context of an other operating system already running. [TODO: more]

    Reference manual for thread interface:
    I don't know of any location where the manual can be seen online. You have quite good chances to get a set of manuals when you ask for it in Usenet or mailinglists. Contact me if you want a photocopy of the thread-specific pages.

    Macintosh Common Lisp

    System abstract:
    MCL is a commercial implementation of Common Lisp with a development environment. It runs on Apple Macintosh computers only (680x0- and PowerPC- based).

    Thread interface abstract:
    Sorry, I don't have access to the thread interface description for now.

    Reference manual for thread interface:
    The demo version ships with example code for stack groups

    Please let me know if you know of a public manual source.

    ECL

    System abstract:
    ECL is a kcl-derivated free Common Lisp implementation (License is GPL), written by Giuseppe Attardi. It is much like GCL (GNU Common Lisp) in that it compiles to C and loads that compiled code into its world.

    Thread interface abstract:
    I quote from the manual: The goals of the present design are to provide a minimal set of facilities, which stand directly on the core of the {\sc Lisp} computational model, and which provide the maximum degree of flexibility to build higher level concurrency and synchronization constructs.

    In a word: This is quite the opposite of my wishes, I want a model with tight binding to a narrowed operating system thread model.

    Reference manual for thread interface:
    To read about the thread interface, get either get the source from ftp://ftp.icsi.berkeley.edu/pub/ai/ecl or ftp://ftp.di.unipi.it/pub/lang/lisp, unpack it and (try to) process the user manual tex files in ecl/src/doc.

    Alternativly, get a preformatted dvi file from ftp://ftp.di.unipi.it/pub/lang/lisp/ecl-man.dvi.gz.

    ILU

    System abstract:
    ILU is an object-oriented distributed environment that is capable of letting modules of different languages talk to each other. Developed at Xerox Parc. The Common Lisp support includes a proposal for a thread interface, written by Bill Janssen. Code is availiable to map the ILU process model to the underlying native mechanism when running on Allegro or Lucid.

    The ILU project somewhat suffers from decreasing interest in Common Lisp. If you are interested in it, give them some feedback.

    Thread interface abstract:
    The ILU process model defines only the basic functionality needed for the ILU project, somewhat oriented on the Lisp implementations it happens to run on. See below for more comments of mine.

    Reference manual for thread interface:
    Get the thread proposal and related source code from ftp://ftp.cs.rochester.edu/pub/archives/lisp-standards/multitasking/pd-code/.

    EuLisp

    System abstract:
    EuLisp is a dialect of Lisp.

    Thread interface abstract:
    The EuLisp standard includes a very basic thread interface. Only starting, waiting and locks without providing any noticable special.

    Reference manual for thread interface:
    The Thread section of the EuLisp manual is at http://www.bath.ac.uk/%7Emasjap/EuLisp/threads.html

    Thanks to Jean-Christophe Pazzaglia for the hint.

    Scheme implementations

    System abstract:
    Scheme is an another Lisp derivate. Since it uses the same basic syntax as Common Lisp, it is probably worth looking at what their interfaces look like. A number fo Scheme implementation have thread interfaces or at least discussed them.

    Thread interface abstract:
    -

    Reference manual for thread interface:
    Look up Guile, esh and a special version of Scheme-48 (I don't know if it is publically availiable). Get a Guile snapshot from ftp://alpha.gnu.ai.mit.edu/gnu/. Documentation is at http://www-swiss.ai.mit.edu/~jaffer/guile_toc.html, thread section (for now) at http://nis-www.lanl.gov/~rosalia/mydocs/guile-programmer_4.html.

    [TODO: Rscheme has threads? What it SIOD who had?]



    More detailed Comments on Interfaces


    Warning

    Anything below this point is still in the works and just a snapshot of a brain in fluid state. Remember, you're reading work in progress.

    Warning

    CLIM
    CLIM specifies only the following calls:

    • Creating and destroying threads. Creating takes a function to be run and an optional name for the thread.
    • Asking the system for a list of all threads and for the ID of the current thread (that is, the one that is calling this query function).
    • Blocking until a "thing" happens, optionally with a timeout. [I think the timeout functionality is important, but I'd like to add a parameter to this call: A function that is run when the timeout elapsed and that will continue the thread when it return non-nil (instead of just continuing).]
    • Functions to give up sheduling intentionally and to run a piece of code without sheduling of other threads.
    • Instead of having individual functions to interrupt or kill another thread than the current, CLIM specifies a function to force another thread to evaluate a function. [I'm not sure how easy this will be to implement. Maybe it's better to have just "kill".]
    • Locking in CLIM is just creating locks and evaluating inside (with-lock-held ...). [I think that is not enough. At least seperate "lock" and "unlock" functions are needed in addition to the progn-like construct.]
    • Additionally, "recursive locks" are locks that can be locked again by the same thread (doing this with a normal lock would block the process). [Mumble. What's this? Isn't it better to have a function that asks whether a lock is already held and by whom, possibly myself? What's better about the CLIM construct?]

    The CLIM specification uses strings to name lock resources. Why? At the first look, Symbols will be faster and more straightforward to use. What did I overlook?

    Lispworks
    The Lispworks thread interface provides the following functionality:

    • Creating and destroying threads.
    • Asking the system for a list of processes. Asking a process for its arrest reasons. The list of arrest reasons is setf -able.
    • You can speficfy the variables that are initially bound in a process. [Examples of use?]
    • Controlling one thread from another includes killing, forcing it to evaluate a function given by the caller of interrupting function and forcing the other thread into the debugger. [Useful, what do others do?]
    • Waiting for conditions is done with an interface for using user-specified fucntions only. There's no default facility to use something like simple MUTEXes (to use the Posix term).
    • In Lispworks, you can have multiple listeners. A thread that is started from one listener has the *standard-output* directed to that listener. [But how can you assign a random listener as output to a thread you started?]
    • They have *important-locks* besides normal locks. [What are they used for?]
    • Lispworks don't have a try-lock function (one that answers whether a lock is held by someone, but does not block the caller). One could use process-unlock, which will fail when the lock is held by another thread. But what if the current process is the holder of the lock and just wants to see whether it already locked it, just to be sure? [I wonder how that is related to the recursive locks in the CLIM model.]
    • Their make-lock function has an optional argument to lock the lock when it is created, so you can avoid an additional call to the locking function.
    • Lispworks has the same process-reset function as Symbolics Genera. I think this can be a very useful feature to avoid creating threads everytime one is needed. Reusing threads (like in "thread pools") is important for accepatable performance in environments that use a single thread only for a small task (like TCP-based servers, especially for protocols like HTTP).
    • Lispworks lacks a function to ask a lock who its holder is. I think such a function is pretty important, mostly to try whether a lock is locked without blocking the calling thread. [I assume it is easy to get that info from a lock's objects, but it is not documented.]

    Allegro
    Allegro's thread interface is different from the above in these areas:

    • While in Lispworks the list of arrest reasons is setf -able, in Allegro you have functions to add and remove arrest reasons. [Is the Lispworks solution more elegant?]
    • [TODO: What about listeners and stdout?]
    • [TODO: What about killing/signaling?]
    • Allegro has lock functions based on simple yes/no variables (MUTEXes in Posix) and process-wait function that evaluates a function. [How time-intensitive is it to simulate simple MUTEXes with the latter?]
    • It has functions to stop and resume a thread.
    • Is has hooks for suspend and resume. [I don't know if this a good idea. Even a simple test for nil takes time.]
    • Like in Lispworks, you can set the initial bindings of a thread.
    • You can ask who locks a lock. That function can also be used as a replacement for a 'try-lock' function (returns nil if lock is not locked). [TODO: Lispworks doesn't have this?]
    • Like others, it has a process-reset function.
    • The function symeval-in-stack-group ca be used to look up a special variable in a certain stack group. [Will have to make up my mind about it...]
    The documentation is very good, with a lot of useful examples.

    ILU
    ILU defines a layer on top of other Common Lisp thread interfaces. Given that, the functionality is rather rich, I assume someone oriented on the implementations used as a base for ILU development.

    • It includes functions to modify the lists of run- and arrest- reasons and, on top of these, suspend and resume.
    • An interrupt function is provided to force a thread to execute a function given by the caller of the interrupt function.
    • Like Allegro, you can ask a lock who its holder is, if any.

    Lucid
    The functionality provided by Lucid is pretty similar to that of Allegro, with the following differences:

    The maunal has a nice entry about multitasking and stream issues in multithreading (page 21-24 in the postscript file linked to above).

    Suitable Interfaces


    Warning

    Anything below this point is even more unpolished than the above paragraphs. Remember, you're reading work in progress.
    Warning

    First, let me describe my current needs for a thread interface to Common Lisp.

    1. It should be relativly easy to implement, to make ports to free implementations of CL likely and to be sure the new interface is easy to support by a layer to an existing Common Lisp thread implementation.

    2. It should have a rather direct mapping to constructs in thread interfaces availiable in OSes. These include POSIX threads, Solaris threads, Win32 threads. All these have different interfaces, but the functionality provided is very similar (and different from what many Common Lisp interfaces provide).

      This should make it easier to use one of these OS thread systems as an underlying construct. To support multiple CPUs with multiple threads it is a requirement to use either the native userlevel thread mechanism or to build an interface to the lightweight processes (LWP).

      Terminology I use here: "Threads" are what is running in the userlevel process, a "leightweight process" is what the operating system kernel shedules, basically, a full process that shares its memory pages with others (rfork(2) call in plan9 and FreeBSD, clone(2) in Linux).

      As far as I can see, no Thread library on an operating system has the neccessary functionality to base a garbage-collected system with its own stack management on. My own plans are to use the rfork() and clone() calls on FreeBSD and Linux and let the Lisp system do the whole synchronisation management, probably taking code from existing userlevel libraries.

      I know, if the whole issue of interfacing with OS threads would be that easy, some of you already did so. But let me take the freedom to work in that direction until I hit the wall :-)

    3. I need a fast threading mechanism.

      My needs for threads include multiplexing network servers and using multiple CPUs for CPU-intensitive tasks. I think these task depends more on efficient thread switching than GUI work (where, pardon my ignorance, the typical situation is a very small number of threads and a clear idea of what the "main" thread is). Therefore, I think some of the existing meachnisms in Lisp systems are not right for my tasks.

    OK, so I want an interface with only basic functionality. What exactly do I want to leave out?

    General Questions

    Where should the output of threads go to? How do we determine which thread will receive keyboard input? Maybe it's useful to introduce a facility like Unix' "controlling terminal", a "controlling listener"? Each Thread can be associated with a Listener, each Listener can have one or more threads. Each Listenere can be applied to the output devices availiable (terminal/network/file streams, GUI windows). Best would be to be able to change the device on the fly and notify a runnign thread of that change.

    Is it useful to specify a "Stack Groups" interface when you define a thread interface anyway? "Stack Groups" accessable by the user allow other useful applications.

    In which Lisps can you save a world with multithreading active? If so, what happens to running stacks? Solving this would in first line require to have a clean solution for the "controlling listener" for each thread. The Listener concept I'm thinking about maps one or more threads to a specific listener and each listener to one controlling stream/window. If its possible to change the controlling device for a listener "on the fly", the whole saving problem will become much easier. Just reassiciate every listener and its threads to a device that is availiable after reloading (i.e. change those listeners that used files that are not longer open or windows that no longer exist). A hook mechanism should be specified to notify a thread that it's output device has been changed and the thread decides whether it is still useful to continue workign with the new device.

    I think I really want a stop/resume functionality. It will be neccessary anyway for most GCs and is probably useful to make threads survive saving and reloading the world.

    Is it true that on most Lisp implementation the smallest availiable sheduling quantum is 1 second. If so, why don't the manuals list it?



    Martin Cracauer <cracauer @ cons.org>
    http://www.cons.org/cracauer/
    Goto the Index of my Lisp Pages.

    ©2007 Martin Cracauer $Id: lisp-threads.html4m,v 1.8 1998/05/27 17:01:08 cracauer Exp $