(when (launch-authorised? president)
  (fuel-missile icbm)
  (open-hatch silo)
  (launch-missile icbm))

Another example of control operator is the very useful and-let* macro, semi-standardised in SRFI-2, tests several expressions in sequence, aborting the computation if any of them evaluates to false, optionally binding variables to the expressions’s resulting values in nested scopes otherwise. If none of the expressions evaluate to false, the body is executed in the resulting lexical environment. Those familiar with Haskell may find it somewhat similar to the Maybe monad.

Actually several of Scheme’s own standard control operators are macros, implemented almost exactly as user-defined macros. A named let can be easily rewritten as a letrec operator:

;; a named let expression
(let collect ((seq (produce-list-of-numbers))
              (even '())
              (odd '()))
  (if (null? seq)
      (cons even odd)
      (let ((num (car seq)))
        (if (even? num)
            (collect (cdr seq) (cons num even) odd)
            (collect (cdr seq) even (cons num odd))))))
 
;; becomes a letrec expression
(letrec ((collect (lambda (seq even odd)
                    (if (null? seq)
                        (cons even odd)
                        (let ((num (car seq)))
                         (if (even? num)
                             (collect (cdr seq) (cons num even) odd)
                             (collect (cdr seq) even (cons num odd))))))))
  (collect (produce-list-of-numbers) '() '()))

Macros are of course not limited to adding new control operators to a language. One can design his own little language for solving specific problems, like stocks trading. Or a data modelling language, a graphics drawing language etc. A language for object-oriented programming such as Meroon can be seamlessly embedded within Scheme. Paul Graham’s On Lisp book on advanced Common Lisp programming has several chapters devoted to macros. In short, the developer becomes almost as powerful as the language designer, and such distinction is blurred. Armed with such power, the developer becomes much more productive and, therefore, happy.

Unfortunately not many programming languages provide good macro facilities. The C preprocessor, for instance, only does textual replacement and is no aware of expressions. Granted, the syntax of the programming languages of the Lisp family is very simple and uniform, making better macros possible. Nevertheless other languages like Haskell and OCaml with more complex syntax also provide such facilities.

I believe I have got the first Scheme application past Apple review into the iTunes App Store. It is yet another Reversi clone, called Reverso. It is a combination of 90% Scheme and 10% Objective-C, written with Gambit-C Scheme. James Long has already shown how to compile Gambit-C for the iPhone, and I started from there. My Scheme code is compiled to C by Gambit and later by GCC to produce native ARM code, bundled in a static library, which is ok with the iPhone SDK license agreement. The Objective-C then calls the library as a pure C library. The Scheme code deals with position evaluation, alpha-beta pruning, transposition tables, move legality, different strategies and so on. The Objective-C code deals with sound, animations, GUI, user preferences, basically everything that calls the iPhone OS API. Reversi was chosen because I like strategy games and it is much more algorithmic than artistic, and I am no artist.

The performance of the code is excellent. I used some Gambit-specific declarations, only fixnum arithmetic, pre-allocated a large heap, and called the garbage-collector every time the user needed to think. The search is not memory intensive, but the transposition tables are. I did not write a specific hash function but relied on Gambit-C’s table type. The boards used during search were retrieved from a pool (the newest Gambit-C has made subu8vector-move! a public API), so they did not put pressure on the garbage-collector. In the end it was a very successful experiment. Developing with Scheme is orders of magnitude more productive than with most other languages. Gambit-C is also one of the best Scheme compilers out there, and made my job a lot easier.

Update: There was a bug in the application that caused it to play poorly. The weak AI was not Gambit’s or Scheme’s fault, but mine. I already sent an updated version to Apple that will play much better. Besides fixing the bug, I am now using Zobrist’s hashing instead of Gambit’s own hash function, that performs poorly for game positions. It is still written in Scheme, though. Now I am waiting for the approval of the update, and to hear further feedback from my users.

Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people.

Now that is a demotivator. But Ruby must be evolving, and that quote may be outdated. So I have thought of a question to the Ruby-lovers, that may help me make up my mind to take the time to learn it. One of the best Scheme macros is, in my opinion, and-let*, from the SRFI-2. How would one write the following in Ruby, a procedure that I wrote a couple of nights ago?

;; Tries to get session from current request
(define (get-session request)
  (and-let* ((cookies (request-cookies request))
             (p (assoc "session_id" cookies))
             (sid-str (cdr p))
             (sid (string->number sid-str))
             ((integer? sid))
             ((exact? sid))
             (sp (assq sid *sessions*)))
    (cdr sp)))

In the and-let* macro, the clauses are evaluated and the variables are bound sequentially, creating new scopes for the next expressions. If any of the forms evaluates to #f (false in Scheme), the computation is aborted, no other expressions are evaluated, and the whole form evaluates to #f. How would I write this in concise Ruby?

• Essentials of Programming Languages, by Daniel P. Friedman, Mitchell Wand and Christopher T. Haynes
• Programming Language Pragmatics, by Michael L. Scott
• Smalltalk-80: The Language and Its Implementation, by Adele Goldberg and David Robson
• Writing Compilers and Interpreters: An Applied Approach Using C++, by Ronald L. Mak
• Modern Compiler Implementation in ML, by Andrew W. Appel
• Lisp in Small Pieces, by Christian Queinnec
• An Incremental Approach to Compiler Construction, by Abdulaziz Ghuloum
• Let’s Build a Compiler, by Jack Crenshaw
• Structure and Interpretation of Computer Programs, by Harold Abelson and Gerald Jay Sussman, with Julie Sussman
• Advanced Compiler Design and Implementation, by Steven Muchnick
• The 90 Minute Scheme to C compiler, by Marc Feeley
• Compilers: Principles, Techniques, and Tools, by Aho, Lam, Sethi and Ulman (the “Dragon Book”)
• Engineering a Compiler, by Keith Cooper and Linda Torczon
• Mike Pall’s guide to the Lua source code
• Anton Ertl’s papers
• David Gregg’s papers
• Michael Franz’s papers
• Garbage Collection: Algorithms for Automatic Dynamic Memory Management, by Richard Jones and Rafael D Lins
• Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp, by Peter Norvig
• The Art of the Interpreter of, the Modularity Complex (Parts Zero, One, and Two), by Guy Lewis Steele, Jr. and Gerald Jay Sussman
• Compiler section on readscheme.org

And anyway, the subject line presupposes that Lisp has not caught on. This is like saying that astrophysics or calculus or brain surgery has not caught on because in relative numbers, there might be more people doing other things. The success of Lisp is not measured in the number of people using it, it’s measured in the utility to those people who do use it. Turning it into C (or C++ or C#) to make it more popular would not be success. In the world’s menu of computer language options, we don’t need them all to be Taco Bell.

I have departed from the traditional approach of implementing a Scheme interpreter in Scheme itself because I wanted to avoid possible confusions between the defined language and the defining language. This has made a lot of the concepts clearer. The evaluator is written in continuation-passing style, which is easily done in Lua because the latter has tail-call optimisations. This way it is easier to reify continuations to first-class values.

I have also added a few primitives to allow me to run some examples, as the factorial and fibonacci functions. More primitives can be easily added if desired.

Ok, I know the World is already sick of Schemes, it only takes a look at this list to know why. But I still think I have to do it, if only for my personal learning. There is no need to bore the World to death with another half-done Scheme. Sometimes I think about features that could make my Scheme useful, though. They are probably never going to be implemented, but here they are:

Full R5RS compliance

The R5RS is a piece of art (R6RS, on the other hand, is a stain, a mess). Of course it does not describe a terribly useful system, but it beautifully describes the core of one. Everything else can then come from there. For more compliance with other systems, the SRFIs can be used as the basis for a standard library.

Bytecode virtual machine

Although Scheme is a dynamic language, there is a lot that can be known about the program before executing it. Lexical variables, never assigned variables, closure and continuation creation etc. This can be processed at compile time, allowing for a simple and fast virtual machine. Interpreting the source code directly is easier but very inefficient for any serious system.

Incremental garbage collection

Stop-and-go garbage collectors are easier to implement, but in some applications the pause is inacceptable. For applications like interactive games, the smooth user experience only can be achieved by an incremental garbage collector. It would be nice if it was a moving collector too, to avoid too much fragmentation of the heap (some naive C programmers think the use of malloc and free can beat a garbage collector in any circustances, until they are hit in the face by heap fragmentation. There is a reason the Apache web server uses memory pools). If I remember correctly the CLR garbage collector is very good and compacts the memory without the use of from- and to-spaces, which is cool, because uses less heap memory. Hmm, while I am dreaming about my ideal garbage collector, make it concurrent too (see below).

Concurrency

These days the hot topic is concurrency. New personal computers with two and four cores sell more and more every day. The programming languages/environments that makes easy for the programmer to use these new cores are going to be big. Erlang is the hottest thing right now in this regard, although Haskell is a strong candidate too. In the mainstream Java has a concurrent VM, but Java uses the old lock/unlock paradigm of other mainstream languages, Clojure is the new language on top of the JVM that leverage that power for the programmer. But although this is all so hot, the only thing that seems to be happening to Scheme is Termite, which as far as I know can only use green threads. My hypotetical Scheme would have a concurrent VM, some Scheme compilation techniques already give a start in that direction. There would be maybe a spawn-continuation form that would take a continuation captured with call/cc and resume it in parallel with the current one, in a M:N model (M continuations spread on N OS threads).

Just-in-time compilation

Well, once dreaming, why not go all the way? One of the nice things about Scheme is the bottom-up, incremental development. Developing at the REPL gives instant feedback and ease debugging. What if it could give the speed of compiled-to-CPU languages as well? Another advantage is hot-swapping code in a running application. This can’t be done with Chicken or Gambit, which are fast, but Scheme-to-C compilers. Larceny, Chez and Ikarus can do this. By using a bytecode VM, the system is portable to wherever there is a C compiler, and by adding JIT support for some architectures it can be made fast too.

Escape continuations

Full, first-class continuations with indefinite extent are a powerful and mind-bending feature, but unfortunately, are usually hard to implement and causes pauses when invoked. Besides them, my hypotetical Scheme would have escape continuations. Instead of completely overwriting the execution stack, they simply unwind it until some recorded point, which can be made very fast. Exceptions would them be implemented as escape continuations rather than full ones.

Easy FFI

I am a long-time Lua user, and it is amazing how they could get the C API so good. Not only it is very easy to add new C/C++ libraries to Lua, it is trivial to embed the Lua interpreter itself in a legacy application, or an application that needs to be mostly in C/C++, such as games. As Scheme, unfortunately, is not one of the World’s favorite languages, the capacity to talk to the outside World is fundamental. And by being embeddable (which is the goal of Guile, but it sucks for Windows), it would be easier to take Scheme to another domains where it is not so popular today.

Well, I guess this is it. Thinking about it, there seems to be no Scheme system today with all these features. Will I have the skills to write something like this after LiSP? I doubt it, only the garbage collector would take me ages. But it would be nice if such thing existed…

Although the focus of Dybvig’s thesis is not on concurrency, I cannot help but think that his compiler could work very well for this too with just minor modifications. If variables (local or free) are never assigned, they could be replaced by their values and several different threads with different execution stacks could run in parallel. If a variable is assigned, the access to its indirection box could be protected by a mutex or similar synchronisation primitive. Of course this would make access to assigned variables even slower, because even variables ever used by only one single thread would need mutex lock/unlock operations. On the other hand, isn’t assignment discouraged in Scheme? The overall execution time would compensate for this, if the rest of the program was parallelised.

But surely I am missing something, because no one ever seems to think Scheme can be a contender in the massive-parallelised future. What bugs me is that I don’t know what it is that I am not taking in consideration.

I am writing a very simple application in Scheme, with Gambit-C Scheme, for helping me trade stocks (nothing fancy, I am still an amateur). This application is going to have a comprehensive set of technical indicators which are calculated based on opening, closing, maximum and minimum stock prices as well as the volume traded, number of operations, number of stocks traded etc. Clearly these indicators must be first-class citizens of this small financial world. Once added to the system, I must know how many I have, what is the name the user should see for each of them and the parameters they take. Besides, its syntax must allow users of the system to add indicators of their own without knowing the inner details of the system.

To accommodate those needs I have come up with a little language for my indicators. For instance, these are the Volume and Force Index indicators:

(indicator volume (x)
  "Volume"
  (volume x))
 
(indicator force-index (x)
  "Force Index"
  (if (> x 1)
      (* (- (closing (- x 1)) (closing x)) (volume x))
      0.0))

The indicator syntax allows me to give the indicator a user-friendly name, and to automatically add it to my indicator list. But there is something more subtle and interesting about them: the helper procedures. The procedures closing and volume are examples of procedures available to indicator writers. They cannot be lexically scoped, because they depend on the current stock being analised and the current sampling frequency (daily, weekly etc.). They must be bound to the appropriate procedures just in the scope where the indicators are being run. In Common Lisp this is easily done with special variables. In Scheme we have the SRFI-39, Parameter objects. But since the parameter objects act syntatically like closures, using them directly would demand that the indicator writer use a cumbersome syntax like:

(indicator force-index (x)
  "Force Index"
  (if (> x 1)
      (* (- ((closing) (- x 1)) ((closing) x)) ((volume) x))
      0.0))

This is very undesirable. The solution, in face of the lack of special variables, is to use a code walker. Due to the unique syntax of Lisp, this whole process can be combined into a simple macro:

(define-macro (indicator name args desc . body)
  (letrec ((helpers '(opening closing maximum minimum
                      volume num-operations num-stocks
                      num-samples))
           (walker (lambda (sexp)
                     (if (pair? sexp)
                         (let ((opr (car sexp))
                               (args (cdr sexp)))
                           (if (memq opr helpers)
                               (cons `(,opr) (map walker args))
                               (cons opr (map walker args))))
                         sexp))))
    `(let ((,name (lambda ,args ,@(map walker body))))
       (set! *indicator-list* (cons (cons ,desc ,name)
                                    *indicator-list*)))))

And that’s it. From now on I need to think only about indicators, and not where to put them, how to supply them helpers, what’s the frequency of their samples etc. They are high-level abstractions which are now part of my language.