A Quick and Dirty Look at Exploratory Programming in Common Lisp

As the uproar subsides over the recent release of Arc, I can’t help but conclude that the folks that are most outraged by the lack of Unicode support would indeed have little interest in the results of Paul Graham’s labors. Their core values and world view are so antithetical to Paul’s, that they simply cannot see the point. You don’t believe that to be true, of course. You just think Paul Graham is stupid. It doesn’t matter that he wrote On Lisp or that guys like Peter Norvig have a few nice things to say about him. You know he’s insane… and somehow, you’re hurt, insulted, and angry all at once. This is understandable, really. Lisp really is sort of an alien technology. The more you use it, the more it infects your brain. You get all wrapped up in what suddenly appears to be the right way of doing things. But this explanation is meaningless to you. You shrug it off and continue to justify your rage.

But at least give me a chance to make one brief analogy. You know how it is with your wife/girlfriend. You want to be nice, so you offer to take her out to eat. You ask her where she wants to go. She says she doesn’t know. You say you’ll take her anywhere she wants. She insists that you pick somewhere that you want to go. You pick a place that you think would be nice– and she immediately says she doesn’t want to go there. You get frustrated and annoyed. Her feelings get hurt because you can’t hide the sharpness in your words… and suddenly you’re stomping off to avoid the litany of complaints that are sure to follow. “You never do x; you always do y; blah blah blah.” As you mope in your computing dungeon, you feel that something significant has transpired. You feel like there was no way you could have won, yet you’re sure you missed something. You, my friend, are now in the dog house indefinitely.

And that’s the truth of it: you live with an “alien.” You look at the same events and view them differently than she does. You think differently. You have different needs. The “alien’s” rationalizations seem incredibly illogical to you… and you just can’t understand it. The same thing is going on between you and Paul. (Not the living together part. You know… the misunderstanding part. It’s an analogy. Stay with me here for a second.) More than likely you’d rather justify your confusion and disappointment. Certainly, sifting through the kibble to find the meat is the last thing on your mind. But I’m going to give you the benefit of the doubt. Perhaps if you had a little more concrete explanation of how Lisp enables exploratory programming… maybe then you would cut Paul some slack. (Just a little, maybe.) Here goes….

Lisp allows you to define symbols that reference atoms and lists. This allows you to create a shorthand for whatever it is that your working on. Below we create a symbol that represents an x-y coordinate and then reference that list in another symbol:

CL-USER> (setf a '(4 2)) 
(4 2) 
CL-USER> (setf b (list 42 a)) 
(42 (4 2))

Now one thing you should remember… that really is a reference to ‘a’ “sitting” in b’s list. If I make a change to ‘a’, then ‘b’ will change, too!

CL-USER> (cadr a) 
CL-USER> (setf (cadr a) 
(4 7) 
(42 (4 7))

I don’t know about you, but in my mainstream OOP language of choice, my tendency is to start with my own Point class in its own code file. Next I’m coding up a seperate Widget class in its own class file. It has an integer property and point property. I code a variety of constructors in both files and code the getters and setters for the properties of each. Then I start feeling guilty about my Widget class because it violates the Law of Demeter, and I want to go back and put a wrapper on the point property so that no user of a Widget object will reach through to the methods and properties on its internal Point object reference.

Notice above, that we could define a getter function with the cadr operater to retrieve the y coordinate of our point. Setf could then be used with that getter function to make an instant setter. So in lisp… once you define a getter, you automatically define a setter, too. This may seem trivial, but as your object model evolves, I sure get tired of mucking around with property definitions in a half dozen files. In Lisp, all of that stuff is optional. You can just jump in and get started. Note also that because lists can reference other lists, you get a “quick and dirty” form of inheritance for free. Sort of. But even better, you don’t have to take any extra effort to write special code for setting up collections and arrays. The list’s hierarchical structure gives that to you for free.

CL-USER> (setf c (list '(x y z) 42 a)) 
((X Y Z) 42 (4 7)) 
CL-USER> (defun tags (x) 
           (car x)) 
CL-USER> (defun weight (x) 
           (cadr x)) 
CL-USER> (defun point (x) 
           (caddr x)) 
CL-USER> (tags c) 
(X Y Z) 
CL-USER> (weight c) 
CL-USER> (point c) 
(4 7)

So above, we demonstrate the creation of property “getter” functions… and also show how one of those properties can be a list of items with practically no extra effort. I know, I know. Strongly typed collections that take advantage of those new fangled generics features are great fun, when you’re prototyping you’re not all that interested in that sort of paranoid defensive restrictiveness. You may not even know that “types” each of your properties should be, yet!

Here we define a symbol that represents a list of our objects. Notice that the list operator is sort of a universal constructor:

CL-USER> (setf d (list (list '(x x) 17 a) 
                       (list '(y z q) 56 a) 
                       (list '(z z z) 73 '(2 1)))) 
(((X X) 17 (4 7)) ((Y Z Q) 56 (4 7)) ((Z Z Z) 73 (2 1)))

We need to instantiate three of our objects and store them in a list? No problem! You just _do_ it…. The dirty thing here is that you don’t have to define any constructors for your objects. Further… you don’t have to have “all” of your object defined or set up in order to test parts of it.

Let’s set up a simple function designed to operate on a list of our objects. (This is a completely arbitrary example.) But wait… we’re not sure what we want to do as we operate on each item. Well… let’s just skip that part and let the caller of the function tell us what to do each step:

CL-USER> (defun do-something (fn &rest args) 
           (funcall fn (car args)) 
           (if (not (null (cdr args))) 
               (apply #'do-something fn (cdr args)))) 

CL-USER> (apply #'do-something #'(lambda (x) (format t ">> ~a~%" (weight x))) 
>> 17 
>> 56 
>> 73

And instead of setting up a variable to hold our list… let’s just use the function with some objects we define on the spur of the moment:

CL-USER> (do-something #'(lambda (x) (format t ">> ~a~%" (weight x))) 
           '((X X) 17 (4 7)) 
           '((Y Z Q) 56 (4 7)) 
           '((Z Z Z) 73 (2 1))) 
>> 17 
>> 56 
>> 73

This may seem weird, but the more you use Lisp, the more natural this becomes. Lists are fundamentally recursive: they’re defined that way, so you tend to write recursive functions to operate on them. Many built in functions will take functions for their arguments, so you tend to do the same thing with your own code. This not only lets you defer some design decisions, but it also lets you build reusable skeleton functions that are broadly applicable. Finally, “rest” parameters give you even more flexibility… and *another* excuse to write recursive functions. It’s trivial to modify a function to utilize the rest parameters, so you end up using it in places you wouldn’t have thought to.

The syntax for what we just did may seem a little hard to keep up with. When do you you use funcall and when do you use apply? When do you use neither and just call the function? It can be frustrating at times, but that’s the sort of thing that motivated some of the changes in the Scheme dialect.

(I know… I could have used a built-in mapping function for the above. But that’s one cool thing about recursion. If you’ve forgotten the syntax or the name of a built-in function, you can generally write a recursive substitute for it in a pinch. This keeps you rolling along even when you are hacking without reference materials or an internet connection handy.)

Anyways, the main thing here is that you don’t have to waste time juggling class files. You don’t waste time writing properties and constructors and overloaded versions of your constructors. You’ll use “getters” as an elementary abstraction barrier of course, but you don’t have to endure constant object instantiation rituals. You just type out the list that represents your object’s state whenever you need a new one. (This means you can write unit tests that are very clean and concise, by the way… and you can even cut and paste them directly into the REPL prompt without near so much fuss and bother as you might have on other platforms.)

Also, you know how annoyed you get with having an IDE reformat your code even when you don’t want it to? Sometimes the strictly enforced “one class to a file rule” can be equally bothersome. I like to group all of my boring “getter” functions together in one place. This helps document the classes’ read/write fields. Also, related methods are kept together even if they belong to different classes. This makes it easier to think about related parts of the program… which leads to better utilities “bubbling” out of the prototyping process. (By the way, the REPL is just plain great. In terms of its significance, I’d say it’s easily up there with Source Control and Unit Tests in terms of how much it impacts your programming. The amount of time developers waste in the compile-crap loop is insane… and it may be that an entire strata of unit tests are necessitated by the lack of a REPL. But that’s a different story.)

Hopefully this little article can give you some insights into some of the “quick and dirty” type things Paul was talking about. This is not advanced voodoo rocket science by any means, but this is stuff you can master fairly quickly as you start programming in the Lisp dialect of your choice.

6 Responses to “A Quick and Dirty Look at Exploratory Programming in Common Lisp”

  1. jartur Says:

    I think Arc is too “common-lispy” for me. I have recently moved to Scheme because I don’t like CL much & Arc is more CL than Scheme.
    It’s just a li’l different attitude towards what one thinks LISP should be like.
    The fun thing here is that Arc is based on Scheme for now, not CL. I really don’t see the point why is this so =\

  2. Anonymous Says:

    Are you defending arc? Arc adds very little to what MzScheme or other Lisp dialects have in terms of exploratory programming. MzScheme supports more than ASCII, this support does not limit one’s ability to hack. I don’t think it is safe to assume that decrying the lack of unicode support automatically means one is a non-Lisp user or one who doesn’t enjoy the merits of exploratory programming.

  3. lispy Says:

    I don’t know. There’s a lot of Lisp hackers that have their own criticisms for Arc… but they don’t tend to bemoan the unicode thing so much.

    At any rate, I’m attempting to shift the focus towards the more practical ideas that were mentioned in the release message. (Though I have to admit, *you* don’t strike me as being much of a Lisp programmer….)

  4. Mark Miller Says:

    Hi Lispy. What you say here reminds me of a post I did last June on my blog, called “Coding Like Writing”. You’ve seen it already, but I thought I’d mention it for others.

    You can do similar things in Smalltalk as you did here in Lisp. I mention this because you say that OOP is a pain in comparison. Yes, the weak OOP languages are a pain to use, but Smalltalk is different. Ironically it’s the granddaddy of OOP languages, but few other OO languages measure up to its sophistication.

    Carrying out the same actions as you put in your post, in Squeak:

    a := {4. 2}.
    b := {42. a}.
    a at: 2 put: 7.

    Result is a = #(4 7), and b = #(42 #(4 7)). Arrays are expressed as “#()”. The {} expressions evaluate their contents and form arrays.

    Going on:

    c := {#(x y z). 42. a}.
    d := {{#(x x). 17. a}.
    {#(y z q). 56. a}.
    {#(z z z). 73. {2. 1}}}.

    “Carrying out the #do-something loop using Array’s do: method, which receives and executes a closure for each item inside itself.”
    d do: [:item | Transcript show: ‘>> ‘, item second asString; cr]

    So you see, OOP can do some powerful things, as long as you’re using a powerful language. There is a Point class in Squeak that’s natively supported, but I wasn’t able to get the same dynamics with it as what you had in Lisp. So I used arrays for points.

    Personally I like writing classes in Squeak. They’re not verbose, and they enable me to write code that’s descriptive. Someone else reviewing my code can see what I’m doing without having to figure out positional arguments, or what functions can deal with which lists. I’d describe Ruby the same way, actually. Because of Squeak’s live, dynamic nature, I can dabble with code and execute it in a linear fashion inside a workspace before committing it to classes.

    I agree that the code you wrote allows you to dabble with ideas, and this helps the creative process, but Lisp isn’t the only place where you can get that experience.

    There have been a lot of bad implementations of OO in weak or bastardized languages, and a lot of bad ideas about it thrown around over the years. Unfortunately this has sullied its usefullness in many people’s eyes.

    I’m familiar with the Unicode issue, because people have complained that Squeak doesn’t support Unicode either. I can see the issue. Unicode is used to internationalize apps., so you can express text in multiple languages. Some character sets, particularly in Asia, require more character bandwidth than what ASCII offers. ASCII gives you 7 bits of bandwidth. Unicode gives you 16. One could work with 16-bit integers to simulate Unicode text, but that’s pretty unweildy. It’s conceivable that one could create Unicode classes in Smalltalk that would work with other classes that normally use ASCII strings. This is because information hiding is used throughout the system. Every class is a black box to code that’s using it. All anyone knows are the messages objects can receive. As long as a Unicode class supported the same messages, no one would know the difference. For Squeak I know it’s not as simple as that, but it’s a start.

    Does Common Lisp support Unicode? I’d be curious to hear how the Lisp community sees the issue. Is anyone thinking about how to support it?

  5. lispy Says:

    Singling out Common Lisp for the examples isn’t meant to be a put down to other good langauges out there; I’m just using the one I’ve got the most experience with…. The contrasting OOP references are drawn from my experiences with a _mainstream_ OOP language… not Small Talk. 😉

  6. Mark Miller Says:

    Part of the reason for my response is in other posts you’ve talked about your negative experience with OO languages. I just wanted to make clear that despite your bad experiences with weak mainstream languages, that doesn’t mean OOP itself gives you that bad experience.

    Being out of it for as long as I have I forget that the developer market is now pretty much dominated by OO languages/environments like C++, Java, and .Net. There is increasing interest in functional languages because of the challenge of concurrency that’s coming down the road. They hold some promise of addressing it. Smalltalk (or any message-passing implementation of OOP) also holds some promise of addressing the problem, though that hasn’t been talked about much from what I’ve seen.

    All I’m saying is I think OOP brings something valuable to the table that’s been missed by most developers–and who can blame them? Most of them are either not shown its true power, or can’t find enough opportunities to get exposed to it, and so settle for something that’s watered down, but popular.

    It deserves reconsideration. Like I said, OOP has become a victim of its own popularity. By virtue of the fact that it’s been misunderstood, I get the feeling people are now looking for an excuse to dump the baby out with the bathwater and go to something totally different because of experiences just like yours, and they probably will when they find the opportunity.

    Anyway, I appreciate what you do here. I guess I feel the need to get back into implementation stuff. That way I could show off the virtues of OOP like you show off the virtues of FP. 🙂 I think they should both be celebrated. I just haven’t found a project yet that matches my skills with my interests.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: