Implementing Normal Order Evaluation with Common Lisp Macros

Section 1.1.5 of SICP introduces the concept of normal order evaluation. Lisp traditionally evaluates all the arguments before applying functions, but with normal order evaluation this is reversed. The functions are expanded out as far as possible before any of the arguments are evaluated. It’s the Evil Universe version of Lisp!!

Now, when I’m confronted with problems having to do with normal order evaluation… for some reason my mind switches off. All of my Lisp experience is with the standard applicative order evaluation… and that’s the way I think. I have a really hard time imagining how normal order evaluation would break things out differently.

(And that’s the thing about SICP. It’s written for guys that scored higher on the SAT than I did… guys that actually paid attention in math class and did their homework. And it’s a tough book even for them– and they have the benefit of peers and TA’s and professors. And just because you can skirt by problems like Exercise 1.5, it doesn’t mean you’re through. This stuff will come back to haunt you in 1.20 and probably even more!)

So there I am having a pity party for myself: I’m a dummy. I have intellectual limits and weaknesses. Google doesn’t return much useful stuff on this topic. I don’t have a com-sci degree. I missed my chance to go to MIT and be somebody…. And then the light bulb goes off: Common Lisp pretty much comes with a normal order evaluator built in. They’re called macros! I can use Common Lisp as a sort of substitute professor to help me practice with normal order evaluation– and Common Lisp will even help me by checking the expansions that I’ve done by hand!

So I rewrite the Scheme example from SICP into Common Lisp macros:

CL-USER> (defmacro square (x)
`(* ,x ,x))
SQUARE
CL-USER> (defmacro sum-of-squares (x y)
`(+ (square ,x) (square ,y)))
SUM-OF-SQUARES
CL-USER> (defmacro f (a)
`(sum-of-squares (+ ,a 1) (* ,a 2)))
F
CL-USER> (macroexpand ‘(f 5))
(+ (SQUARE (+ 5 1)) (SQUARE (* 5 2)))
T

Oops. We’re close here, but we’re still not getting the same expansion as the book. For some reason we’re not expanding out the SQUARE macro. It must be because the reader waits until we have a macro name an the far left side before it expands one.  Maybe we can make it work by using a series of macroexpand-1 commands:

CL-USER> (macroexpand-1 ‘(f 5))
(SUM-OF-SQUARES (+ 5 1) (* 5 2))
T
CL-USER> (macroexpand-1 ‘(SUM-OF-SQUARES (+ 5 1) (* 5 2)))
(+ (SQUARE (+ 5 1)) (SQUARE (* 5 2)))
T
CL-USER> (macroexpand-1 ‘(+ (SQUARE (+ 5 1)) (SQUARE (* 5 2))))
(+ (SQUARE (+ 5 1)) (SQUARE (* 5 2)))
NIL
CL-USER> (macroexpand-1 ‘(SQUARE (+ 5 1)))
(* (+ 5 1) (+ 5 1))
T
CL-USER> (macroexpand-1 ‘(SQUARE (* 5 2)))
(* (* 5 2) (* 5 2))
T
CL-USER> (+ (* (+ 5 1) (+ 5 1)) (* (* 5 2) (* 5 2)))
136

Okay, we can force it work… and then cut and paste the results together to get the same expansion that the text comes up with. But I hate cutting and pasting in Emacs. Let’s rewrite the sum-of-squares macro so that it does all of that work for me:

CL-USER> (defmacro sum-of-squares (x y)
(list ‘+
(macroexpand-1 `(square ,x))
(macroexpand-1 `(square ,y))))
SUM-OF-SQUARES
CL-USER> (macroexpand ‘(f 5))
(+ (* (+ 5 1) (+ 5 1)) (* (* 5 2) (* 5 2)))
T

There’s probably a more elegant way to pull that off, but we get the job done. So now we not only have a little practice with macros and normal order evaluation, but we have techniques at our disposal that allow us to switch back and forth from normal order to applicative order evaluation whenever we choose. In what other language can you do that?!

(To see these techniques applied to  SICP exercises 1.5 and 1.20, please see my study notes in this text file.)

SICP Section 1-1: Just lulling you into a false sense of security…

SICP doesn’t read like a typical programming book. It’s a lot more like math. Real math, now… I’m not talking about some sort of crummy calculus course.

It brings back memories. I was once one of four students at a small college taking Foundations of Mathematics. The teacher was a pretty old guy and I was encouraged to go ahead and take the course with him because he wouldn’t be around much longer. I remember talking about a question he had up on the board. “Well… it’s like this, you know? This and this go like this… and this is what’s happening, you know?” “No, we don’t know,” he says to me and the class.

I don’t think I’ve used the phrase “you know” in conversation since.

The thing about the subsections of a chapter in SICP… I can read over them and have no idea what the point is. I can reread a section and then sorta get a better idea. But when I do the exercises, then it’s like, “oh… that’s what they were trying to tell me!”

(Oops. It looks like I still need to work on over-using the word “like.”)

My learning style is really a trial-and-error kind of thing. I mentally propose hypotheses… and then test them until I get closer and closer to understanding what’s going on. I’m like a human version of the sqrt function from section 1.1.7. But the bad thing about that kind of black-box mess-with-it-until-it-works approach is that I can alter code to do new things without even understanding it. It wasn’t until I solved exercise 1.8 that I understood how that sqrt function really worked! And that was even after messing with it heavily in Exercise 1.7….

There was nothing truly mind-bending yet in this chapter, but it is interesting to know the difference between applicative-order evaluation and normal-order evaluation. I can see how that might have a major impact on certain algorithms. A few other tid bits included the fact that you can put an expression in place of a function name– an if statement, for example. (I’d never seen that before.) This could allow for some pretty concise code in certain situations. The block structure demonstrated in section 1.1.8 is similarly expressive, and can be implemented similarly in Common Lisp with the labels special form.

All in all, this first little taste of SICP is not all that fun. If not for the large amounts of praise this book has received, I probably wouldn’t suffer through its exercises. My work is here if anyone’s interested in comparing notes. I did all of my work in Common Lisp and the only real problem I ran across was that the sqrt function would crash slime badly when I ran it with large values and I’m not sure why.

Because Java Programmers can Suck, too

I started out as a VBA programmer. This means I was looked down upon by the VB6 programmers… who in turn were looked down upon by Java programmers…. Ah, Java was so cool then.

Yeah, it stunk being a VBA guy. You always picked up these projects by some guy who was long gone. You had to add a feature or fix something and you just couldn’t figure out what was going on. Crap was being stored in global variables and all the work was being done on the event triggers of the GUI. What a nightmare! It’s a pain because the state of the program could appear to change randomly… and it was very hard to figure out by trial and error what it was that was breaking things. This lead to bug reports from clients that you couldn’t reproduce… and you ended up looking really incompetant.  (Now I showcase my incompetance with blog entries like this.  There– I said it before a commentor could say it for me!)

Well, Java programmers generally have more class than that. (The language forces them too. Heh.) But there are ways of introducing such CodeMunging in all languages, however fastidious they might be. It’s worse, though, when it’s introduced by a catchy Gang of Four design pattern that makes you sound cool to the uninitiated. But if you thought you could just cut and paste this stuff and automatically be a better coder, then I guess you get what you deserve. Yep, you too can bring all the joys of a crappy VBA debugging nightmare into just about any popular programming language: just get ahold of the “singleton pattern” and start applying it indiscriminantly and you can have all the pain of a classic VBA global variable munge-fest– even in a language that outlaws global variables outright!

I used a singleton once. Once. After first being annoyed that it wasn’t completely clear how to implement one in my language, I released one into the wild of a production system. “There should only be one of these objects in existence ever– therefore this MUST be my chance to finally apply this real live design pattern!” A classic hammer-looking-for-a-nail scenario.  After letting it go, I had a distinct sinking feeling in my stomach. Somehow the experience wasn’t all that fulfilling. I moved on to other things and tried to avoid such pedantic NDD (Nerd-driven development) in live systems where other people would surely come after me and hate my guts for such pointless smarty-pantsness.

But now I know why the technique didn’t stay in my tool-kit. Sure, it ultimately didn’t pass my personal smell test… but could I articulate why? “In some cases they can make testing difficult and hide problems with your design,” says Google’s Dion Almaer.  (Google has gone so far as to release a Singleton Detector to help people identify and remove Singletons from production code.)  Yeah… that’s it.  I knew there was a reason that my coder-sense was tingling….

In the mean time, if you aren’t an expert on Design Patterns… then watch out for questions about them in job interviews. Talking about them without understanding them is a good way to make yourself look stupid.  A lot of those com-sci kids pushed out by the Java Schools nowadays use Singleton to un-OOP their OOP code.  Don’t let a potential employer get you confused with them….

Beyond Data-Driven Design: Are your abstractions “Hairy at Both Ends”?

Going into chapter two of Paradigms of Artificial Intelligence Programming, we get a deceptively simple program to experiment with. The code for this chapter is located in the file simple.lisp, but note that the file does not include the mappend function from section 1.7. (And you thought that “mappend” was just a clever play on words!!)

The whole purpose of this chapter is to compare two approaches to programming. One translates a problem into code following a “quick and easy path” that appears to be the “simplest” but locks your solution into hard coded assumptions. The other approach uses data to describe the problem at a higher, more abstract level. The functions that “do the work” can be attached to varying data sets… and therefore can solve what, on the surface, appear to be different problems. Also, he demonstrates how the same data can be used for different programs. He promises to use this latter, more abstract approach throughout the book.  He calls it a “Rule-based” approach, and says that it’s more work for small problems, but that it minimizes the amount of the solution that’s “written directly in Lisp.”

Norvig’s treatment of the chapter is different because he emphasizes dynamism and code reuse on all sides of the problem. I typically think in terms of data driven design being a way to allow for infinite customization within a single clearly specified problem. Somehow, Norvig’s abstractions are “hairy” at both ends… allowing for an order of magnitude more flexibility. I do some of the things he demonstrates already. For instance, he uses a list to store data and even includes extra non-functional fluff to make it more readable. Then he makes the Lisp equivalent of property accessors in order to allow the program to be self commenting and easier to modify later. But there’s something about the way he thinks that’s foreign to me… or at least, does not come naturally. He doesn’t quite make such a hard separation between code and data that my experiences with OOP Frameworks and Relational Databases have drilled into me.

The chapter really should be no big deal, but my attempts at exercises 2.1 and 2.2 blew up in my face. The first one wasn’t too bad, but I was ignorant of the fact that setf could be used with lexically scoped variables– and also that setf returns the value. Not knowing those two things can lead to some pretty ugly Lisp code, so I’m glad I’ve learned them. The second exercise I completely misunderstood. I don’t know why. Maybe it has something to do with the fact that his “rewrite” function is named in a counter-intuitive way. I even completely missed it after glancing at the answer. Why is it that when Norvig asks me to do something pretty simple I completely blank out?

At this point I should either keep on moving and start on the next chapter… or I should go back an reread Chapter 2 and try to soak up everything he said without missing anything. I can’t tell if I’m overreacting or not. At any rate, Norvig’s writing is fairly dense and requires a great deal of attention. You can’t just skim it and expect to get much out of it.

Closures, Objects, Forklifts, and the quest for truth

My first post on Closures has so far netted 2,235 hits. My second one took in 1,346 while my CodeMunger post has picked up a similar 1,338. The majority of this traffic has come in from Reddit with a smaller chunk coming in from YCombinator’s Reddit-like “Startup News” page.

I wrote these posts because the subject matter was interesting to me. I really had no idea that they’d be read this extensively. (At the time I wrote them I was planning on increasing my readership by doing a few more Emacs articles in order to pick up an extra 10 or 20 hits a day from Google!) The response was largely positive, due in no small part to the timing of Joel’s recent piece on the general “dribble of morbid, meaningless, thoughtless comments” that blog entries so often provoke. There was one guy, however, that was so angered by my overall stupidity (and the fact that I’ve introduced a lot of noise into the search engines) that he needed to go into graphic detail about the violent things he wanted to do to me. (Dude… I’m not the one that posted the article on Reddit. Sheesh.) I have felt pretty lame for posting my ideas to the web anonymously– it’s rather cowardly– but given my unanticipated infamy I think I’ll continue to hang back for a while.

At the same time, traffic such as this does not necessarily equate to readers. It certainly doesn’t mean thoughtful attention. Even Sean Ross’s balanced and well stated post seems to miss the things that I think are key to the discussion:

1) The entire point of the closure/OOP articles is to explore what Paul Graham meant when he wrote on page 2 of ANSI Common Lisp, “With macros, closures, and run-time typing, Lisp transcends object oriented programming.” So yes, I’m writing more from a explorative standpoint. I haven’t yet come to a conclusion about what I think about these things– but these first insights and ah-ha moments that I’m getting by pursuing this line of thought have been more fun than any other programming exercise I’ve ever undertaken.

2) And no, it’s not about closures vs. CLOS or anything like that: closures and objects appear to have an unusual tao-like relationship. And OOP itself is such a slippery concept that the idea that there is “one true implementation/approach” is… well… unrealistic. (And it’s pretty cool that you could, if you wanted to, implement in Lisp whatever vision of OOP you prefer. You don’t currently have the freedom to go the other way in the current reigning OOP languages.)

3) My 7 years of professional life have been predicated on the assumption that OOP should be the fundamental to all that you do in your programming labors. Some really smart people have done just fine without OOP… and see no need to incorporate it into their designs. How is it that folks like that are solving problems with closures and so forth? Are they imitating OOP techniques when they solve real problems? Or is it the other way around? Do advanced OOP programs tend toward having a veneer covering a hacked-out typeless stew of ugliness that belies the core premises built into the design of their languages?  What’s going on here?

And this brings me to Miklos Hollender’s quiet plea for sanity. Am I missing something important in my current quest for beauty and truth? Yes our tools might suck and our languages might hobble us… but shouldn’t I find a zen like contentment in my professional life in spite of all this? My answer to this is two part. One: if you are a professional programmer and you lack a fluency in the core concepts of real computer science, you ought to be doing something to remedy your ignorance. (And if ignorance doesn’t make you feel stupid, then there’s not much anyone can do for you.) Two: normal people in any other line of work would be angry if they were forced to do a job with markedly inferior tools. Programmers should not be content to carry water with pails and a yoke when forklifts and fifty gallon drums are available. While indeed there are many zen-like pleasures to be had in the simple things of life, maintaining badly architected code is not one of them. (At the same time, Miklos probably has a good point that meeting our clients’ needs with tools that are reasonably suitable for the task may not be “fulfilling”… but there’s no reason it can’t be a pretty cool job that we programmers execute well in spite of some of these “academic” issues.)

Nevertheless… there are questions that seem to have gotten lost in the curriculum and tool-kits and language wars. I, for one, would like to look into them. Of course… the next time I post something that difficult and controversial that I can expect to get page views in the thousands… well… I just might take the time to clean it up a little more. (And maybe avoid such shameless hyperbole, too.)

SICP can wait… I’m taking on PAIP

A lot of people are recommending Structure and Interpretation of Computer Programs, but I just can’t bring myself to dive into it. I’ve got a lot of blood sweat and tears wrapped up in Common Lisp and Emacs and I dread moving to another milieu. I really like Peter Norvig’s teaching style, though… and I’m so excited about Paradigms of Artificial Intelligence Programming. Boardgame AI is one of those topics that I’ve always thought of as being way beyond my abilities. I just can’t even think of how to even brute-force my way through such a task. My default trial-and-error code-munging style is completely inadequate there…. (It’s sort of like having to learn how to read music in order to become a better musician. There’s just something here that I need to learn.)

So my deductive reasoning (ie debugging) skills are moderately strong, but my advanced analytical skills are pretty pathetic. Maybe I’m not ready for PAIP…. How can I know? Well… I’ll just start anyway and if I end up having to take a detour then so be it.

His first problem set seems deceptively simple to me. The questions are very simple to state, but to get through them, you have to be fluent in recursion, recursion in trees, and even in getting Common Lisp to do your recursion for you. These are techniques that I really agonized over when I was working through Shapiro. It’s nice to see that I’ve internalized the them. I didn’t have to think about some of the problems and a couple of them worked on the first try.

The very first problem had a couple of snags that caused me to waste a lot of time. One was how to deal with symbols that had commas in them. Of course, you can sandwich anything you want between a couple of pipe characters to allow commas to be inside your symbol names. But how do you clean them out? After several failed searches and not being able to find anything relevant in Common Lisp the Language, I ended up finding what I needed at that always useful Common Lisp Cookbook. My solution may not be beautiful, but it gets the job done without being overly munge-tacular.

Another thing I did was come up with a way to recurse backwards over a lisp. There’s no way that my (reverse (rest (reverse list)))) approach can be at all efficient, but again… it’s very difficult to know what they call that function in the Common Lisp culture.

If anyone else is working through this classic programming text, please drop by and compare notes with me. Maybe we can put together an informal study group or something.  My code for the Chapter 1 problem set is here.

Seibel Admits that Lisp is not the Answer

“Most software shops would probably get more bang for their buck from adopting techniques that allow groups of people to work together most effectively than they would from switching to a better programming language, even Lisp.”  – Peter Seibel

For those worried that I might “drink the Kool-Aide” in regards to the many bits of enticing Lisp propaganda that stalk the web, I offer you this deceptively titled blog post.

Oh, and Peter Seibel does have a blog now.  So far he’s put several interesting essays on the topic of Brook’s Law there and even an article on how to write a book, among other things….

Enjoy!

Closures + Lambda = The key to OOP in Lisp

As Truls Becken remarked in a comment on a previous post, “when it comes to encapsulation, the very foundation is that all state must be accessible only from inside the object’s methods.” On the one hand, it was loads of fun for us to be able to add methods to our “obi” on the fly at run-time. But on the other hand… those methods would have no clue about the state of the object. It seemed like a fatal blow to my approach. Maybe closures weren’t the key to home-grown object orientation in Lisp after all….

I puzzled over this for a while and tried to think about how I could solve this. I’ll never know if I could have figured it out on my own or not because I happened to pick up Peter Norvig’s Paradigms of Artifical Intelligence Programming last night. Chapter thirteen is all about objects and it just so happens that closures are the key to making things happen. Unlike Paul Graham, who began his OOP example from ANSI Common Lisp with inheritance (and multiple inheritance!), Norvig emphasizes encapsulation (via closures) as the cornerstone to his OOP design. If you ever wondered about the phrase “Lambda the Ultimate,” well… this is one example that illustrates why lambda really is the ultimate… uh… thing.

Here is a proof of concept of from the REPL prompt demonstrating that we’ve solved Truls’ problem:

CL-USER> (setf example (create-obi))
#<COMPILED-CLOSURE CREATE-OBI-1>
CL-USER> (setf setter (funcall example ‘set-size ‘() ‘()))
#<COMPILED-CLOSURE CREATE-OBI-1-1>
CL-USER> (funcall setter ‘huge)
HUGE
CL-USER> (setf shower (funcall example ‘show-size ‘() ‘()))
#<COMPILED-CLOSURE CREATE-OBI-1-2>
CL-USER> (funcall shower)
The size is HUGE
NIL
CL-USER> (funcall setter ‘extra-large)
EXTRA-LARGE
CL-USER> (funcall shower)
The size is EXTRA-LARGE

As you can see, the variables referenced in the set-size and show-size methods are the same… and they are protected from general access by the closure.

And here is my current code file. You’ll see there that I’ve been experimenting with reflection and a recursive approach to evaluating long chains of object references. Nothing earth-shattering, though. Yeah, I know… I still need to add class definitions and inheritance into it. And I know that I’ve got at least three conflicting strands of thought emerging there right now and that a lot will have to change in the coming iterations. If you really want to see how to do things right, go read Graham and Norvig.

But what I really wanted to say about all of this is that I’ve been puzzling over some of Paul Graham’s remarks for some time. How is it that closures can be so powerful? When I wrote my last post on closures and OOP… it had just come to me in a flash of insight that you could do stuff with closures. (It wasn’t really written to change the world or tell people how they should do things… and I was shocked that it was even read at all, much less taken seriously by anybody.)

And macros. Yeah… I could see that you can define looping constructs and so forth… and that such things couldn’t be done with a mere function. But my question always was, when will I ever want to do this? I hear lots and lots of people on the web droning on about their lisp-illumination, but I don’t see so much practical advice on how to get illuminated and what you can do with your new powers once you have them. I know what the Blub paradox is… but… how do you get out of it? Well for me it began when I realized that I could implement my own vision of OOP in Lisp if I wanted to. (That’s something that’s impossible in my professional toolset, now. And as a result, I think of that as a sort of thing that only ‘real’ programmers with hairy chests can ever even think about futzing with.) And as I first started experimenting, I saw that there was going to end up being a lot of repetitive code to get classes to work right. And then it sunk in… that I could write some code to write that code for me. And it would of course be macros that I would use to do that. And I could make things as elaborate as I wanted and all the work would be done at compile time… my constructs would hardly impact performance at run time.

And that leads inexorably to a third Ah-ha moment. With lambda, closures, and macros… I can build not just my own implementation of OOP… but I can build anything. I finally see what Alan Kay meant when he said, “Lisp isn’t a language, it’s a building material.” Lisp is amazing not because of what the Paul Grahams of the world do with it, but because it allows average programmers to solve problems that they thought were only solvable by geniuses.

So anyway… if you want to get illuminated… what do you do? Stay away from the web a bit. A lot of it will just confuse you unnecessarily. Stick with the very best books on Lisp… and tackle the hardest problems you can think of. You know, the one’s you’ve always felt were above your abilities. It will be like starting to learn programming all over again. But the sinking feeling in your stomach that you get at that realization will be offset by the fact that programming will again be as fun as it was when you were first learning.

If you are bored with your current set of tools, languages, and projects… that’s a sign that it’s time for you to start learning Lisp.

—————————————

Related Articles:

Closures + Hash Tables = As Much OOP as You’ll Ever Need

Closures, Objects, Forklifts, and the quest for truth

Dark Secrets of Emacs Word Wrapping

This one is for those that want to prevent Emacs from getting pwned by advanced modern applications like… Notepad.  Heh.

“MS-DOG” users that have been “brain damaged” by Windows generally think of text editors as… well… a text editor. They like Notepad because it’s fast, easy, and it doesn’t do anything to your files behind the scenes like just about every other frickin’ Windows program.

Now, for just the general “I’m not going to do any serious editing, I just want to rt-click -> Open this text file and scroll around on it to see what’s going on” type of thing, Notepad has this truly tremendous feature called “Word Wrap.” If it’s on, you get soft word wraps added to your text to make it more readable. As is standard in the Notepad way, this does not do anything to your file, it’s just a feature to make it look good on the screen whether you resize it or not. If you turn word-wrap off then you don’t get any wrapping at all– this is good for looking at files that have fixed length columns.

Emacs defaults to a form of wrapping that puts in “soft line breaks” where ever a line goes off the screen. These are marked on either side with little squiggle arrows. This approach has its merits in that you can see all of your file on screen at one time in a mostly sort-of kind-of the way it is sort-of way. To turn this off, type M-x toggle-truncate-lines. Now you will have a text editor that’s almost as good as notepad with “Word Wrap” turned off! This should make the wrapping lines extend off the right side of the screen.  Note that this command makes using the SQL mode much nicer… though sql-ms/isql still screws up your column headings with the timer text: “1> 2> 3> 4>”.

One core NotePad feature has only recently become available to Emacs users via Emacs 22. M-x longlines turns the feature on, after which M-x longlines-mode toggles it on and off.  Longlines mode gives you the easy reading soft word wrapping of NotePad with “Word Wrap” turned on.

Note that, splitting the screen horizontally vertically with C-x 3 seems to give you “Word Wrap Off” huge lines scrolling off the screen whether you want it or not. And no horizontal scroll bar for you, either! The only way to see the text that is off-screen in this case appears to be to scroll by hitting C-x > and/or C-x <. Alternately you can resize the windows a bit by hitting C-x } and/or C-x {. (For that last one you may want to use the repeat command of C-x z and then as many z‘s as you want after that. But don’t blame me if you accidentally shut down Emacs when you try But don’t blame me if you accidentally shut down Emacs when you try this!) If you really have to use the mouse, then you can click on the arrows on either side of the frame/window/whatever.

I, CodeMunger

I came in off the street into an Access/VB/SQL shop after a summer spent reading books about Excel macros. I was disappointed in myself for not being able to grok the O’Reilly book by itself and having to resort to a 21 days book, but functions were a new concept at the time and there was lots to play with. It was distinctly comforting to me then than Basic had not only survived, but was bolted onto the market dominating office suite of the day. (Yeah, I was nostalgic about buying magazines that had type-in programs in them.)

For months I frantically coded during the day and read programming books when I went home. Early on I realized that one of my first assignments was so impossibly designed that the 60+ year old lady that would end up using it would never be able to master it. Very close to the go-live date, I proposed changing the design to use a tree-view control and a streamlined approach… my boss grudgingly agreed and I ended up finishing it on site with the client’s big cheese smoking a stinky cigar while looking over my shoulder.

Later on a comp-sci student came to work with us. He was a very slick middle-eastern guy; always wore nice suits. He did okay around the shop, I guess… but one night I get a call from him to please come over to the university lab. He needed help with a class assignment. It was pretty lame, but I managed to fix it for him in ten minutes or so. It felt weird bailing guys like this out. I was a faker… a charlatan. No training and no education… just a few months job experience.

Fast forward to now. I get to work with guys that have Master’s Degrees. Woo-hoo. But I can’t relate to them, somehow. I’m always flirting with some new idea… and I’m never content unless I have some crazy mad-scientist scheme being developed in some obscure folder on my hard drive. And these other guys… they just never have anything technical they want to talk about. There’s never a technique or idea or tool that they get excited about and have to discuss over lunch. They’re are actually content with the status-quo… which is strange given how annoying and unreliable things can be. And if I go off on some technical tangent… there’s actually someone around that’s going to call me a nerd. Dude: we’re computer programmers. We code for a living. Aren’t we nerds by definition?!

I guess not. We… as a group… are something else entirely. We’re all CodeMungers.

A CodeMunger has no need to master Data Structures, Classic Algorithms, or Regular Expressions. A series of nested for-next loops and a slew of global and local variables are all he needs to calculate anything.

A CodeMunger has no need to really ever design applications or tools. The framework/library of his tool set has solved most of the problems he’s going to come up against… and he generally works in large existing programs anyway. He can learn everything he needs to get the job done by imitating the existing code base. Everything else that could possibly matter has been covered by an article on a site like the CodeProject.

A CodeMunger has no problem with maintaining 50 near-identical forms… each with redundant GUI and data-access code. A CodeMunger is irritated if the forms rely on subroutines that exist in other modules. A CodeMunger is lost if he has to track down a subroutine that’s inside a separate class.

A CodeMunger has no problem maintaining identical or near-identical versions of the same form or report in different applications.

A CodeMunger does not have the influence to convince the team to fix an architectural flaw even if he manages to notice one. The team is lucky to be using any form of source control as it is… and there are no regression tests to speak of. In this environment, he cannot make a case for the business value of eliminating design debt.  The CodeMunger knows that if he trys anything “fancy” that the rest of the team will protest the complexity of such abstraction.

A CodeMunger is content. Even if there are things about his existing tools that annoy him, he accepts them as just being part of the background reality. They’re just the way things are.

Yeah, it was all exiting at first– just to be able to say, “I write database applications” when strangers asked, “so, what do you do?” I remember all of the whining when .Net was released. It was very stressful trying to keep a job post 9/11 while getting up to speed on the new thing. C# 2.0 was a big deal to me because a simple change to the language resulted the elimination of a couple hundred lines of code in a pet project. (Now I study Lisp because I want to experience that again and again.  Expressiveness is the crack cocaine of programming.) The .Net Framework 3.0 was a complete bust, though. I’ve just got no interest in screwing around with a byzantine object model that doesn’t address something I’m trying to do right now– and that may or may not be here three years from now. But now that I have to maintain applications that exist on both COM and on varying flavors of .Net… well… it’s really just not fun anymore. Every time I have to wait for any one of five IDE’s that I use to load up I get bitter. Every time I have to wait for a file dialog I get angry. Every time the application freezes because of some stupid process or useless GUI, my blood pressure goes up that much higher.

But those things are all superficial compared to the pain of being a CodeMunger. Well… not so much being one… but being aware that that’s what I am… and that that’s what I’m surrounded by.

I don’t know what the next step is. I suspect that too many comp-sci departments have a hand in creating this herd of CodeMungers… but Eugene Wallingford’s blog is comforting. They don’t all have to be that way. (Man, I wish I could get into that Compilers course somehow….) Until my options become clearer, there’s always On Lisp, Paradigms of Artificial Intelligence, and the Stucture and Interpretation of Computer Programs to keep me company…. Maybe the world has no use for yet another BiPoler Lisp Programmer, but I feel I must at least try to go beyond the bag of tricks I picked up “on the street.”