The Future of Programming

“84 months worth of work was reduced to 2 months, and the results were error free.” It’s stories like this that fire our imaginations. Whether it’s Paul Graham fixing bugs while the client is still on the phone or Peter Siebel’s dad finishing a project with only half a budget, we want to know the secret.

And what were some of the attributes of this  latest successful project? “I didn’t have to code the changes for each machine; it would create what was needed from the machine specifications…! I didn’t need a new release, all I needed to do was apply my new business rules to the existing system…! This is what development was supposed to do for us.”

I could see some of this. Configuring systems with tables stored in a database or in XML files is not enough. Each installation is different… and if enough are them are different enough that we’re forced into making changes in the actual code, we’re hosed. We get sucked into an endless treadmill of patching, redeploying, gathering more requirements, putting out fires…. Source Control, Unit Testing, Agile techniques, and good coding style all contribute to making this somewhat manageable. But they don’t address the core issues. They will eventually fail us on the interesting problems.

My own little toy project, though fun, is deficient not only because of its amateur technique, but rather because it works against the grain of the language. It did achieve a moderate level of configurability via a human readable configuration language, but it was accomplished in a brute force manner… and extending the new language is not a lot of fun. While we managed to abstract away the essence of the generation definitions, we nevertheless violate the closure principle: we’re frozen at a single “pretty good” level of abstraction. And unlike a true embedded language, my custom language does not benefit so much from the features of the parent language.

How can the success story be recreated? Quoth the hacker, we need “grammars to read and execute specification files….” This, of course, points back to Norvig’s deceptively simple PAIP chapter 2… a theme that sets the tone for his entire book.

To say that we are going to invent custom languages on a problem by problem basis is misleading. We’re going to be extending existing languages in expressive ways– without burning the bridge back to the parent language’s core idiom. As pico explains, “a DSL isn’t really writing a new language, but rather manipulating an existing language to define your problem, or domain, in a more natural form. It’s designing objects and writing methods that isolate the problem and illuminate your business rules.”

We are not “true believers” in any single programming language, but we recognize that some languages are friendlier to our creativity than others. As far as is possible, we will not allow any language to limit our imaginations. And we will code solutions to problems that chafe at the constraints imposed by relational and object-oriented assumptions. Still, the question is not whether or not to write DSL’s or embedded languages, but when.

Boilerplate code says a lot about a language…

 I ran across this recently:

“In fact the problem of ‘boilerplate design pattern code’ says nothing about the quality of the language; it says only that the programmer are working at the wrong level of abstraction and are the victims of that all too common sickness: bad design. Picking another language is not the cure for this disease. (More than likely it’s only going to make things worse!)”

This is just plain wrong– except for the part about the programmer working at the wrong level of abstraction and being a victim of “bad design.” The question is, how much of that has been foisted onto the programmer by the design of the language he’s writing in? And if much of it is due to the language, then it’s certainly not off base to go shopping for a new one!

If you’re not clear on this issue, then you need to watch SICP Lecture 3a: Henderson Escher Example. Early in the lecture (00:05:45) he points out the importance of closure in any means of combination in a programming language. (Now this isn’t about “closures”, now… this is a more general mathematical principle.) Anyways, Abelson says, “Closure was the thing that allowed us to start building up complexity…. The things that we make… those things themselves can be combined…. to make more complicated things…. When you look at a means of combination, you should be asking yourself whether things are closed under that means of combination.”

Obviously, it’s much better to be able to make an array of arrays than it is to be restricted to only storing numbers or strings in them. Whenever the principle of closure is violated, you’re going to be limited in your ability to formulate useful abstractions. Just like you lose the ability to express certain ideas cleanly (or even at all) when functions are anything other than first class citizens in your programming language. Programming languages really do have significant differences; they’re not all the same.

After going through his example step by step (and incorporating everything covered so far in the course), he (01:02:37) sums up why the ability to create embedded languages is so important: “That’s the important point: the  difference between merely implementing something in a language and embedding something in a language so that you don’t lose the original power of the language. Lisp is a lousy language for doing any particular problem; what it’s good for is figuring out the right language that you want and embedding that in Lisp. That’s the real power to this approach to design.”

I think a lot of people get the heeby jeebies when they hear Lisp programmers talking about creating new languages to solve problems. They’re thinking, “if people start writing their own macros, how will I understand the code when I have to maintain it?!” Well, yeah. That’s only going to be a problem if you don’t know how to use macroexpand to show what’s going on “under the hood.”

But when you look at what’s going on in the example from the video, you can see that the process of implementing languages to describe what’s going on actually makes the code much easier to understand. The languages are implemented at each level of abstraction. Each language doesn’t “care” how the lower level languages are actually implemented– you can think at each level in terms of that level without being confused by unnecessary detail. And each level can be manipulated with and integrated with all of the usual idioms of the Lisp language itself. It’s not like you’re randomly inventing a new Python or Perl depending on the problem you’re solving. You’re making languages that are fully embedded in Lisp… and there’s nothing keeping you from utilizing other Lisp tools, techniques, and elements with these language layers.

So don’t let all the talk about DSL’s and embedded languages scare you. Expressive code can actually be easier to understand, extend and maintain.

Update 10/2/07:   ”discipline and punish” responds with Programming Languages, DSLs, Static Typing, and the Answer to Life, the Universe and Everything.