“84 months worth of work was reduced to 2 months, and the results were error free.” It’s stories like this that fire our imaginations. Whether it’s Paul Graham fixing bugs while the client is still on the phone or Peter Siebel’s dad finishing a project with only half a budget, we want to know the secret.
And what were some of the attributes of this latest successful project? “I didn’t have to code the changes for each machine; it would create what was needed from the machine specifications…! I didn’t need a new release, all I needed to do was apply my new business rules to the existing system…! This is what development was supposed to do for us.”
I could see some of this. Configuring systems with tables stored in a database or in XML files is not enough. Each installation is different… and if enough are them are different enough that we’re forced into making changes in the actual code, we’re hosed. We get sucked into an endless treadmill of patching, redeploying, gathering more requirements, putting out fires…. Source Control, Unit Testing, Agile techniques, and good coding style all contribute to making this somewhat manageable. But they don’t address the core issues. They will eventually fail us on the interesting problems.
My own little toy project, though fun, is deficient not only because of its amateur technique, but rather because it works against the grain of the language. It did achieve a moderate level of configurability via a human readable configuration language, but it was accomplished in a brute force manner… and extending the new language is not a lot of fun. While we managed to abstract away the essence of the generation definitions, we nevertheless violate the closure principle: we’re frozen at a single “pretty good” level of abstraction. And unlike a true embedded language, my custom language does not benefit so much from the features of the parent language.
How can the success story be recreated? Quoth the hacker, we need “grammars to read and execute specification files….” This, of course, points back to Norvig’s deceptively simple PAIP chapter 2… a theme that sets the tone for his entire book.
To say that we are going to invent custom languages on a problem by problem basis is misleading. We’re going to be extending existing languages in expressive ways– without burning the bridge back to the parent language’s core idiom. As pico explains, “a DSL isn’t really writing a new language, but rather manipulating an existing language to define your problem, or domain, in a more natural form. It’s designing objects and writing methods that isolate the problem and illuminate your business rules.”
We are not “true believers” in any single programming language, but we recognize that some languages are friendlier to our creativity than others. As far as is possible, we will not allow any language to limit our imaginations. And we will code solutions to problems that chafe at the constraints imposed by relational and object-oriented assumptions. Still, the question is not whether or not to write DSL’s or embedded languages, but when.
Learning Lisp 
October 5, 2007 at 9:21 am |
Well I think every structured program is all about creating DSL’s, the only question is the depth and expressiveness.
Procedures, functions and methods are verbs. Classes, objects, variables and constants are nouns. Object attributes are adjectives. Higher-order functions are gerunds (“I like swimming”, “keep going” etc.).
And perhaps this later one is the most important. Somebody pointed out in another blog that “mainstream” Steve.TakeOut(TheGarbage) OO-style is very compatible with the SVO word order of spoken English. Indeed – only that you sound like a Russian general in the bad movies when all you have is plain SVO. To put things with something resembling a style, to say the programming equivalent of English expressions, like, “having said that”, you need higher-order functions i.e. having(said, that) when “said” is a function.
Higher order functions mean the difference between talking to the computer like a coal miner high on weed, or talking to the computer like we do. The question is: do we need to go further? Do we need macros?
Take, for example, the try – catch – close it in finally overhead whenever you process a simple text file. You don’t really need a macro to solve it: you can just have a process_file function that takes a file name as a parameter and a function which it will apply to each line, and return the lines transformed by this function in an array, list, whatever. Or in a generator, something that yields a line at each call. Maybe you can pass two functions, one is mapped and the other as a filter, or even an optional third, which will be a reduce.
And then you got your file processing control structures just right, without macros. In fact using macros may encourage you to write throwaway code right into the with-open-file. Not having macros, using higher-order functions encourages wrapping logic into functions, thinking about reusability.
So I say we can live without macros. As I said above all structured programming is a DSL because at the very least you define new verbs. And yes, that qualifies as a DSL: if I’d change all the verbs in my comment to Hungarian you wouldn’t have the slightest idea what I’m talking about. So if all structured programming is about creating DSL’s the question is how far you want to go. And I think higher-order functions are far enough, we can live without macros. And reason that is important is that however much I like LISP itself, I hate the implementations of it like the plague. The only exception might be newLISP but even that’s weird a bit (every hashtable must be in the global namespace?).
And if we can live without macros, we can use a programming language with libraries for about everything, growing industrial acceptance, .NET and Java bytecode compiler, one that makes fairly sure you’ll be able to read your code 6 months later and strikes a good balance between being conservative on one hand and expressive and flexible on the other hand: Python. I had a long journey of trying many programming languages and finally I realized you have to keep the coolness factor in balance with conservativism i.e. enforced readibility which Ruby, Smalltalk and LISPs lack. The language can be powerful but it must be powerful in clear, obvious ways. So the search is over for me. Power is not everything, things should be in balance, just like in the meat world.
October 5, 2007 at 2:21 pm |
So maybe a sufficiently developed object model is indistinguishable from a DSL? Hmm…. Yeah, it’s all just code we’re talking about… and even an embedded language won’t necessarily look too different from regular code. But macros are not the issue, here. Also, we’re setting aside the question of which language to use in order to focus on generic design strategies. As it is written in the book of SICP, Chapter 2 verse 2:
“We have also obtained a glimpse of another crucial idea about languages and program design. This is the approach of stratified design, the notion that a complex system should be structured as a sequence of levels that are described using a sequence of languages. Each level is constructed by combining parts that are regarded as primitive at that level, and the parts constructed at each level are used as primitives at the next level. The language used at each level of a stratified design has primitives, means of combination, and means of abstraction appropriate to that level of detail.”
There’s much more going on here than what we see in a typical library or object model. You can solve a lot of problems in the usual way, but if you’ve got a really hard problem where that is not good enough, then you might try borrowing techniques from “real” engineers in other fields. Stratified design provides us with a concise way to frame a lot of the things we already intuitively strive for in our industrial languages.
October 5, 2007 at 3:23 pm |
Well if a sufficiently developed object model means functions are objects themselves i.e. one can use verbs in the place of nouns (subject, object) then maybe yes. To be honest I think about programming as applied linguistics, not as applied math, therefore I find the SIC a bit hard to understand. But if I get it right primitives are nouns, means of combination might be a verb and abstaction well, I guess it could be higher-level definition of nouns and verbs. Meaning classes and higher-order functions.
Although it will still remain a programming language. What I find interesting in LISP that it blurs the line between configuration languages and programming languages, and because a domain-specific configuration language is a tree, it can be presented to the end-user in a GUI form. An EMACS user can grow from setting configuration options on the GUI to setting it in a file, then adding some conditions and loops to the file, then wrapping them into functions etc. while in the background he is actually programming in LISP all the time. Fascinating, but I’m not sure that even if a language can blur the line betwen GUI configuration, configuration language and programming language, the meatware at the other can blur it too. I think people are quite distinctly separated into no-configuration, GUI configuration, configuration file, scripter, and programmer types, and it might just be an illusion to try to bring this under one hood.
I’m thinking a lot about it because it kind of pisses me off that analysts and support people who are happy setting 1000 configuration options in huge grid forms are afraid to write a simple loop that would throw all products supplied by a vendor to a purchase order – 10 lines of plain simple code it is in Navision, I don’t get what are they so afraid about. Maybe a DSL would help, but in what ways? If it’s 4 lines instead of 10 that won’t make them more eager to do it if it still feels like programming, with all the parens and expressions. However if it’s something English-like – nah, it is practically proven that’s a leaking abstraction. That won’t work. All we can do is to lover the entry. If Navision was written in Python, I could set an arbitrary number of filters onto a record variable with keyword arguments, reducing 4 line of filtering to one. Instead of looping through the records, the record object would have a method that would accept a function which it would evaluate for all records. And there would be another built-in function to insert records with keyword arguments. So all they’d have to do is to write a function that maps the fields of the product to those of the purchase order line, and then set filters for the product record variable and supply this function. This might be easier, not easier to write, but easier to debug, and therefore could lower the entry. But not too much I’m afraid.
October 5, 2007 at 4:31 pm |
Yeah… that is the other question. How did that hacker create “grammars to read and execute specification files” and what class of problems is that going to be the tool of choice? As I said, I think PAIP goes into detail on that. And from what I’ve heard, you need to grok SICP before you can imbibe PAIP.
And as for your last conundrum there… I think you’re going to see more of that kind of thing emerging in mainstream programming languages– where they get a kind of embedded (or bolted-on?) SQL-like language. (Of course, wasn’t SQL meant to be as English-like as possible so as to be usable by manager-types??)
October 5, 2007 at 11:42 pm |
I think we agree more than we disagree but I still think your assertion that DSLs are always the answer is offbase. What about situations where something like a Rules Engine is called for? What if you’re solving a very, specific well-defined problem? Is it worth it to create a DSL for something as trivial as Tic-Tac-Toe? I’ll come back to this next week but when it comes to using complexity people need to think carefully about their ‘input complexity’.
October 9, 2007 at 1:00 pm |
No, I’m not saying they’re always the answer. SICP Chapter 2 style stratified design may be a good option for really tough problems that have frustrated you with other approaches. Custom “grammers” may be the key to providing a much more elegant and comprehensive solution to configuration issues.
October 10, 2007 at 1:45 pm |
I think what became of SQL proved once and for all that the inherent complexity of programming is not in using a weird syntax, but in turning a vague specification of a problem into an accurate and correct one.
Let me repeat myself: any kind of (sane and non-trivial) programming activity is about writing a DSL because even if you just define a bunch of subroutines, that’s defining a bunch of verbs, which means the language you write in from on kulonbozik from the original one. (“kulonbozik” is “is different” in Hungarian, just as a demonstration 🙂 )
Thus you cannot not write DSL’s. And as every serious person I know accepts that OO done right (custom types, not necessarily wrapped tightly together with methods, they can be more generic) and higher-order functions are both a great idea, the only real question is, I think, whether you actually need anything more, i.e. macros.
I’m thinking about, for example, the using statement in C# which disposes of an open file even if there is an exception. In a language with higher-order functions I can write a function that takes a filename as a parameter, opens the file, calls that function, catches and rethrows all exceptions and closes it in finally. Then in the function I want to process the file in I define an inner function and call this function with it. Not bad, I guess. However it would be nicer if wouldn’t have to type the function name twice. Python should have anonymous functions, not just anonymous expressions.