Advanced Codemunging: A Report from the Trenches

A year ago, I began studying Lisp and Emacs. I would ultimately get through about half of SICP and code up a few small Lisp projects that mainly showed how far I was from grasping the point. Having Emacs skills did come in handy a few times, though. Engaging in such focused deliberate practice was difficult: I rarely came up with anything truly innovative or useful, but instead had to constantly face my weak areas and incompetencies– and doing all of this pretty much in public would ensure those weaknesses would become notorious among the illuminated. I wasted some time blogging about programming practices and things gradually culminated into a manifesto of sorts. While I now had enough technical know-how to address some of the biggest problems I’d faced as a career programmer writing simple database apps and web pages, I also had a minor crisis in choosing which language to implement my new vision. Circumstances at work (and a large decrease in free time to experiment in) would soon narrow my options for me, though. This blog post is about what happened next.

I got really lucky in the way things panned out in my career. You’ve probably read where Joel Spolsky was talking about how awful it is to get stuck being an in-house developer. More recently, Paul Graham has gone off on how we’re all supposed to be like the start-up coders that he funds. Really, most of us aren’t *that* talented to begin with… or at least, we’re not willing to live the start-up lifestyle or take on the associated risks. But there is another option that can be almost as satisfying as working in a Graham/Spolsky type setting: be the *only* developer in a company that’s large enough to have interesting problems and that’s looking to cut the cord with as much of their legacy software as they can as they move to a new system.

If you can land that sort of job, you’re probably savvy enough to avoid most of the stupidity that comes with dealing with corporate type development. Because you’ll be they only developer on staff, you won’t have to waste time building consensus with fearful managers and mediocre “day coders”– and face it, they would never have gone along with your insane ideas to begin with. Also, you have a huge business value bearing down on the side of good design: you’re essentially getting paid to eliminate and refactor away all of the crappy development projects that have accrued over the years. The scope of such a change is so big (and everyone else is so busy), you’ll have to come up with big architectural ideas on your own and you probably won’t even have to pitch them to anybody except the other techy type people that are in your (small) IT department. You’ll just have to make the right decisions on your own.

Looking back on the most painful moments of my career, I think I have pinpointed the cause of most of the disasters. Because I needed the experience, I allowed myself to get into positions where I had to maintain code that I had no authority over. Whether out of my own fear or by decree of my immediate manager, I was isolated from the problems I was supposed to solve. I would continually get requirements that were just enough wrong to be useless and would often spend as much time removing them as I did implementing them. The common denominator in most successful projects/modifications were in situations where I rewrote entire sections of code (with or without permission) and where I also took the time (with or without permission) to hash things out with some smart folks lower down on the totem pole– people that actually did the work and people that would end up using the application. Talking to people like that would often uncover the real problem– but then I’d have to “sell” a new solution to managers at one or more companies. I’d often spend more time talking about the issues with management than I did writing code. And its those sorts of interactions that become the fodder for an unlimited number of Dilbert strips. And because of all of the bad requirements, IT departments start to withdraw from their users and create buffers of bureaucracy to protect themselves from getting “burned.” You can let this insanity grind you down… but you can also take advantage of it if you play your cards right.

You’ve got to realize that while management types generally can tell you what to do (and sign your paycheck and stuff), they can’t really help you with the bulk of your development work. Too many of them are going to just tell what kind of buttons to put in the application. “I just need one that has ANSWERS written on it.” Right. If you can just get your foot in the door, you won’t be so dependent on them. Any sort of application that you can roll out is your ticket to finding our how things really should work. Talk to your users at all levels. “Joe Six Pack” knows a heck of a lot about the company and will not waste your time with stupid requirements. Generally speaking, no one asks his opinion: he’ll be grateful for giving him a voice and making his work easier.

Growing up as a geek, you probably played off-beat rpg’s and war games in your basement. In order to run them, you had to have the bulk of the system “in your head”. This type of thing is very similar to a company’s overall business process– and your contacts with people on the ground and your familiarity with the back end data structures will allow you get large portions of a company’s business processes “in your head” as well. (As long as there’s some room left up there what with all the AD&D stuff junking it up.) The trick is to address the process issues in a coherent way: what’s convenient for one person may not be efficient for the whole. It’s this kind of knowledge and savvy that separates you from the code-monkies. People that have a slightly smaller interest in programming will often use the position they gain from this to become de facto controllers of the company. But you’re a geek, so you’ll probably just be glad to have the chance to deal with management as an equal instead of as a flunky. That way, you can get a chance to implement the code designs of your dreams! Party!

So, once you’ve got a decent job where you have the authority to develop code effectively, how do EMACS, Lisp, and SICP impact your day to day work? The main area is in your attitude. Even if you’ve never truly mastered those tools, you end up cultivating a conviction that your program should be dynamic and malleable at all levels. Your abhorrence of repetition and bloat will be cranked up to “11” and you’ll just have this default expectation that your code should magically respond to you no matter what you’re trying to achieve. You’ll also be exposed to enough unusual ideas that you’ll see things you can do back in “blub” that you never would have imagined. You’ll have a wider range of programming styles at your disposal, and your ability to discern various “code smells” will be much sharper.

Back on the job, I did choose a ubiquitous “blub” as the primary language. The lesson of Lisp is that it takes very little to engineer a custom programming language: just a few axioms will get you something reasonably complete in short order as long as you’re able to combine the pieces into more complex concepts. I realized that if I stole somebody else’s evaluator, I had 90% of what it took to make a dynamic language. I embedded the evaluator into an “environment” that could manage the overall state. I then wrote a small deck of crappy string functions that I used to “compile” lines of text into “code objects”. From there it was a piece of cake to write code to “execute” the objects by running a series of operations on the environment variables based on what’s specified in a list of those “code objects”. This was all done with the crappiest of quick-and-dirty programming techniques. Just awful. It didn’t take long, though. And it doesn’t have to be great. It just needs to be “good enough” for now.

Once the core of this (albeit awful) custom language is done, you’re (generally) only about two hours away from being able to implement any given language feature that you want to do. And thanks to a small familiarity with Lisp, I realized that there was a lot of slick ideas that are opened up once a programmer has gains control over what gets evaluated and *when*. Macro-like functionality became possible, among other things…. In practice though, coding a new language feature take me maybe ten times as long as it would to write a comparable function to do the same thing. There are things that can be done to cut the time off of that, and it’ll be fun to someday compare and contrast my ideas with what can already be done with List “out of the box.”

What I ended up using my roll-your-own language for was to build a scripting language for customizing my main application. When the application is run, it reads a script to set up the environment. All of the GUI is bound to the environment. Advanced users can customize 90% of the application’s functionality without recompiling– and as a bonus, I have a way to test and experiment with malfunctioning desktops without having to use the old “it works on my machine” excuse. Not quite as good as a REPL prompt, but good enough. (For now.)

Because I’m a database app developer, 80% of my code is about connecting to database X, prompting the user for some stuff, getting a deck of rows, and maybe doing something for each row in the deck. (I get so sick of seeing the same crappy DAO, ADO.Net, and stored procedure call code over and over. Blech!) It sure would be nice to have a language designed completely around doing exactly that. And the thing is… I do! My custom language has all kinds of features for manipulating blobs of text. My “environment” stores the blobs in a prominent place, and the GUI automatically picks them up, displays them, and knows how to operate them. This may sound crazy at first, but I’ve been able to come up some powerful front end features because of this approach. Features that I never would have thought up emerge naturally and give my users an unprecedented amount of functionality. Dozens of minor problems routinely get addressed all at once with relatively simple abstractions.

For the first time, I am coding against the problem at both ends. The blobs of text that represent the fundamental unit of work in my system are abstractions that I once referred to as being “hairy at both ends” after trying describe something Peter Norvig was doing in an code sample. These abstractions allow the functionality of the program to be extended gracefully– and I’m generally only a “very short time” away from being able to extend those functions for either myself on the code side or my users on the GUI side. No… for the first time, I have an architecture that acknowledges myself as being the primary “user”. Refactoring is now easier than any other alternative– and the business case for it can be made convincingly at every stage of the project. I can code features that make it easier to manage growing collections of business object blobs in my heavily syntactic sugared scripting language. New blobs are automatically picked up by the GUI… and serendipitous GUI features emerge naturally because of the consistency inherent in the abstractions. Each layer of the architecture builds on and magnifies the efforts of the underlying layers. And every line of code I write is impacted by my (admittedly mediocre) understanding of recursion and the closure property.

You see a lot out there about those supposedly super-duper type programmers that are worth ten times as much as their lesser colleagues. While I will not claim to be one of those guys, I do feel at least ten times as effective of a programmer as I was a year ago. At any rate, I love my code base, I enjoy improving it continuously, and I enjoy seeing my users immediately benefit from the wacky ideas I come up with back at my desk. For the first time in a long while, I wake up excited every morning and I actually look forward to going to work. Life is good.

Thanks John, Harold, Jay, Peter and Paul. You guys have given me answers to the core problems I face every day– even as a lowly database app developer. I may not have the privilege of being able to code in a lisp dialect professionally, but at least I can now begin to “Greenspun” with the best of them….

5 Responses to “Advanced Codemunging: A Report from the Trenches”

  1. kib2 Says:

    Thanks for all these posts about the way you’ve learned Lisp, I had to do it myself and I think they may help a lot.

    See you.

  2. giles bowkett Says:

    _You see a lot out there about those supposedly super-duper type programmers that are worth ten times as much as their lesser colleagues. While I will not claim to be one of those guys_

    You ridiculous monkey. Of *course* you’re one of those guys. The tagline of your blog:

    _(notes from an average programmer studying the hard stuff)_

    No surer sign than that of the presence of “one of those guys.” (Or girls.) There’s no such thing as programming geniuses. We’re all average programmers, and some of us work on the easy stuff, and some of us work on the hard stuff. Keep working on the hard stuff long enough, and people will go “wow, you’re a programming genius.” Because they think programming geniuses exist in the first place. The whole point of the idea of a programming genius is that it’s an excuse not to work on the hard stuff. There’s really *no such thing*.

    You’ll see it happen in a couple years, and you’ll go, “Wow. Hunh.”

  3. links for 2008-05-14 « that dismal science Says:

    […] Advanced Codemunging: A Report from the Trenches « Learning Lisp (tags: emacs greenspun lisp programming sicp toread learning norvig) […]

  4. Mark Miller Says:

    I enjoyed this post. You’re a testament to the idea that it can be done. It gives me something to shoot for. While the tagline on my blog doesn’t say it directly, I thought yours summed up my motivation pretty well: An average developer trying to do the hard stuff. I guess what Giles is saying is even the programmers we regard as geniuses now started out where you were. Maybe they just got the same realizations you did faster, but they went through the same process.

    What you’ve described of your journey reminded of something Alan Kay said in “The Early History of Smalltalk”, a paper he wrote years ago:

    “A twentieth century problem is that technology has become too ‘easy’. When it was hard to do anything whether good or bad, enough time was taken so that the result was usually good. Now we can make things almost trivially, especially in software, but most of the designs are trivial as well. This is inverse vandalism: the making of things because you can. Couple this to even less sophisticated buyers and you have generated an exploitation marketplace similar to that set up for teenagers. A counter to this is to generate enormous disatisfaction with one’s designs using the entire history of human art as a standard and goal. Then the trick is to decouple the disatisfaction from self worth–otherwise it is either too depressing or one stops too soon with trivial results.”

    I have a couple questions.

    It sounds like you found a company that was willing to throw away the legacy stuff and start fresh. How did you convince them that you were the one to help them do that? I would imagine with a resume that only had prior work on typical database apps. that you wouldn’t come across very convincingly. Did they have someone who was perceptive interviewing you who could see that despite no prior experience with doing this on your part that, “The Force is strong with this one”?

    You talk about being “the lone developer”. I’ve explored a corrollary of this idea: What if you leave? Who will carry on the work you started? I think you have the right idea about what you’re doing. It sounds like you have advanced the “state of the art” for the company you work for, and that’s definitely a good thing IMO.

    I had a conversation with a friend of mine last year kind of along these lines. He thought the whole exercise of learning more advanced computing through software was futile, because while I may be smart and able to get this stuff, 98% of the programmers out there are totally unaware of what any of this stuff means (commonly referred to as “code monkeys”). He said that “when you leave–and you WILL leave,” they’re not going to understand a wit of what you wrote, and they’re going to just scrap it and use typical run-of-the-mill solutions instead, because that’s what they understand. My question about this is how do you leave a legacy behind so that “when you leave” there will be people to replace you who will continue the work you started? That’s a big question I have, and I think it’s critical for strategies like yours to succeed. I wish you much success! 🙂

  5. DBlanchard Says:

    Great post. I just discovered your blog and will dig into it tonight when I get off work. Looks like some great gems here.

    I worry about the same “when you leave” problem at my job. I work pretty independently in a variety of languages on a variety of projects. I love the freedom and flexibility this gives me, but it also scares me. I can’t expect anyone else to pick up all the work I currently have when I leave. I can tell you though, that coding for the lowest common denominator is not the answer.

    Code well and demonstrably expect others to code well. Simply expecting more of people won’t get results. They need to know you expect greatness, and to see that you are willing to work with them to do it.

    Seeing how things have been left here after others’ departures, I now only expect that certain people will pick up certain parts of my workload and code base upon my departure. I also have come to accept that someone refactoring my code or even rewriting my apps isn’t bad. It isn’t my ideal, but hopefully they’ll improve my algorithms if not my code, learn something from my code, or at least wait till requirements have changed to start in on it. It is good for them to have my model of how something can be done, but my way isn’t the only good and reasonable way. As long as their way still produces solid code that meets requirements, it doesn’t really matter to me.

    By my guess, I’m still many years away from leaving, but I’ve already started making premature, but conscientious decisions about who I would pick to take on which of my projects and have started using them individually as my resources for bouncing ideas off of and after I make a design decision, I let them know, usually in an e-mail what I chose and why. In addition to my regular documentation, which I often create from these e-mails, he or she now has a linear chain of reasoning on the design decisions for the project that I would hand them if I needed to. If they leave before I do, I still have that same chain and can bulk forward it to someone else, or just bundle it into the regular docs as an added resource.

    I’ve also started a lunch group where once a week someone presents a new skill or paradigm they’ve learned in the recent past. This has varied from lesser-known keyboard shortcuts, to AHK scripting demos, to better ways to talk to people (“Hey, are you OK today?” is very pragmatically different from “What’s wrong with you today?” even though they’re semantically very close).

    Aside from the obvious benefits of learning new things and networking, this lets me see who else is interested in learning for the sake of learning and what their interests are. These are the people I would want to pick up the more interesting and challenging things that I leave behind. They also turn out to be really cool people to eat lunch with.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: