Still Average After All These Years

I got the chance to browse one of those mega-sized book stores the other day. I walk in with a chip on my shoulder, daring the place to have a game or a book or a magazine that catches my interest. I just find myself drifting further and further out of the mainstream as “normal” and “regular” things pale in the face of my own private obsessions. The computer book section had shrunk since my last visit. It’s like its slowly dawning on the owners that they don’t make enough money selling them to cover the costs of having to put obsoleted editions into the dumpster every couple of years. I used to could spend hours browsing the shelves, but things look strangely mundane now. Then my eyes rest on a fat shiny Apress book on Ruby.

I want that book. I want everyone to go away and I want a weekend where I can just sit and read it cover to cover. But I don’t have a single project that I need Ruby for. I don’t even really want to make a project in Ruby. I want to know what Matz did and why. I want to get the gist of it so I can understand blog posts by people that write in Ruby for no other reason than that I like those guys and feel like they care about the same things that I care about.

It’s crazy, of course. Lisp deserves at least another six months of attention and experimentation. Just the other day I was thinking that it deserved three years: one for Common Lisp, one for Scheme, and one for elisp. I was even dreading the thought of picking another language for the whole “learn one language a year” thing. Never mind the fact that I’m already procrastinating my SICP problems. Why do I want that Ruby book all of a sudden?

Ah but my head is swimming with ideas. What I really want is to apply them in a creative way to make something cool. It’s time for a pet project… but what to do?

A scan of the job boards confirms my suspicions. Lots of C# jobs sprinkled here and there. Tons of them to choose from down in the big city. But only a couple of Ruby jobs for the entire state– and no Lisp jobs. Am I wasting time? Should I hedge my bets? On the one hand I should accept the fact that the inertia of my previous work experience means I should settle for a C# job. The postings for the offbeat stuff all want super-geniuses or something. I’m just an “average” developer. And besides, if I go with C#, I can have more options in terms of where I live and what kind of environment I have to work in… right? Or have I turned a corner now that I can’t go back on? Which is it?

It’s too big a question. I need a pet project so I can explore these issues and test them in code. Talk is cheap. Code is the only real test of the ideas. Or rather, working applications, to be even more specific….

But I know how these projects work for me. I start with a sense of elation: new tools open up a door to solve a problem I’ve always wanted to solve. It looks so easy. I charge in and make inspiring amounts of progress in the beginning. Then I start to come up against the limitations of my abstractions. The pressure to get some semblance of working code is so great that I push things as hard as I can anyway… but then the muck starts to creep in. Code that looked brilliant one week becomes an embarrassment the next. And then there’s the unanticipated hard problem that saps my will to continue.

It’s like that every time. My mental vocabulary increases from one project to the next– but even powerful “new” abstractions have their limitations and blind spots so I’m always having to learn more. And while a “hard” problem that stopped me in my tracks years ago is no longer that big of a deal, there’s always more ready to come along because the scope of my ambitions is always rising faster than the rate at which I master new ideas.

In other words, there’s only trivial problems… and ones I can’t do. There are trivial abstractions… and ones that I can’t even imagine a need for. So no matter how good I get, I still feel like an “average developer”. The things I’ve mastered seem trivial– only an idiot would fail to understand them. And it’s those unanticipated problems– the ones that I least suspect– that are going to force me to expand my scope of what I consider to be trivial. But I’ll assume the tools to solve those problems are irrelevant ivory tower academic garbage until I find myself in a situation where they are the only way to get out of an ugly coding nightmare.

But when I’m in the nightmare, I won’t necessarily know what tool I need to master to get out of it. I might not even know what I need to google for! If that’s the case, then the only way forward is to work on projects that are more on my level so that I can expand my imagination to the point where I’m able to even ask the right questions. (This is why the line between doing and talking in programming is so difficult to pin down.)

Anyways… I’ve got two ideas for pet projects.

One is a Common Lisp non-application application to solve problem that’s not really a problem… but really is just a piece of problem for a lot of only marginally related problems. I’ve solved enough “hard problems” for it at this point that it should be trivial to make a single-case solution (especially if I munge just a little bit from Peter Seibel’s code examples.) The challenge would be to take that single case… and then write a few macros that are capable of generating all of that code. I’d end up with a language for solving a certain class of problems… and an application that cheats by using the REPL prompt as the primary user interface for using it. (In the process of writing it, I’ll surely end up accidentally writing my own version of some existing Common Lisp feature.  I do it every time I write in Lisp.) The resulting solution will look silly to any expert Lisp user and advocates of other programming languages will tell me how they can do the same thing using some other tool.

The other is a C# application that I was working on last year. I’d solved enough “hard problems” to convince myself that I could take it on, but got stalled by an unanticipated “hard problem” that wasn’t compelling enough for me to want to tackle on my own. The project was written as a single-case solution. What I want to do is vivisect it until it becomes a tool for solving the general-case solution. I want to use every single functional technique I learned this year to get the abstractions right this time and eliminate the muck that had crept into my bloated object model. I want to write my own scripting language for it so that it’s possible for people to do unanticipated things with my solution. Success would be a working application that ends up being used by anyone other than me. (Unbelievable success would be that my solution is used by someone else to solve the “hard problem” that killed the project the year before—not at the code level, but at the meta level!)

Which one should I choose? Or should I do something else? What do you think?

Advertisements

14 Responses to “Still Average After All These Years”

  1. Empathetic Says:

    Here’s (roughly) my cycle:

    1. Start on SICP, again.
    2. Get frustrated that I can’t figure out a problem even after rereading, although each time I retry SICP I get further.
    3. Worry that I’m just an average idiot for awhile.
    4. Ooh look, something easy and shiny (Rails, Django, jQuery, ErlyWeb, , new book, etc)
    5. Annoyed that I’m not progressing in any one direction, just learning the basics of each thing.
    6. Survey of language landscape.
    7. Return to 1.

    I identify with your worries about Common Lisp and how it’s not “mainstream” enough. At the end of the day, does Lisp really make building something that much easier or more elegant than Python or Ruby, etc? Am I focusing too much on grokking lisp instead of building something useful in a so-called lesser language? Do Lispers focus too much on the telescope instead of look at the stars? Imagine if I had just stopped learning at Python–I’d have several projects done or in progress by now instead of working very hard to get to the point of even starting one in CL. What advantages does learning lisp give you over knowing Python, algorithms, and good CS fundamentals? These are the things that echo in my head whenever I get stuck in SICP or lost in CL while working on something personal.

    Anyway, back to #1.

  2. Ben Says:

    You sound a lot smarter than you give yourself credit for.
    Your post falls apart at the end though, neither idea sounds worth it or even well defined.
    My advice – try option 3, give it up, you will find the other path soon enough.

  3. vanekl Says:

    You ask a number of questions. I have one rather simplistic suggestion.
    Work with C# during the day to feed your stomach.
    Work on CL, Ruby, Erlang, Scala, etc., at night and on the weekends to feed your brain.
    Rinse and repeat.

    Don’t think you have to become a proficient coder in any language to understand the major lessons; good programmers can recognize other good programmers when they read their blogs (except for the nearly insane programmers like _why the lucky stiff who are on a totally different plane from us mere mortals).

    You don’t have to do it all alone, or start from scratch each time you try a new language. Once you’ve identified the experts in the field, just read what they have to say, or politely ask. They are usually very helpful if you don’t come off like a know-it-all idiot or a school boy. If you continue to study your craft and act like a professional, you will be treated like one. That doesn’t mean you have to learn the syntax of every programming language, but you should be interested in learning about the high-level concepts, because once you understand the high-level concepts and whether your language supports them or not, you can always drill down for the details as to how to implement them. I guess what I’m saying is, try to work from top down, instead of bottom up. Instead of attacking learning from a low-level implementation standpoint, try to understand the highlights from a conceptual level. You know whether you understand them if you are able to tell other people what you learned. For me, Ruby taught me about message passing, blocks, and mixins. CL taught me about first-class functions, (real) macros, and code-is-data. Java introduced me to the concept of code serialization. But none of the languages were able to teach me about some of the theory, such as the intractability of some problems, how to estimate the time complexity of algorithms, what FSAs are, etc. You sound like you’ll figure it out. You’re asking interesting questions. As long as you continue, you’ll be OK.

    I’m sure you have a perfect understanding of what you are asking in the rest of your post, but it’s not clear enough to me to comment.

  4. pete Says:

    I settled for one language – Ruby.

    The other languages just come and go. The best combination is ruby with either C, Java, C#.
    (If possible I’d still pick C, simply because C is the most important language despite the success of “OOP” into c-like languages. But then again I guess anyone who has to use Java for their dayjob would disagree on it)

  5. newstuff Says:

    There are lots of new stuff with C# 3.0 — make sure you understand/master the new stuff before you dive back into CL (stay “current” in the C# field). I must be more average than you as it takes all my time just to stay up with C# advances (of course that includes all the plumbing, environment, 3rd party tools, etc.. all of these keep me current).

    For example, if you program in Windows, you should be at least familiar with good utilities, such as:

    4NT, 7-Zip, ActiveWebCam, ASP2ASPX, ActiveState, AwaveStudio, Beyond Compare, BartPE, BitPIM, BlackICE, Blacklight Rootkit, CCleaner, CDex, ClamWin, DAEMON Tools, Dimension4, FileACL, FileZilla, GetRight, Hiren’s BootCD, Hamachi, hMailServer, IEInspector HTTPAnalyzer, ImTOO MPEG Converter, Instant C#, JkDefrag, Junction Link, LINQPad, MacDrive, MagicISO, MBSA 2.1, MIDI Bag, Mono/GTK#, .NET MemProfiler, NAT32, NetTools, nLite, OpenCYC, OpenVPN, Paint.NET, PortReporter, PowerGREP, PowerShell, PureText, PureData, PuTTY, RASMaster, Resharper, RegExBuddy, SmartSniff, SQLTac, SQLyog, SUA Utilities, synergy2, SysInternals Suite, TreeSizeFree, Trillian, TrueCrypt, Tunnelier, TweakUI, UltraEdit, uTorrent, VMware, VLC, VNC, WinDefrag, WinHTTrack, WebLogExpert, WinSSHD, xB Browser.

    (the point is knowing your environment and also knowing quality utilities can be just as important as knowing how to program better)

  6. Will Says:

    You guys really shouldn’t be struggling with SICP. 6.001 is hardly a weeder.

    If you honestly believe you can’t hack a freshman CS course, throw in the towel. Otherwise, get over your Lisp/MIT fetish and find a text that speaks to you.

  7. Fran Says:

    Choose the Lisp project. From your post, it seems that this is where your interests lie. The self-deprecating words about re-implementing features already in the language and having experienced Lisp programmers look down on your code is all the more reason to embark on this project. You will learn much more quickly with constructive criticism.

  8. Daniel Ehrenberg Says:

    I relate to this a lot. I used to be exactly like this, at least, where I’d always be looking for the next abstraction, and reading about that. Fortunately, I’m still a student and don’t need to make much of my own money, so I could keep doing this, but instead I’ve opted to write a bunch of semi-useful open source code in a functional language (Factor), for that same goal of learning.

    I think your Common Lisp idea is a good one; it’s probably one of the best ways to learn about functional programming (or, alternatively, *not* learn about it, if you create too many imperative abstractions). Writing your own scripting language might be more fun, though. But you can combine these: write your own scripting language (and the solution to that vague problem) in Common Lisp, and when you get to someplace where you need more abstraction in Lisp, write a macro.

    If you want to look at a language that’ll confuse/amaze you at least as much as Lisp does, but in a different way, take a look at a stack-based language like Factor.

  9. Justin George Says:

    I have to say, I identify with this pretty strongly.

    In my experience, the key is to get someone else to work on it with you. Something that keeps you honest. It’s like a gym partner, but for programming.

    Another thing that I’ve always wanted to try was routine pair programming on pet projects. You sit down once a month or twice a month, for two or four hours or some such, and split your time evenly between guy A’s project-du-jour and guy B’s project du jour.

    I would be so fantastically more productive this way, it’s not even funny. Anybody want to start a google group?

  10. Mark Miller Says:

    @Lispy:

    I can really identify with your doubts. I believe I’ve told you my story, but there was a time when I really doubted that the effort I’d put into learning about Lisp and Smalltalk, and the concepts that have gone into them, was worth continuing. I’ve thought about just getting a C# job many times, or pursuing a Ruby position.

    I’ve been taking time off from work to learn this stuff. I look at it as going back to school. That’s really what it feels like. The reason I’ve done this is I know how it goes with me and work. I end up spending all of my creative energies on work projects. When down time comes I just want to rest. It’s hard for me to muster the energy to spend time learning about something that’s not work-related.

    Paul Graham wrote an article on this called How to do what you love. He offered two strategies. One was to just go for it. The other was to do a different job for money, and then spend time in a disciplined way on the thing you really want to pursue. The article you cited in a previous post by Gabriel and Goldman called “Mob software: The erotic life of code” had a line that I think applies to those of us who are pursuing advanced knowledge:

    “So gloomy. Bitter and depressing. Deeply conflicted.

    The way out requires just one thing from us—a strange, frightful thing—something slant. It is this: Find a way to fight our fear of death.

    First, though, the gloom. Feel it.”

    I think this sums up what you’re encountering in the projects you’ve tried, but to me it told the story of the cycle of excitement and doubt I’ve been through with what I’ve been doing with my life.

    I saw a saying recently and it really resonated with me. It went something like: “In order to sail to a foreign land, you must consent to lose sight of the shore for a long time.” It was meant to be metaphorical, and it encapsulated what I’ve been doing. The journey I’ve been on is an effort to reorient my mind around the whole enterprise of software development. I’m questioning everything I once knew. Put another way I’m trying to “unlearn what I have learned”. In a way it’s a blessing for me to take time away and forget about C# and .Net, so that if and when I pursue it again I can do so with fresh eyes.

    What I’ve become disgusted with is the very fact that I was once so used to that world that I thought .Net, Java, C, and C++ were all that existed, and were the only languages/technologies worth pursuing and critiquing. I knew of the more “obscure” languages, but I thought they had no relevance to the current world of computing. I’ve discovered they’re more than relevant.

  11. Anonymus Says:

    “there’s only trivial problems… and ones I can’t do”

    Mastering something means to make it trivial. You haven’t mastered it if it’s not entirely obvious. In that sense, triviality is not a setback but an achievement, you should be proud. Now, just the problems “I can’t do” need to be bashed to triviality, too 🙂

    Functional programming makes many things easy, but IMHO the pinnacle of “trivialization” is not LISP (too impure, lack of pattern matching, lack of type system, creepy continuations instead of explicit side effects) but Haskell, I can only recommend to learn it, for instance starting with Bird’s “Introduction to Functional Programming in Haskell” (second edition).

    Concerning C#, I’d go for F#. It’s a functional language (ML-style) but you’ve still got the # in there 🙂

  12. Vraj Mohan Says:

    I felt like I was reading about myself.

    I have been struggling for over a year now, trying to learn Scheme (through SICP) and Haskell, but frequently enticed by Ruby and Python. One of my frustrations has also been working through SICP with Mzscheme. I am constantly confused between the core R5RS scheme, the version in SICP and the “batteries-included” Mzscheme. Do you face the same issue with Common Lisp?

    I also discovered that emacs is a pre-requisite for Lisp/Scheme. Having been a Vi(m) user for nearly 20 years, I am teaching myself emacs now.

    Perhaps we should form a study group or pick a problem to tackle…

  13. the_dormant Says:

    I second Anonymous’ suggestion for learning Haskell, it seem like you’re settled with Lisp, I have nothing to say about that, I’ve been learning scheme and CL through SICP and Norvig’s PAICL and I think that there isn’t a single dynamic language which can beat lisp for the simple reason that it’s only one where data and code have the same shape.
    ML, type theory and Haskell specially will by the way, give you more insights about lisp and functional programming in the large. Books like Pierce’s Types and Programming Languages is simply a piece of art (even though at the end, you’ll be just looking at ruby as being simply a little hack). There’ve been numerous discussions about dynamic vs static typing, by most of the time it was Java/C# vs Ruby/Python/Lisp .. But all the arguments against java doesn’t stand in the case of Haskell/ML. Hey you can even look at the Qi project, it’s still lisp but uses sequent logic for type checking. I have no idea about all things 2 years before, but following the path and reading some essential books will light some new bulbs in your head and make you look at your problems in a different way, and well in my sense it’s the most interesting thing about learning new skills, the more you have, the more you’re able to tackle problems the right way, I even found my self working through Category Theory and algebra and there you really get much more understanding of what functional programming really is. See McCarthy was a student of Alonzo Church, the father of lambda calculus, who has also Turing and Smullyan for students, Smullyan is a famous logician who has written entertaining books about combinatory logic, a field where Haskell Curry has excelled and there’s the Curry-Howard isomorphism to know about, which states for example that simply typed lambda calculus corresponds to propositional logic and if you give for example p to (p => q) it gives you q this corresponds to the type signature (p -> q) -> p -> q which corresponds for example to the type of the function “apply” (even though we can give it more parameters). See, all these topics are really interrelated, I think mastering lisp without knowing about all this is like scratching the surface.
    Also, if you want to solve hard problem, you have to specialize yourself in certain domains, AI for example is well suited for lisp, so it could be interesting for you to work through the AI Modern Approach book like Ravi did, he’s getting a lot of fun because he augmented his skills to access a wide range of interesting problems to solve that you or me will never know about probably if we don’t go the same path (or just try).
    This sounds like a lot of hard work, but It’s a long time investment, you can be sure that one of these days if you persist, you’ll get some return..

  14. Mark Miller Says:

    @Lispy:

    In other words, there’s only trivial problems… and ones I can’t do. There are trivial abstractions… and ones that I can’t even imagine a need for. So no matter how good I get, I still feel like an “average developer”.

    I was reminded of this part of your post by Anonymus. I can relate to this. As I look at what’s been accomplished with the better computing models, I realize I’m not at a point yet where I’d feel confident I could equal these accomplishments. That feels a long way off yet. The way you characterized it reminded me of something Alan Kay said in one of his “The Computer Revolution Hasn’t Happened Yet” speeches. I instantly identified with it. He said something like: in our work, once we’ve mastered something, we get bored with it. And when we get into a new field of study that we have no mastery over, we feel anxious, “So we spend most of our lives feeling either bored or anxious.” I’ve gone back and forth between these two states of being for a long time, even when I was working with the popular technologies.

    Also, I meant to say earlier that I can definitely identify with your experience in the bookstore. Since I’ve gotten deeper into this stuff I’ve gone into Barnes & Noble a couple times and looked at their computer magazine and books section. I used to find it interesting, particularly the .Net stuff, but now it’s dull and mundane to me for the most part. It’s as exciting as going through the laundry detergent aisle in the grocery store: “New!”, “Cleans better than before!”, “Gets out tough stains!”, “Ultra powerful!” (yawn).

    After I’d spent several months getting into this stuff, I used to sometimes feel like that back stabber in “The Matrix” who said, “Why didn’t I take the Blue Pill?” Sometimes I’ve thought life would be so much easier if I didn’t know what I know now, and just went along with “the herd.” The thing is it’s so compelling I don’t want to forget it. Sometimes I’ve thought I’ve doomed myself to either working with the popular technologies while going through a kind of mental torture of knowing it could be better, but having to settle for less; or just getting out of the field altogether, leaving it to its ignorance and nightmares. I don’t want to do either. I am passionate about this stuff, and I just have to trust that this is all leading me to something good.

    If you’re looking for a project idea, I had a brief e-mail conversation with Alan Kay about a month ago, where we talked about the importance of software architecture, and ideas of scientific discovery (as they apply to computing). He suggested something for me to try: Take something that seems really big, like an operating system or maybe a word processor for example, analyze it (at the source code level, of course), and try to simplify it. See what new architectural ideas emerge, and then try to “build everything else out of it”. I think I’ve seen this before. The Gang of Four devised their Design Patterns through a project where they wrote a word processor. Lots of people have found uses for these patterns in lots of different projects that have nothing to do with word processing. Sounds like what he’s talking about. Kay said they used to do exercises like this at Xerox PARC as part of their research in the 1970s. They help you learn about software architecture. Design Patterns has given people an opportunity to go down that road, though unfortunately many who have taken that opportunity have adopted the patterns as a religion, and totally missed the point.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: