An Excel Macro for Pulling in Data from Another Worksheet

You may be a super hacker, but your users aren’t.  All the things you take for granted in a quick query that joins a few tables is far beyond anything they’ll ever try to take on for themselve.  Some of them are pretty good getting around in Excel, but only one in a hundred will touch something like Microsoft Access.  So they come to you for every little thing.

So, in the interests of “faking” database functionality on the Excel side of things, here is a little macro code for looking up data in another worksheet and copying it over into where you’re working.  It looks at the cell to the left of your current cell, then finds a row that matches it in a worksheet named “Data”.  The row of data is then copied back into the current cell and the 9 cells to the right, but if data is already there it does NOT overwrite it.  The scenario here is that someone sends a list of part numbers or customer numbers in an email and your manger types and power users can now have way to quickly append the usual bits of information that they need to really work with it.  Note that in this case, individual users are assumed to be maintaining their own Data worksheets.

Sub DataPull()
'
' DataPull Macro
' Copy text from the "Data" sheet based on the cell to the left of the currently active cell.
'
' Keyboard Shortcut: Ctrl+d
'
    On Error GoTo DataPullError
    Dim Find As String
    Dim Cur As String
    Dim x As Long
    Dim y As Long

    If ActiveCell.Column = "1" Then
        MsgBox "Data macro looks up based on the cell to the left of the current cell.", vbExclamation, "Data Pull macro"
        Exit Sub
    End If

    Find = ActiveCell.Previous

    Do
        x = x + 1
        Cur = Worksheets("Data").Range("A" & x).Value
        If Cur = Find Then
            For y = 0 To 9
                Cur = Worksheets("Data").Range(Chr(66 + y) & x).Value
                If ActiveCell.Next(1, y) = "" Then ActiveCell.Next(1, y) = Cur
            Next y
        End If
    Loop Until Cur = ""

    Exit Sub
DataPullError:
    MsgBox Err.Description, vbCritical, "Data Pull macro"
End Sub

Note: I tried using VLOOKUP for something like this, but performance appeared to be awful for the size of the datasets I was working with.  The “dumb” just-copy-some-data-in-here-and-don’t-worry-about-staying-in-sync approach is used here to eliminate the possibility of any lame slow-downs or program hangs that can be caused by a large number of VLOOKUP’s.

OOPSLA 2008 in Nashville: Let’s go!

John McCarthy will be there.

Mark Jason Dominus, too.

I reckon it’d be a shame if I didn’t go down yonder to see them folks, seeing as how it’s just down the road and all….  Other lispy/geeky blogger types out there: can you make it as well?

The Reason Why the Popularity of Unit Testing is Waning

Unit Testing is essential in two places: if I’m working on a problem that’s really hard and if I’m working on something that’s going to be refactored *a lot*. Your average developer doesn’t do a lot of the latter and does the former only rarely.

Coders tend to “compile-crap” things into existence and then drop into a fearful sort of maintenance mode after that where they improve code only if they have to. Developers simply do not have a vision for anything beyond that. They fear that unit testing is just more of the same sort of stifling bureaucracy that they’re used to in their suffocating work environment. They have no inkling that they can gain several orders of magnitude in effectiveness and productivity by applying themselves in a concentrated study of programming tools, principles, and concepts.

Their managers understand these things even less: they can’t even identify a good programmer when they see one, much less encourage things that “grow” good programmers over time.

The big fat ERP system my company just bought comes with absolutely no regression testing framework whatsoever– and they’re constantly rolling out patches and updates to clients that have made their own extensive in-house modifications. If ever there was a place where unit testing could offer massive benefits in terms of reducing developer exasperation, this is it. Yet the tribal practice is essentially to rely on the users to do the testing for IT.

If unit testing had gone from being buzzword among the “elite” developers to being something that average managers *insist on* whenever they buy a large system like this, then it wouldn’t be on the wan.  At the moment, though, there just doesn’t appear to be anyone on that scene willing to put up a fight for it.

To Develop a Good Design, You Must Begin by Caring About People

“An all-encompassing design culture and strategy in every aspect of your thinking is a more tangible representation of your identity than any clever mission statement or advertisement. And if your design sucks, it simply means you don’t care about people.” — Joshua Blankenship (italics mine)

Josh very succinctly states in his blog what I’ve been gradually discovering over time as a developer. He notes that “it isn’t easy” and quotes Tom Peters as saying it’s “damned hard work,” but he doesn’t go into specifics about how to establish and develop a culture centered on good design.

I think my recent entry on the topic explains in detail how to pull this off– even though I couldn’t put into words what the ultimate objective really was.  However, I think the real payoff will come only if you begin to include the programmers and code maintainers among those that you take time to invest in and care about.  Too many IT managers fail to realize that sacrificing the needs of the coders will have a direct impact on the experience of the end users.  The very fact that most development jobs are mere maintenance work on poorly architected systems is a testament to the fact that we value neither the coder or the user.

How to Deploy Your Skunkworks Application– and Take Over Your IT Department’s Software Development in the Process

The first thing you have to realize is that everyone else in the business is scared to death of software development. If you allow your development to be driven entirely by fear, then you’ve got a one-way ticket to maintenance hell. But how do you get control of the development process if your authority and influence is limited? The direct approach will not work. Your arguments for investing in skills, design, and refactoring will fall on deaf ears. Your only hope is to combine an array of guerilla tactics with an understanding of the terms with which those in control view reality.

There’s only three units of time that register in the consciousness of project managers: 5 minutes, two days, and two weeks. For most programmers, there’s been many times that these little “5 minute” changes have exploded into hours of pain, so most developers never let on that it’s possible to get anything done that fast. So really, the smallest unit of development time that a project manager is going to hear is going to be in the two day range– no one’s going to let on that anything faster is possible, and (in the tradition of Mr. Scott) everyone will pad their estimates by at least a factor of four.

“Two weeks” is a code phrase signifying that the people involved in the problem really have no idea how difficult the problem is or how long it should take. If there’s ever something really hard that has to be done, a manager will assume that two weeks is plenty of time to accomplish it. Anything that *you* want to do that will take two weeks or more to accomplish is going to be immediately rejected. The trick is to never ask to take on anything that’s that ambitious in scope, but rather to focus your efforts over time to gradually increase what’s possible in the “5 minute” range. Once you have a framework in place, the general demands of the users will allow you enough leverage to take on more sophisticated tasks that can take up to two days or so to implement. That’s the general idea, anyway. Below are some specific techniques and maxims you can use to work your way up and take control of the development process:

* While no manager will listen to your crazy ideas, they generally don’t care how you accomplish your tasks. This is the leeway you’ll have to leverage to turn things around in your IT department– at the start it’s your only avenue for improvement.

* If something actually does turn out to take the “five minutes” that you thought it would, use the extra time that you’ve scored to improve the elegance of the code or to learn a better way to accomplish the same task. (How this impacts the project is irrelevant: you’re doing this to invest in *yourself*.)

* When other developers are wasting time playing solitaire, trading stocks, or surfing the web, invest your time in developing tools that help you manage your tasks, problems, code, and data.

* How do you know what tools you should build? Look at where the friction is. Anything that’s tedious, irritating, and boring is something you should look at automating or tooling up for.

* A quick-and-dirty tool can help you get a job done in about the same time as the other fellow that does it the hard way. Tasks that you have to get done weeks or months down the road will give you an excuse to improve your tool(s).

* The thing that’s going to give you an edge is the fact that you’ll have a whole library of tricks, a framework that integrates them, and a “sense” for what fancy-smancy techniques are worth learning and applying on the fly.

* At some point, someone’s going to come in with a task that you’ve pretty much already solved. This is your chance! Basically, you can throw together a simple GUI, code a couple of routines, and suddenly you have pretty sophisticated solution that’s much more robust than what the other guys would have thrashed out as a hand crafted one-off. The trick is that your GUI is just a thin veneer on top of the tools you’ve been using for yourself all along!

* Your solution is not something you could have proposed at the very beginning. You just didn’t know enough back then. So don’t fault your manager for not listening to you back at the start!

* People’s ears will perk up when you say, “I’ve actually got something pretty close to that built already.” The fact that you’ve been the primary user for your stuff for a long while will mean you’ve got a lot of confidence in it. And nobody’s going to take a chance on “vaporware” or “castles in the sky.” “This is practically done,” you say, “I’ve just got to make a couple of quick modifications and we’re there.”

* The fact that you can solve an entire class of problems in 5 minutes that would take your peers two days is what will get people’s attention. Nobody cares what you can do in two weeks time. They don’t even care what you can do in two days time. Five minutes is all anybody’s going to care about at this point.

* Once your solution is deployed, there’s a whole world of forces that your pride and joy will be subjected to. The users are going to be a great resource for new ideas, though. Go watch them work and keep talking to them until they can give you some real constructive feedback. After one minute they’ll assure you that everything’s fine. After five they’ll mention a couple of things that could be better. After ten or so, they’ll be telling you what you really want to know: “Now I don’t know anything about computers, but what I really need is….”

* The worst thing about in-house development is that applications only have to be “good enough.” Once they get that far along, the creativity stops and everything freezes into maintenance mode. You can identify a complete lack of design in an IT department when you see a proliferation of forms, screens, procedures, and programs. You’ve got a lowest common denominator philosophy because nobody’s taking the time to find the common denominator to all of these forms, screens, procedures, and programs.

* If you’re *designing* software, you’re not just adding new screens or modifying existing ones. Real design is going to incorporate subtraction and abstraction as well. A good abstraction is a technique that allows you to synthesize many ideas at once– and once it’s implemented, you’ll be able to subtract away a lot of unnecessary code.

* As your users begin to get the hang of your application, be “jonnie-on-the-spot” when it comes to all of their “5 minute” type requests. If they feel like you’re listening to them and taking care of them, they’ll be more willing to give you a chance to invest in the stuff that will provide the real payoffs.

* Anything that the users ask that’s more in the “two hour” range” should be put on hold until you have at least a dozen such requests. List them all on a whiteboard and kick ideas around with the one person at the company that’s willing to listen to you. Then wait some more and let these ideas percolate a while.

* Code up proof-of-concepts for some of the hard stuff and expose it inside your application with little regard for design. Walk some of your power users through special “previews” of these features so that you can have a chance to soak up as many good ideas as you can from them.

* Now go back and look at the white board. Erase it and reorganize everything into related “buckets” of functionality. In the bottom half of the board, sketch out the GUI for something that could address the bulk of the requirements. Underline the stuff you can address right away with only a minor reorganization of existing code along with new GUI elements. Separate out stuff that appears to be hardest.

* There are undoubtedly areas of the code that have known quantities of suckage in them. There’s areas that really irritate you every time you have to go in to modify them. Avoid wasting time fixing this sort of thing until they’ve had time to fester for a while. As you’re about to go in and fix everything up for your users, be sure to invest some time in cleaning these things up for yourself while you’re making such big changes. If there’s a huge visible leap in the functionality, your users will at least have something they can blame it on. “By the way, when you added this hot feature that I use every day now, this minor thing over here stopped working the way I want it to.” You’ll never have time to work on something just to make it objectively cleaner for “no reason”, though.

* If this takes longer than what folks want, don’t sweat it. Remember, in-house software only has to be “good enough.” You’ve already addressed all of the critical stuff that people absolutely have to have to do their jobs. This reorganization of the code combined with a dozen sorta-optional type features just isn’t as urgent. It’s more important for you to control the growth of the application in a sustainable way. Listen to them as if they’re the first person to suggest whatever feature it is that they’re asking for, but remind them that as long as the code is “good enough” for them to do they’re jobs, they’re just going to have to wait. If you have to respond to these demands immediately, make sure that whatever they’re asking for fits in to your overall scheme for how the application should be structured. Better yet, use some of the time you spend focusing on their needs to flesh out, extend, or improve areas of the code that you can use to nail down your vision further down the road.

* You’ve gotten this far by investing in tools first and foremost for *yourself*. You’ve consolidated your power by training the users to think in terms of tools they can use to solve their problems– as opposed to a screen with buttons on it that they just go in and push.

* You may not always have managerial support for each new order of magnitude increase of functionality in your application. If you’re in doubt of this, build it anyway. Do it on your on time if you have to. Roll the application out on a “Beta” basis to a few of your more savvy users. When your managers find out about the application, they’ll more than likely make you stop everything and load it up on their machines and pester you like crazy for new features.

* Each order of magnitude of increase in application functionality will take more time to develop than the one before– but each one will put more and more features within the “five minute” range of your grasp.

* The final coup de grâce is when you’ve developed your application so much that you’ve gone through several refactorings and paradigm shifts. Your design is so clean and your code so expressive… you’ve rolled with anything the users could throw at you and made something truly beautiful. Then someone comes by and mentions that folks in another department have very similar needs as the ones you’ve been coding for. You listen to their requirements for a while, stroke your chin thoughtfully and say, “I’ve actually got something pretty close to that built already. If I make just a few modifications to the framework I’m using for this other application, I think we’d really only be a day or two away from having something that could work for them.” And so, you get the green light for another project and you are able to justify ever more ambitious features and refactorings because each hour of effort you invest in either can be harnessed to benefit both projects.

* Once you’re this far along, the question will come up as to what the company will do when you “get hit by a bus.” If you get cornered on this, just start talking about how they really have a good point on that. Explain how you’ve had to sacrifice code quality in order address all of those urgent feature requests– and they’re some areas of the code that even you’ve forgotten how they work. Really, the company would benefit greatly if you spent a month or so polishing and cleaning up that stuff. Even better, you should take some time to write articles about the trickier stuff you’ve done in the application. Once published, the articles will generate valuable feedback from more sophisticated developers that simply cannot keep quiet when they see a technical flaw or innefficiency. Those articles can then be folded back into the project documentation for whoever inherits the code. (Cue maniacal laughter….)

* Of course, you and I both know that no matter how great your application is, all it takes is one random IT manager to come in two years down the road and insist that 6 mediocre developers get hired to rewrite everything in the latest Blub IDE or whatever. More than likely, your application will remain “good enough” for quite some time and the average developers that come after you will make only surface changes to it. What will probably happen is that the creative integration of various requirements will cease and the new developers will gradually get bogged down as they create one lame hand crafted form/screen/procedure after another and begin to spend all of their time on maintenance instead of design. But that’s okay. They only needed to make something that was “good enough” for the moment. By the time the company fails because of their negligence, no one will suspect how much they were responsible for so many folks losing their jobs….

Advanced Codemunging: A Report from the Trenches

A year ago, I began studying Lisp and Emacs. I would ultimately get through about half of SICP and code up a few small Lisp projects that mainly showed how far I was from grasping the point. Having Emacs skills did come in handy a few times, though. Engaging in such focused deliberate practice was difficult: I rarely came up with anything truly innovative or useful, but instead had to constantly face my weak areas and incompetencies– and doing all of this pretty much in public would ensure those weaknesses would become notorious among the illuminated. I wasted some time blogging about programming practices and things gradually culminated into a manifesto of sorts. While I now had enough technical know-how to address some of the biggest problems I’d faced as a career programmer writing simple database apps and web pages, I also had a minor crisis in choosing which language to implement my new vision. Circumstances at work (and a large decrease in free time to experiment in) would soon narrow my options for me, though. This blog post is about what happened next.

I got really lucky in the way things panned out in my career. You’ve probably read where Joel Spolsky was talking about how awful it is to get stuck being an in-house developer. More recently, Paul Graham has gone off on how we’re all supposed to be like the start-up coders that he funds. Really, most of us aren’t *that* talented to begin with… or at least, we’re not willing to live the start-up lifestyle or take on the associated risks. But there is another option that can be almost as satisfying as working in a Graham/Spolsky type setting: be the *only* developer in a company that’s large enough to have interesting problems and that’s looking to cut the cord with as much of their legacy software as they can as they move to a new system.

If you can land that sort of job, you’re probably savvy enough to avoid most of the stupidity that comes with dealing with corporate type development. Because you’ll be they only developer on staff, you won’t have to waste time building consensus with fearful managers and mediocre “day coders”– and face it, they would never have gone along with your insane ideas to begin with. Also, you have a huge business value bearing down on the side of good design: you’re essentially getting paid to eliminate and refactor away all of the crappy development projects that have accrued over the years. The scope of such a change is so big (and everyone else is so busy), you’ll have to come up with big architectural ideas on your own and you probably won’t even have to pitch them to anybody except the other techy type people that are in your (small) IT department. You’ll just have to make the right decisions on your own.

Looking back on the most painful moments of my career, I think I have pinpointed the cause of most of the disasters. Because I needed the experience, I allowed myself to get into positions where I had to maintain code that I had no authority over. Whether out of my own fear or by decree of my immediate manager, I was isolated from the problems I was supposed to solve. I would continually get requirements that were just enough wrong to be useless and would often spend as much time removing them as I did implementing them. The common denominator in most successful projects/modifications were in situations where I rewrote entire sections of code (with or without permission) and where I also took the time (with or without permission) to hash things out with some smart folks lower down on the totem pole– people that actually did the work and people that would end up using the application. Talking to people like that would often uncover the real problem– but then I’d have to “sell” a new solution to managers at one or more companies. I’d often spend more time talking about the issues with management than I did writing code. And its those sorts of interactions that become the fodder for an unlimited number of Dilbert strips. And because of all of the bad requirements, IT departments start to withdraw from their users and create buffers of bureaucracy to protect themselves from getting “burned.” You can let this insanity grind you down… but you can also take advantage of it if you play your cards right.

You’ve got to realize that while management types generally can tell you what to do (and sign your paycheck and stuff), they can’t really help you with the bulk of your development work. Too many of them are going to just tell what kind of buttons to put in the application. “I just need one that has ANSWERS written on it.” Right. If you can just get your foot in the door, you won’t be so dependent on them. Any sort of application that you can roll out is your ticket to finding our how things really should work. Talk to your users at all levels. “Joe Six Pack” knows a heck of a lot about the company and will not waste your time with stupid requirements. Generally speaking, no one asks his opinion: he’ll be grateful for giving him a voice and making his work easier.

Growing up as a geek, you probably played off-beat rpg’s and war games in your basement. In order to run them, you had to have the bulk of the system “in your head”. This type of thing is very similar to a company’s overall business process– and your contacts with people on the ground and your familiarity with the back end data structures will allow you get large portions of a company’s business processes “in your head” as well. (As long as there’s some room left up there what with all the AD&D stuff junking it up.) The trick is to address the process issues in a coherent way: what’s convenient for one person may not be efficient for the whole. It’s this kind of knowledge and savvy that separates you from the code-monkies. People that have a slightly smaller interest in programming will often use the position they gain from this to become de facto controllers of the company. But you’re a geek, so you’ll probably just be glad to have the chance to deal with management as an equal instead of as a flunky. That way, you can get a chance to implement the code designs of your dreams! Party!

So, once you’ve got a decent job where you have the authority to develop code effectively, how do EMACS, Lisp, and SICP impact your day to day work? The main area is in your attitude. Even if you’ve never truly mastered those tools, you end up cultivating a conviction that your program should be dynamic and malleable at all levels. Your abhorrence of repetition and bloat will be cranked up to “11″ and you’ll just have this default expectation that your code should magically respond to you no matter what you’re trying to achieve. You’ll also be exposed to enough unusual ideas that you’ll see things you can do back in “blub” that you never would have imagined. You’ll have a wider range of programming styles at your disposal, and your ability to discern various “code smells” will be much sharper.

Back on the job, I did choose a ubiquitous “blub” as the primary language. The lesson of Lisp is that it takes very little to engineer a custom programming language: just a few axioms will get you something reasonably complete in short order as long as you’re able to combine the pieces into more complex concepts. I realized that if I stole somebody else’s evaluator, I had 90% of what it took to make a dynamic language. I embedded the evaluator into an “environment” that could manage the overall state. I then wrote a small deck of crappy string functions that I used to “compile” lines of text into “code objects”. From there it was a piece of cake to write code to “execute” the objects by running a series of operations on the environment variables based on what’s specified in a list of those “code objects”. This was all done with the crappiest of quick-and-dirty programming techniques. Just awful. It didn’t take long, though. And it doesn’t have to be great. It just needs to be “good enough” for now.

Once the core of this (albeit awful) custom language is done, you’re (generally) only about two hours away from being able to implement any given language feature that you want to do. And thanks to a small familiarity with Lisp, I realized that there was a lot of slick ideas that are opened up once a programmer has gains control over what gets evaluated and *when*. Macro-like functionality became possible, among other things…. In practice though, coding a new language feature take me maybe ten times as long as it would to write a comparable function to do the same thing. There are things that can be done to cut the time off of that, and it’ll be fun to someday compare and contrast my ideas with what can already be done with List “out of the box.”

What I ended up using my roll-your-own language for was to build a scripting language for customizing my main application. When the application is run, it reads a script to set up the environment. All of the GUI is bound to the environment. Advanced users can customize 90% of the application’s functionality without recompiling– and as a bonus, I have a way to test and experiment with malfunctioning desktops without having to use the old “it works on my machine” excuse. Not quite as good as a REPL prompt, but good enough. (For now.)

Because I’m a database app developer, 80% of my code is about connecting to database X, prompting the user for some stuff, getting a deck of rows, and maybe doing something for each row in the deck. (I get so sick of seeing the same crappy DAO, ADO.Net, and stored procedure call code over and over. Blech!) It sure would be nice to have a language designed completely around doing exactly that. And the thing is… I do! My custom language has all kinds of features for manipulating blobs of text. My “environment” stores the blobs in a prominent place, and the GUI automatically picks them up, displays them, and knows how to operate them. This may sound crazy at first, but I’ve been able to come up some powerful front end features because of this approach. Features that I never would have thought up emerge naturally and give my users an unprecedented amount of functionality. Dozens of minor problems routinely get addressed all at once with relatively simple abstractions.

For the first time, I am coding against the problem at both ends. The blobs of text that represent the fundamental unit of work in my system are abstractions that I once referred to as being “hairy at both ends” after trying describe something Peter Norvig was doing in an code sample. These abstractions allow the functionality of the program to be extended gracefully– and I’m generally only a “very short time” away from being able to extend those functions for either myself on the code side or my users on the GUI side. No… for the first time, I have an architecture that acknowledges myself as being the primary “user”. Refactoring is now easier than any other alternative– and the business case for it can be made convincingly at every stage of the project. I can code features that make it easier to manage growing collections of business object blobs in my heavily syntactic sugared scripting language. New blobs are automatically picked up by the GUI… and serendipitous GUI features emerge naturally because of the consistency inherent in the abstractions. Each layer of the architecture builds on and magnifies the efforts of the underlying layers. And every line of code I write is impacted by my (admittedly mediocre) understanding of recursion and the closure property.

You see a lot out there about those supposedly super-duper type programmers that are worth ten times as much as their lesser colleagues. While I will not claim to be one of those guys, I do feel at least ten times as effective of a programmer as I was a year ago. At any rate, I love my code base, I enjoy improving it continuously, and I enjoy seeing my users immediately benefit from the wacky ideas I come up with back at my desk. For the first time in a long while, I wake up excited every morning and I actually look forward to going to work. Life is good.

Thanks John, Harold, Jay, Peter and Paul. You guys have given me answers to the core problems I face every day– even as a lowly database app developer. I may not have the privilege of being able to code in a lisp dialect professionally, but at least I can now begin to “Greenspun” with the best of them….

Infocom Drive Discovery May Lead to Recovery of MDL Source Code

An Infocom drive (or a CD containing the contents of it) has been recovered that contains the source code for every released and unreleased game produced by Infocom.

As a result of this discovery, some private emails from the Infocom Implementors have been published on a blog. Anyone who’s anyone in the world of “interactive fiction” has responded– including Anita Sinclair of Magnetic Scrolls, several Infocom authors/programmers, Graham Nelson and Emily Short.

Lisp enthusiasts will note that the discovery of this drive may lead to the recovery of the source code to the MDL programming language. Zork coauthor Dave Lebling appears to be in contact with the author of the blog (and hopefully the owner of the drive/CD) to discuss this and other topics.

Update:

In the comments, Tim Anderson writes,: “the disk seems to be what was shipped to Activision (along with its containing Sun server, of course) when Infocom Cambridge was shut down. No particular reason, therefore, why it should have included the full MDL source–no one at Activision would have known how to do anything with it.”  Meanwhile, Andreas Davour has found an MDL interpreter.  Mark Blank is posting tips and references….  Wow, this is cool.

Another Update: Allen Garvin has posted a link to the MDL primer.  Sweet!

.Net Framework 3.5 Killed My Box– Update

[Sorry... no Lisp/SICP type content here.  Just another public service message.]

We have a slew of old HP machines running at 2.6 GHz, 504 MB RAM, and with mediocre hard drives with at least 25 GB of hard drive space on them. I expected this to be ample performance to run bleeding edge .Net applications, but it may have been pushing it a little. In any case, the machines have been running continuously for five years or so– they never get shut down. I was not aware of this when I did the installs, and later found out that a couple of machines had “trouble” when they were rebooted.

One machine simply died when I shut down before beginning the .Net install. I was rebooting everything when I was doing the load because I wanted to pin down the exact install that (potentially) could kill the machine. Little did I know at the time, that I was probably better off attempting the complete deck of installations without any rebooting.

Two other machines would attempt to boot up, but take you to the safe mode menu. Attempting to boot from there resulted in the system hanging after it loaded a file called mup.sys. One of them was that first machine that died after installing .Net on it. Another just seemed to come by this naturally.

After looking at all of the facts, it does not appear that .Net is directly to blame for any of the problems. Two of the boxes were seriously abused before I even showed up. The one that I thought I had killed probably had a dodgy hard-drive. Information about the exact cause of the mup.sys hang is sketchy at best– it can apparently be caused by any number of things.

Most users seem to be running fine. Some have complained that installing the new software has slowed their boxes down, however. There’s been no major disasters so far. Users are hitting the application pretty hard now and things seem to be holding up. I did not have to change to a web app at the last moment. We’ll keep an eye on things, but I’m not worried.

.Net Framework 3.5 Killed My Box

This is a non-Lisp related Public Service Message:

The boxes some of my users are running are pretty mediocre– running XP with really old hard drives that just chug along….

Installing .Net Framework 2.0 Service Pack 1 and .Net Framework 3.5 and then trying to reboot, I find that the box is completely hosed. Plugging in the hardrive to another box… the computer no longer even recognizes it as NTFS!

After putting the backup image back onto it and adding some other software, rebooting several times, everything seems fine. But putting 2.0 SP1 and 3.5 on it again hoses it all over again. I don’t know if this is an isolated case or not– I don’t know what will happen if I try to deploy to other old boxes. I’d already deployed to several other machines just fine… but maybe not to any this old.

Maybe I missed some fine print somewhere, but this is not what I expect when doing what I’d otherwise expect to be a trivial installation.

I will follow this post up with additional details if I find anything out. I may be converting my desktop application to a web app if this happens again….

A Quick and Dirty Look at Exploratory Programming in Common Lisp

As the uproar subsides over the recent release of Arc, I can’t help but conclude that the folks that are most outraged by the lack of Unicode support would indeed have little interest in the results of Paul Graham’s labors. Their core values and world view are so antithetical to Paul’s, that they simply cannot see the point. You don’t believe that to be true, of course. You just think Paul Graham is stupid. It doesn’t matter that he wrote On Lisp or that guys like Peter Norvig have a few nice things to say about him. You know he’s insane… and somehow, you’re hurt, insulted, and angry all at once. This is understandable, really. Lisp really is sort of an alien technology. The more you use it, the more it infects your brain. You get all wrapped up in what suddenly appears to be the right way of doing things. But this explanation is meaningless to you. You shrug it off and continue to justify your rage.

But at least give me a chance to make one brief analogy. You know how it is with your wife/girlfriend. You want to be nice, so you offer to take her out to eat. You ask her where she wants to go. She says she doesn’t know. You say you’ll take her anywhere she wants. She insists that you pick somewhere that you want to go. You pick a place that you think would be nice– and she immediately says she doesn’t want to go there. You get frustrated and annoyed. Her feelings get hurt because you can’t hide the sharpness in your words… and suddenly you’re stomping off to avoid the litany of complaints that are sure to follow. “You never do x; you always do y; blah blah blah.” As you mope in your computing dungeon, you feel that something significant has transpired. You feel like there was no way you could have won, yet you’re sure you missed something. You, my friend, are now in the dog house indefinitely.

And that’s the truth of it: you live with an “alien.” You look at the same events and view them differently than she does. You think differently. You have different needs. The “alien’s” rationalizations seem incredibly illogical to you… and you just can’t understand it. The same thing is going on between you and Paul. (Not the living together part. You know… the misunderstanding part. It’s an analogy. Stay with me here for a second.) More than likely you’d rather justify your confusion and disappointment. Certainly, sifting through the kibble to find the meat is the last thing on your mind. But I’m going to give you the benefit of the doubt. Perhaps if you had a little more concrete explanation of how Lisp enables exploratory programming… maybe then you would cut Paul some slack. (Just a little, maybe.) Here goes….

Lisp allows you to define symbols that reference atoms and lists. This allows you to create a shorthand for whatever it is that your working on. Below we create a symbol that represents an x-y coordinate and then reference that list in another symbol:

CL-USER> (setf a '(4 2))
(4 2)
CL-USER> (setf b (list 42 a))
(42 (4 2))

Now one thing you should remember… that really is a reference to ‘a’ “sitting” in b’s list. If I make a change to ‘a’, then ‘b’ will change, too!

CL-USER> (cadr a)
2
CL-USER> (setf (cadr a)
               7)
7
CL-USER> a
(4 7)
CL-USER> b
(42 (4 7))

I don’t know about you, but in my mainstream OOP language of choice, my tendency is to start with my own Point class in its own code file. Next I’m coding up a seperate Widget class in its own class file. It has an integer property and point property. I code a variety of constructors in both files and code the getters and setters for the properties of each. Then I start feeling guilty about my Widget class because it violates the Law of Demeter, and I want to go back and put a wrapper on the point property so that no user of a Widget object will reach through to the methods and properties on its internal Point object reference.

Notice above, that we could define a getter function with the cadr operater to retrieve the y coordinate of our point. Setf could then be used with that getter function to make an instant setter. So in lisp… once you define a getter, you automatically define a setter, too. This may seem trivial, but as your object model evolves, I sure get tired of mucking around with property definitions in a half dozen files. In Lisp, all of that stuff is optional. You can just jump in and get started. Note also that because lists can reference other lists, you get a “quick and dirty” form of inheritance for free. Sort of. But even better, you don’t have to take any extra effort to write special code for setting up collections and arrays. The list’s hierarchical structure gives that to you for free.

CL-USER> (setf c (list '(x y z) 42 a))
((X Y Z) 42 (4 7))
CL-USER> (defun tags (x)
           (car x))
TAGS
CL-USER> (defun weight (x)
           (cadr x))
WEIGHT
CL-USER> (defun point (x)
           (caddr x))
POINT
CL-USER> (tags c)
(X Y Z)
CL-USER> (weight c)
42
CL-USER> (point c)
(4 7)

So above, we demonstrate the creation of property “getter” functions… and also show how one of those properties can be a list of items with practically no extra effort. I know, I know. Strongly typed collections that take advantage of those new fangled generics features are great fun, when you’re prototyping you’re not all that interested in that sort of paranoid defensive restrictiveness. You may not even know that “types” each of your properties should be, yet!

Here we define a symbol that represents a list of our objects. Notice that the list operator is sort of a universal constructor:

CL-USER> (setf d (list (list '(x x) 17 a)
                       (list '(y z q) 56 a)
                       (list '(z z z) 73 '(2 1))))
(((X X) 17 (4 7)) ((Y Z Q) 56 (4 7)) ((Z Z Z) 73 (2 1)))

We need to instantiate three of our objects and store them in a list? No problem! You just _do_ it…. The dirty thing here is that you don’t have to define any constructors for your objects. Further… you don’t have to have “all” of your object defined or set up in order to test parts of it.

Let’s set up a simple function designed to operate on a list of our objects. (This is a completely arbitrary example.) But wait… we’re not sure what we want to do as we operate on each item. Well… let’s just skip that part and let the caller of the function tell us what to do each step:

CL-USER> (defun do-something (fn &rest args)
           (funcall fn (car args))
           (if (not (null (cdr args)))
               (apply #'do-something fn (cdr args))))
DO-SOMETHING     

CL-USER> (apply #'do-something #'(lambda (x) (format t ">> ~a~%" (weight x)))
                d)
>> 17
>> 56
>> 73

And instead of setting up a variable to hold our list… let’s just use the function with some objects we define on the spur of the moment:

CL-USER> (do-something #'(lambda (x) (format t ">> ~a~%" (weight x)))
           '((X X) 17 (4 7))
           '((Y Z Q) 56 (4 7))
           '((Z Z Z) 73 (2 1)))
>> 17
>> 56
>> 73

This may seem weird, but the more you use Lisp, the more natural this becomes. Lists are fundamentally recursive: they’re defined that way, so you tend to write recursive functions to operate on them. Many built in functions will take functions for their arguments, so you tend to do the same thing with your own code. This not only lets you defer some design decisions, but it also lets you build reusable skeleton functions that are broadly applicable. Finally, “rest” parameters give you even more flexibility… and *another* excuse to write recursive functions. It’s trivial to modify a function to utilize the rest parameters, so you end up using it in places you wouldn’t have thought to.

The syntax for what we just did may seem a little hard to keep up with. When do you you use funcall and when do you use apply? When do you use neither and just call the function? It can be frustrating at times, but that’s the sort of thing that motivated some of the changes in the Scheme dialect.

(I know… I could have used a built-in mapping function for the above. But that’s one cool thing about recursion. If you’ve forgotten the syntax or the name of a built-in function, you can generally write a recursive substitute for it in a pinch. This keeps you rolling along even when you are hacking without reference materials or an internet connection handy.)

Anyways, the main thing here is that you don’t have to waste time juggling class files. You don’t waste time writing properties and constructors and overloaded versions of your constructors. You’ll use “getters” as an elementary abstraction barrier of course, but you don’t have to endure constant object instantiation rituals. You just type out the list that represents your object’s state whenever you need a new one. (This means you can write unit tests that are very clean and concise, by the way… and you can even cut and paste them directly into the REPL prompt without near so much fuss and bother as you might have on other platforms.)

Also, you know how annoyed you get with having an IDE reformat your code even when you don’t want it to? Sometimes the strictly enforced “one class to a file rule” can be equally bothersome. I like to group all of my boring “getter” functions together in one place. This helps document the classes’ read/write fields. Also, related methods are kept together even if they belong to different classes. This makes it easier to think about related parts of the program… which leads to better utilities “bubbling” out of the prototyping process. (By the way, the REPL is just plain great. In terms of its significance, I’d say it’s easily up there with Source Control and Unit Tests in terms of how much it impacts your programming. The amount of time developers waste in the compile-crap loop is insane… and it may be that an entire strata of unit tests are necessitated by the lack of a REPL. But that’s a different story.)

Hopefully this little article can give you some insights into some of the “quick and dirty” type things Paul was talking about. This is not advanced voodoo rocket science by any means, but this is stuff you can master fairly quickly as you start programming in the Lisp dialect of your choice.