Archive for June, 2007

Replacing MS SQL “Enterprise Manager” with Emacs

June 28, 2007

Emacs can be used to talk to MS SQL-Server. The benefits include, among other things, that you get a history of all of your commands in a single text file and you can bring up previously entered comands with M-n and M-p just like you do at the Lisp REPL prompt. Sometimes its just not worth opening up Enterprise Manager if you only have a quick thing or two to edit or check on. And if you’re doing your development in EMACS, then you might as well stay inside of it instead of jumping out to another program….

To connect, just type M-x sql-ms. You’ll need to know your server name and password and stuff, of course, when the command window prompts you. Once in, you can type SQL commands to your heart’s content. If you’re wondering why your commands don’t return anything you either forgot to type “GO” to send the command… or maybe there was a syntax error in your SQL. Type something like “Select 1” and “GO” to find out what the error message was. (For some reason, you have to send a working query in order to get the error message back.)

The following queries can replace some common tasks done from within Enterprise Manager:

–Get Table List
SELECT SUBSTRING(name, 1, 20) FROM sysobjects WHERE xtype = ‘U’ ORDER BY 1

–Get Columns List
SELECT SUBSTRING(name, 1, 20) FROM syscolumns WHERE id=object_id(‘Your_Table_Name_Here’)

–Get Stored Procedure List
SELECT SUBSTRING(name, 1, 20) FROM dbo.sysobjects WHERE (type = ‘P’) ORDER BY name

–Get Stored Procedure Parameters
SELECT SUBSTRING(parameter_mode, 1, 10), SUBSTRING(parameter_name, 1, 20), SUBSTRING(data_type,1,10)
FROM information_schema.parameters
WHERE specific_name=’Your_Proc_Name_Here’

–Get Text of Stored Procedure (ugly)
sp_helptext ‘Your_Proc_Name_Here’

And now the bad news…. Due perhaps to quirks of both isql and emacs, queries with several columns can look pretty ugly when you get their result set. That’s why I used SUBSTRING in the commands above. (I can’t figure out how to turn off the automating line-wrapping.) For one thing, the column names get shifted over annoyingly because a string representing the amount of time it took to run the query gets printed at the beginning of that line. Also the column widths are generally way too big.

If you want to take the text of a stored procedure to another buffer, just type C-SPACE to set the mark after you type GO for that command and just before you hit ENTER to execute the SQL. When the results are returned type M-w to put it in the kill ring. Use C-x C-f to open up a new file… and C-y to paste in your stored procedure code. If you’re going to be editing a lot of stored procedures then you’ll probably want a macro to clean up all of the erroneous white space you’re gonna be pulling in with this process. Type C-x (to start recording a macro. Type C-M-% to do a regex replace. Replace [ ~]+~[ ~]+ with ~, where the ~ character is a carriage returned entered by typing C-q C-j. Type M-< to go to the top of the file. (Each square bracket should contain both a space and a carriage return.)  Type ! to apply the replace to the whole file from the cursor point down. Then type C-x ) to finish recording the macro. To reuse the macro on other files, just type C-x e.

This is overall a pretty mediocre experience, but it demonstrates that it is possible to work on SQL Server from Emacs. With the right .emacs file, though, much of this can be streamlined and made easier to deal with….  Of course, you may not always be working on a machine that has Enterprise Manager installed, in which case going this route allows you to avoid some of the irritating free-ish versions of Query Analyzer that I’ve seen floating around….

Update 7/12/2007:

Fix the Word-Wrap nightmare while in the sql-mode with M-x toggle-truncate-lines.

Emacs tips for wannabe Lispers

June 26, 2007

Learning Emacs means you no longer have to depend on somebody else’s slow, bug-ridden, and bloated graphical interface. Making the switch can be painful for a while, though. The hardest thing about it is that setting things up to do what you want is too often like an Infocom adventure game. Most of the people using it don’t want to do the kind of crappy Windowsy things that my job requires of me, so I can’t always find what I want to know via Google. Even the user manual has a pretty bitter attitude towards poor schmucks like myself running on “MS-DOG.” Meh. But we’ll just focus on Lisp hacking for now. Lisp in a Box takes care of most of the annoying set up issues, so we can just start coding.

Once you’re in Emacs, C-x C-f lets you open a new file or an existing file. If you don’t know the name of the file you’re looking for you can hit the space bar to get a new window to pop up showing you the files in the current folder. Every windows application that has a graphical file dialog generally takes eight to ten seconds to get the same thing done (Argh!), so this is one more area where Emacs can lower your blood pressure. You won’t even need the “poor-man’s” file selector most of the time. As you type out your file name you can hit tab to auto-complete stuff. Be careful, though… hitting tab too soon can take you to a list for clearing up the ambiguities….

If you see a file in windows explorer that you want to look it, just drag it onto your Emacs window.

To switch between buffers, you use the C-x b. This gives you a list of all of your buffers and you can select one by typing out the first couple of letters of its file name and hitting return. This feature is much better than the standard windows Control-F6 bit. Not only do you get a list, but often you tend to go back and forth between just two files for a while. Because the buffer list is ordered by when you last used them, it defaults to the last buffer you touched. That’s cool– especially when you’re coding in lisp switch back and forth from your code file to the REPL prompt.

The scroll bars in Emacs are really crappy. At your REPL prompt, you might scroll up to look at some of the stuff you’ve previously typed in. Then when you want to enter a new command into the top level, the scroll bars freeze up just before you get to the bottom of the buffer. The answer to this is that, first of all, you should feel guilty for even touching the mouse to begin with. Use M-> to zip down there immediately without ever leaving the safe confines of your keys.

C-x C-b opens a buffer and splits the screen horizontally. If you want to split the screen vertically, type C-x 3. Word-wrap gets turned off in that last deal, so you may find it less than useful sometimes. Your code will just trail off into the ether in vertical split mode… and worse… when an error occurs from the REPL prompt, you’ll only the first little part of the message sometimes! I’m sure this can be fixed, but I keep putting off doing the necessary research. Doing C-h a to pull up the Apropos for “wrap”, “word-wrap”, and “word” doesn’t help… but it is good to know that Emacs can studlify-word whenever I need it to. (Oh, and by the way… if you only know the Windows term for a feature… you suck! Thank-you, Emacs self-documenting features, for making my day….)

If you’re editing a function in your code file and you want to make sure that commands entered at the REPL prompt pick up the change, C-c C-c will do the trick. If you’re at the REPL prompt and want to operate on the results of a previous command, *, **, *** are variables that are bound to the last three outputs. If you want to re-enter a previous command to either run it again or edit it, M-p and M-n cycle through the list of previous commands. If you’ve already typed in the first part of the line, M-p and M-n cycle through the list of commands that start with those letters! Very handy…. And of course, the tab key grants the usual auto-completion feature… even from inside a string, too. It ain’t quite intellisense, but typing in a function name will print out its name just below the command window– it will also list its arguments, but with generic names sometimes instead of the variable names you gave it in your code file.

Cutting and pasting in Emacs is one of the hardest things to get used to if you’ve been crippled by years of habitual Windows use. Highlighting text with the mouse automatically puts the text into the copy buffer. C-w cuts the highlighted region. C-y pastes it back in. (No, I’m not yanking your chain, either!) This is really hard to get used to. I get by most of the time, but sometimes I still open Notepad if I’m going to be pushing a lot of text around. I look at the additional commands for these sorts of things but never really retain them– probably the only way I’ll learn these extra motion and editing short-cuts is to use Emacs from Cygwin without any mouse at all….

Oh, and watch out for the Undo command. It doesn’t come with a redo, but instead let’s you undo your undo’s. Use a source code control like SVN and check in regularly!

Also… when you lock up the REPL because you’ve called an out-of-control recursive function, just call M-x slime to fire it back up again.

Anyways, this is most all you really need to know to code Lisp from Emacs. If you have any additional tips or tricks– especially for Windows nerds just now getting up to speed– please post them in the comments.

A Hash Table Full of Closures

June 12, 2007

“Closures are one of the uniquely wonderful things about Lisp. They open a door to programming techniques that would be inconceivable in other languages.” — Paul Graham, ANSI Common Lisp p 109

“Object-oriented programming is exciting if you have a statically-typed language without lexical closures or macros. To some degree, it offers a way around these limitations.” — Paul Graham

“Lexical closures provide a way to get the effect of subroutines when the ui is just a series of web pages.” — Paul Graham

A closure is simply a function within a specific context. When you use Common Lisp’s lambda to create a new function on the fly, you can pass it any number of variables that are currently in scope. The resulting function will “remember” those values regardless of what else happens in the program.

I must admit that I don’t immediately see the full extent of the power this concept. I do know that in object oriented programming as I’ve done it in the past causes me to define numerous small classes each with their own private fields, constructors, and Get/Set properties. It can be a pain to constantly write these out each time you want to add a new “type” your the application. With Lisp, you can skip all of this busy-work and focus on the part of the program that actually does something. With closures, state can be hard-wired into dynamically generated functions– because in many situations there’s really no need to specify an entire class.

Working with C# 1.0, I had to use an entire file to make a strongly typed collection class. With “generics” introduced in C# 2.0 I could create them with a single line of code. Obviously an improvement! (Though, being infected by smug Lisper memes I now question the “essential-ness” of compulsively strongly typing everything in sight….) But the point is that anything that eliminates entire files from your project improves the expressiveness, conciseness, and maintainability of your code. Closures are similar to generics in that respect, but they operate at a more fundamental level.

Closures are a concept that exists in a nebulous no-man’s land somewhere between a function and a class. Programmers bred in today’s OOP-obsessed environment will have to unlearn a great many habits and assumptions in order to use them. But with “a hash table full of closures” to define our system… what exactly have we got? We end up in a situation where, in a sense, the properties of our system do much of what classes do in other programming languages. We can forget about an entire level of implementation… and the basic building blocks of our thought processes gain an order of magnitude in their scope and depth. In short, we can think bigger ideas… and implement them faster.

Ah well. As I began writing this, I was disappointed at how most material on Common Lisp glosses over the concept of closures when guys like Paul Graham speak of them in such glowing terms. I was going to write an small article explaining exactly why I could only muster up a low-key “so what?” after reading about them. This post represents an attempt to parse a few cryptic remarks based on only my own meager programming experiences. Take them with a grain of salt!

I’ll see if these “egg-head” ideas actually hold up in the face of getting something non-trivial done with Common Lisp. I’ll report back here on my experiences as I try to apply these ideas….

Update 6/22/07:

After coding a bit, I can see that closures are just one of many elegant tools provided by Common Lisp that allows you to avoid writing tedious and unnecessary code:

Common Lisp gives you the ability to store functions in variables and pass them around to other functions. The mapcar function gives you the ability to apply those functions to each item in a list. Recursion on lists allows you to avoid declaring and managing iteration variables.

Common Lisp gives you access to the evaluator. Functions are themselves stored as lists… and the evaluator can be applied to parts of those lists selectively. This gives you the ability to quickly write dynamically defined functions that are initializable according to the current environment. Closures expand that capability by allowing you to persist parts of that environment along with your function.

Because lisp instructions are stored as evaluated lists that are expressible in plain text, you get a form of “object serialisation” for free. Each data type has its own textual representation… and when lisp objects are written out to text files, they are generally human readable and editable. I imagine this technique existed well before the XML craze and so forth, but even today this is an attractive feature. Unfortunately, closures don’t play nicely with this, so you can only save more mundane objects like lists and hash-tables, but still there’s a lot of power here that comes built in to the language.

Even a mediocre developer like myself can accomplish things that are relatively difficult in other languages. Yes, you will have to write a few “library functions” here and there that you get for free on other platforms– but this is much easier that writing an evaluator and/or a dynamic function generator.  Closures are one of many features of Common Lisp that enable clean and concise dynamic designs without requiring a lot of housekeeping.

Using “Query replace regexp” to remove blank lines from a file in Emacs

June 7, 2007

This guy took years just to discover how to use a carriage return in an EMACS regex search. Yep… just do C-M-s to fire up the search… then ^ C-Q C-J and you’re rocking. Nifty.  For some reason I didn’t think it would work for a while, but the same trick does indeed work from regex search and replace (C-M-%) as well.  How long does it take me to figure out how to apply this trick to an editing problem? It ended up taking… well… more than an hour. (Grumble grumble….) Never mind how long it took for me to realize that this was the key to solving the problem! (Grumble grumble grumble….)

In the hopes of preventing similar pains in other peoples lives, I will explain my discoveries….

What I wanted to do is delete all of the blank lines in a file. C-c C-o just kills the next chunk of whitespace that follows the line you’re in. I’d like it do the entire file….  (Picky me.)  Many a google a search will lead you to bark up the wrong tree, but Derek Slager’s cool screencast was the place to go all along.

Drop into regex search and replace with C-M-% and for the “where” part, type C-q C-j C-q C-j +. (That’s two carriage returns followed by a plus sign.) For the “with” part type C-q C-j and press return. (That’d be a single carriage return.) What we’re doing is replacing each chunk of 2 or more carriage returns with a single carriage return.

An alternative solution would be to replace ^ C-q C-j with “nothing”.  (The caret matches all lines that match the text that follows it.)

Pretty cool!

I forgot what I was trying to do before I got sucked into this, but now that I’ve solved this minor gotcha there’s nothing to prevent me from mastering Emacs in another ten years or so! Yee-ha!

Too bad I’ve got carpal tunel syndrome….  (Oh the pain!)

Maybe Those “Deliberately Chosen Limitations” Weren’t Such a Good Idea…

June 6, 2007

“Yet, the most popular form of software patterns is exemplified by those found in Design Patterns, by Gamma, Helm, Johnson, and Vlissides, which contains little more than techniques for coding in C++ constructs found in other programming languages– for example, 16 of the 23 patterns represent constructs found in the Common Lisp language…. Down with quality, up with clever hacks. Why worry about what makes a user interface beautiful and usable when you can wonder how to do mapcar in C++.”

— Richard P. Gabriel in “Back to the Future: Is Worse (Still) Better?”, 2000

“This practice is not only common, but institutionalized. For example, in the OO world you hear a good deal about ‘patterns’. I wonder if these patterns are not sometimes evidence of case, the human compiler, at work. When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I’m using abstractions that aren’t powerful enough– often that I’m generating by hand the expansions of some macro that I need to write.”

— Paul Graham in “Revenge of the Nerds”, May 2002

Peter Norvig (of Google fame) appears to have done the initial work on identifying the 16 Lisp constructs that correspond to the Gang of four patterns. Here’s his talk where (i think) he first made that point back in March of 1998. He reiterated these points at a presentation in October 1999 and he said that Lisp is the best host for first class patterns.

What blows my mind about all of this is that these discussions are all taking place in the nineties. The nineties! And in spite of that knowledge, we end up today in pretty much the same place even in 2007:

“More than half of the code in every Java enterprise framework exists purely to work around well-known, deliberately chosen limitations at the language level. Smart Java developers have paid a staggering price to prop up the illusion that the Java language is easy.”

Stuart at Relevance, LLC

For the past eight years I’ve struggled to make my programs as dynamic and configurable as possible. Many times I’ve thought to myself, “what I really need to do is write my own language for solving this type of problem.” I’ve literally thought that a half dozen times before reading a word of Paul Graham’s essays…. The idea seems so natural, I always wondered why it felt like I was the only person in my niche that wanted to do that. And it seems that the answer to that is that the development environment I was locked into was specifically engineered to make it difficult to do those things–it even made it difficult to even imagine doing those things.

This is pretty disappointing….