Gosh, that’s scary! They don’t use friggin source control??? Gaagh!! Well, I gotta admit. When I was working a job on my own a couple years ago I didn’t use source control either. I just ZIPped up my source directories and put a version number on it. It’s not the way I prefer to work, but all the source control systems I knew about at the time were commercial, and I wasn’t sure if I wanted to put down the dough to get one. I’d heard of CVS, and I had used it before, but didn’t particularly like it.

There’s a generation or a culture gap. None of them have this attitude of experimentation and exploration built in as a component of every development project they undertake. No immediate perceived business value, I guess.

More like they don’t know how to do it. Believe me, it’s not because it doesn’t need to be done. It’s not merely a “right way” orthodox approach either. Alan Kay has talked some about this, that the tech culture is rather primitive, even though it pretends it’s “up-to-date” and “advanced”. There is some consideration now for how people interact with computers, but not enough. None of the popular development environments allow you to do what you’re doing. They’re all early-bound environments.

What surprises me a little is that nobody you’re talking to can even relate what you’re talking about to something they know. I remember several years ago seeing articles in .Net programming magazines (maybe like two or three) that talked about extending your .Net app. via. scripts. I knew a Visual C++ developer (coworker) who talked with me about doing that with C++ back in 2000.

I have met developers who know about the idea of creating “data driven” apps., that is creating an app. framework that responds to parameters entered into a database repository. So there are analogies around.

I’m getting this image that these people you’ve been hanging around are “plug it together” developers, who know how to deal with component-based frameworks, but that’s it.

But maybe there is no need for me in this setting. Maybe it’s just overkill.

I can understand why it would feel that way. I think that’s what a lot of typical developers would tell you, “You’re making it way too complicated for what it is. It doesn’t need all this.” I think a quote from Alan Kay addresses this concern well. This is from his 1997 speech, “The Computer Revolution Hasn’t Happened Yet”:

“I just played a very minor part in the design of the ARPANet. I was one of 30 graduate students who went to systems design meetings to try and formulate design principles for the ARPANet, also about 30 years ago. The ARPANet of course became the internet–and from the time it started running, which is around 1969 or so, to this day, it has expanded by a factor of about 100 million. So that’s pretty good. Eight orders or magnitude. And as far as anybody can tell–I talked to Larry Roberts about this the other day–there’s not one physical atom in the internet today that was in the original ARPANet, and there is not one line of code in the internet today that was in the original ARPANet. Of course if we’d had IBM mainframes in the orignal ARPANet that wouldn’t have been true. So this is a system that has expanded by 100 million, and has changed every atom and every bit, and has never had to stop! That is the metaphor we absolutely must apply to what we think are smaller things. When we think programming is small, that’s why your programs are so big! . . .

[The] way to stay with the future as it moves, is to always play your systems more grand than they seem to be right now.”

The point of his speech is just what you’re talking about: Most developers don’t think like this. We need to! What he was saying is by thinking “this is all I need right now” for a small app., you’re dooming yourself to creating a huge mess if and when your “small app.” needs to become huge. By thinking about architecture ahead of time, even for small problems, you’re ensuring that your solution can scale up.

One of the things Kay is really into is creating systems that are malleable enough to deal with change well, and are able to go from small to huge without major disruptions.

I get the sense that your solution has kind of been hacked together. I don’t know. Maybe you just think it’s hacked together, but you’re using a good enough architecture so it won’t create problems. You could run into limitations with your approach eventually. That’s the price we pay if we don’t think about architecture. It sounds though that you’re getting “real bang for your buck” because what you’ve done is better than what anyone else there has tried. This isn’t surprising, but a compliment to what you’ve learned.

It merged with some of my homegrown tools that I used daily and ate part of a pet project I’d been spending a few week ends on… and serendipitously, I just addressed a dozen minor requirements with an idea that came out of nowhere. I hardly even decided to make an application– it just happened. It’s a glorified proof of concept at this point… but the question is… what will happen over time as I face more wide ranging and more thorny problems… and grow into them… and discover new opportunities that I never thought of.

This is the reason I brought up the issue of scale above. If you’re using good architecture it’s more likely to scale well to those future unknowns than what anyone else is trying.

I know you’ve told me your opinions of OOP before, and I’m not trying to sell you on it here. I talked with Alan Kay briefly about programming in general via. e-mail last year, and he went on at length about the importance of architecture, and asking questions about the relationships between components of a system. He talks about OOP some, but his main point is thinking about architecture, which could just as easily be non-OOP:

“I would characterize the ARPA [work we did in the 1960s] (and then the concentration at PARC) as mainly interested in a ‘no-centers’ style scaling architecture (personal computers [as opposed to centralized mainframes], Internet, Ethernet, objects, ‘no OS’ [as it's traditionally defined], ‘no Apps’ [they created 'widgets'], etc.).

If we think of a big computer system as necessarily being made out of components at one or many levels of scale, then we really want to be asking questions about the relations between them. This will come down (partly, but in a pretty large way) to what the components know how to do vs. how much they have to be told how especially in the moment. Data structures don’t know how to do anything and have to be told all – this doesn’t scale – so I abandoned this idea in 1966. Objects can know quite a lot and thus don’t have to be told much. We did not go nearly far enough in this direction in the 70s (in no small part because of the weak and small computer resources — and our weak and small minds).”

So if you want to take a look at a software system, you could look at its fundamentals: data access/serialization, rules, system communication, human-computer interaction, etc. and ask questions (to yourself I suppose) about the relationships between them, and then write bits of software that will deal with those domains, and facilitate the relationships between them, bringing in the same properties that you’ve come to like about your current creation.

Another thing Kay communicated to me is that software architecture is the area of computer science where the most research is needed. CS in academia tends to not talk about architecture much, so not much work has been done on it. I say this so you won’t feel bad if you feel inadequate/overwhelmed by the idea of trying to form a better architecture for whatever large system you’re looking at. One of the things he said he’d like to learn is how do people learn architectures and abstractions. So even he doesn’t have a good answer. His advice seems to be to just go for it and give it your best shot. You’ll learn something valuable from it, and I would add, “even if you fail,” though I understand perfectly well that failing is a scary prospect.

My dream is to take the scripts generated by the power users and integrate them into my solution somehow. Basically… I’m bringing the spirit of the EMACS community directly into a crappy database application. I don’t think any of the dozens of people I’ve talked to the past few days would even know what I was talking about if I tried to discuss this with them… and even if they could they would not see the business value of it.

Maybe it isn’t “worth it”. For me, though… to be able to go to those little 5 minute demo’s… and to be able to answer their criticisms and issues right when they’re looking over my shoulder…. Now we can go on and deal with the real issues instead of getting stalled. And if that stuff’s what we want in the release, all I do is kick out a script for what we just did and fold it into the scripts that initialize the system. No recompile. Or I could send a test script for the power users to beta for me first if I wanted….

None of the vendors here can do that. One application we use forces us to maintain modifications by hand that could have had something like what I’m working on, but there’s no vision for anything like this sort of dynamism/programmability.

I’m so far out of the mainstream… I don’t think any of the hundred people that I’ve talked to the past few days could even listen long enough to get to this. None of these people have even heard of Unit Testing or Regression Frameworks… and many of them do not use source control. There’s a generation or a culture gap. None of them have this attitude of experimentation and exploration built in as a component of every development project they undertake. No immediate perceived business value, I guess.

So it gets back to how I want code. I can set up my architecture however I want…. Why not do so in a way that makes everything easier for me and that puts so much more within the 5 minute zone??

But maybe there is no need for me in this setting. Maybe it’s just overkill. I wish I could be two people so I could handle more of the hard core business stuff on top of this. But keeping up with all of the applications and support and technologies that are thrust on me is plenty as it is. Yet, all of those issues that I contributed to eliminating… that were never on anyone’s to-do list because they weren’t even something that anybody imagined could be addressed with a little software… that *did* have an impact on our business. The pat on the back may not have been there– and we’re back to “what have you done for me lately”–, but it was significant what I did.

I think the thing to do is continue to follow my destiny even though no one understands. I’m mentally in shock because there’s a huge need in front of me now that’s many magnitudes greater in scope than the simple problems I’ve just solved. I can address it without transforming into a CFO/CPA type. My work style is very unorthodox, but once I understand something and codify it, I never have to think about again. (“Good enough” for most people is nowhere close to that.) There’s just a whole new class of users that I have to engage now and the relationships aren’t there yet.

But Mark… don’t blow my solution out of proportion. It’s just a start. It emerged out of nowhere as something that grew on top of a very simple application that just needed to print some stuff out. It merged with some of my homegrown tools that I used daily and ate part of a pet project I’d been spending a few week ends on… and serendipitously, I just addressed a dozen minor requirements with an idea that came out of nowhere. I hardly even decided to make an application– it just happened. It’s a glorified proof of concept at this point… but the question is… what will happen over time as I face more wide ranging and more thorny problems… and grow into them… and discover new opportunities that I never thought of.

I don’t even know what the next thing is beyond the very next order of magnitude refactoring and extension…. No idea at all. Why is it that I feel like I have the most comprehensive plan for software development of anybody on my block… but I just know that to the normal people here I’d sound like I’m just some crazy guy that doesn’t know what he’s doing?

I think… maybe… the fact that the problem is indeed so hard… maybe that mean’s that contrary to “common sense”, my approach really is the best fit for this. The status quo for my industry is so far from what’s tolerable to me… and if I’m going to be responsible for it, I have to take control of it and put it in terms that I can assimilate.

It’s a *right way* mentality dropped into a maelstrom of needs, unknowns, and entrenched mediocrity. You pick your battles and bide your time, but I think there is value in it even if nobody else around me can see it until it’s five minutes away.

Perhaps what you are describing is another angle of this trend I’ve heard about for a few years, that what businesses really want is someone who is technically knowledgeable and has some domain knowledge as well, like someone with CS knowledge, and understands how the railroad industry works, too, for example.

Your description of a lack of a common language, even within groups, is real interesting to me. The first thing that came to mind was “Design Patterns”, written by the GOF. In the introduction I remember they said what they were searching for when they came up with DP was what they have in the field of architecture (as in buildings), called a “pattern language”. Because of the book they wrote there is now a recognized pattern language for certain structures. From what I’ve heard though the particular patterns that were described in the book have become canonized, and some of the message of the book was lost. The point was not “these are the only patterns you will ever need”. It was supposed to be a “conversation starter”. The authors derived those patterns from constructing a word processing application. The fact that these patterns have been found useful in many other types of applications shows that they did some good work in finding a new architecture that improves what we do, but there are many more patterns to be discovered.

What I’ve thought about from time to time, prompted by a brief conversation I had with Alan Kay by e-mail, is that the very programming languages we use are pattern languages. What most programmers miss is they are in fact using an architecture with every line of code they write. Instead they use these languages as though they’re standardized communication protocols, both for the computer and each other. We know somebody came up with them, but we don’t seem to care how, or even if they’re appropriate to the task. In any case they’re godly smart, we assume, and so we should just use them without question.

I think a big part of this is the majority of programmers are never asked to learn this stuff. They are presented with frame-based solutions, which can be reliably used for certain classes of problems. As the name implies they also have boundaries, which are comfortable to people who are used to case-based learning.

Even our modalities in software design are themselves designed around frame-based solutions. Many expert developers are only familiar with the terminologies used for them, those particular pattern languages.

Really innovating with design means realizing that there’s a world beyond the frame-based solutions, and instead of trying to shoehorn every problem into these frames, you can ask the question, “What are we really trying to do? What are we as an organization really trying to be?” With your training in SICP you can take a modeling approach to real problems. I think that’s what you’re seeing. You can literally model portions of a business, using a pattern language/architecture that’s appropriate to the domain, and express it in a language of your own making. Secondly, you can see the a computer as a dynamic simulation environment, and as such, you should be able to a) experiment and dabble in ideas, to try to find the best solution to a problem, and b) see results of changes quickly and easily. That’s a tall order to most people, but it’s an idea that I find really interesting. I’ve yet to try it out. It’s only something that’s been gelling in my head.

After talking to dozens of IT Managers, Developers, and CFO types the past few days, I can see that things are much worse than I thought. There simply isn’t enough common ground between these groups. Even savvy developers that I admire can have radically different terms of viewing things… and we all suffer from a collective dearth when it comes to having a shared language for discussing architecture and software design.

The only way forward is for someone to take initiative and gradually prepare themselves (and their code) for the day when they can open things up with a substantial proof of concept. You have to be addressing real needs at every step and you have only five minutes to walk anyone through a given use case. The attention spans and the default fear is just too great.

Even for myself, seeing this and knowing this… there are so many practical business concerns that are impacted by the stream of business data– and no one else is in a position to address them. This is why the CPA that picks up technical skills and/or the tech that picks up business skills can end up becoming the de facto controller of the company. No one else is in the position or has the leverage & connections from which to act decisively. There’s a whole world of action there that I have yet to dedicate myself to…. I’m not even sure I want to go down that path. But if someone carried the same intensity to the data and business processes that I take to the code architecture….

It sounds to me like you take this up another level, to not just improving code, but improving how you create solutions as well.

The way you are writing this makes it sound like you are not working as a group member, but just finding solutions to company problems by yourself, and doing it effectively. Am I right?

Reading Giles’s response was interesting to me, because he kind of had me pegged on this as well. I’ve hoped that there are enlightened places to work, but wondered, “Where the heck are they? How do I find them?” The same question you’re asking. The image I have is that these are places that I’d find tough to get into because they’ll always ask what I call “the genius puzzle-solver” test questions. They won’t ask me to code something. Instead they’ll have me solving tricky math problems, which I find rather infuriating, because I feel like, “Look. How about you ask me what my philosophy of computing is and its relationship to society, or ask me about the philosophy of programming with objects, or ask me to define what programming is, or have me define my own metalanguage for data access, etc.?” I’d even be willing to entertain some lambda calculus problems. You know, something in relation to computing as opposed to testing my puzzle solving skills!

This is something I realized that I think is wrong with many of the practitioners in our profession: We’re all trying to perfect our puzzle-solving skills. Software development is not just a problem-solving/puzzle-solving exercise. Sure, these are valuable skills to have, but I now think that if puzzle solving is our only skill then what we’ll produce is exactly what you see most of the time: poorly designed hacks.