Stardate 20020728.1258

(Captain's log): Yesterday morning, I met Eric S. Raymond and had breakfast with him. He is the first person I've met face-to-face as a result of writing this blog. In addition to his numerous activities involved in the Open Source movement, Eric also runs his own blog, which like mine concentrates primarily on writing rather than on linking. Eric refers to these as "essay-blogs", and I'm happy with that term.

On Friday, we were exchanging email about some things, and he wrote and asked me where I was physically located, and when I told him I was in San Diego he wrote back to say that he was in San Diego at that very moment. He was here to attend an O'Reilly conference. So we hurriedly arranged to meet for breakfast yesterday, just before Eric caught his jet back home.

I felt very comfortable talking to Eric. He and I think a lot alike in many ways. (Which, I suppose, should have been evident by comparing our respective writing styles, but that's not as obvious as it might seem.) We didn't talk about shoes and ships and sealing wax, but we did talk of many things. I'm glad I met him, and I think we may become friends.

Which is not to say that I agree with him about everything. I suppose it was inevitable that we should eventually talk about open source, that being Eric's life, but it only came up late. He made a provocative statement that Moore's Law was decreasing the cost of computing at an exponential rate, and that software packages were growing as a result, and that the process of development of software wasn't able to keep up. He then said that the only solution on the horizon was Open Source.

There was a beat. Then I said that I wasn't sure I agreed with that. He asked me why, and there was another beat. Then I said that I needed to think it over, because the answer was complicated, and maybe it would be better if I posted it. Welcome to the result.

I presume that Eric will want to respond to the points I'm about to make, presumably online, and if so I will certainly link to him and probably respond.

So first let's state the fundamental problem: as the size of a software package grows, the tendency for bugs to appear in it grows even faster. There are several reasons why, but they all come down to the fact that a rising proportion of the bugs are interactions between different sections of the code rather than microscopic implementation mistakes in the code itself. It's not so much that the code doesn't do what the person who wrote intended, as that the design itself contains faults because different parts don't play well together.

Left to itself, this would place an effective upper limit on the size of software, because at a certain point the packages would be so unreliable as to be useless, even if delivered on time and having all required functionality. The kinds of factors involved in making computer hardware cheaper over time don't apply to software, and for most kinds of applications of software, the cost of the software as a proportion of the whole engineering effort has been rising for a long time, and is expected to continue to do so.

One of the reasons why is that software as a product discipline and form of engineering has unique properties. By far the most important is that almost the entire expense associated with software is in engineering. All software is cheap to manufacture, and the value to the customer (and price the customer is willing to pay) has no correlation at all to the cost of manufacture.

The only other field I can think of where that's true is in pharmaceuticals, where the process by which a drug is made, and the cost per dose, have little to do with the clinical value of the drug and thus how much it would be worth based on what the market would bear. The big difference is that drug manufacture tends to respond to economy of scale, so drugs which sell in high quantities can be produced more cheaply.

Software manufacture largely does not respond to economy of scale, simply because it's already cheap even in small quantities. On the other hand, software as a product is even more susceptible to economy of scale because more units sold provides a bigger base over which to amortize the one-time engineering costs.

Software thus lends itself uniquely to product differentiation by competitors. Two products with identical manufacturing costs can be perceived as being radically different in value by potential customers as a function of the code which is incorporated into them. And since the cost of computing has long since reached the incredibly cheap, it's almost suicidal now not to include computers in your product if there's any way you can. It's been possible to buy greeting cards with computers in them for ten years; and now they're so cheap that they're even being included in the packaging for laundry detergent for advertising purposes. We are surrounded by immense amounts of software that most of us don't even perceive, and the industry has been two steps away from crisis for years because of the problem of scaling.

Can open source solve the problem? That's what we're here to discuss, in this multi-part article.

Part 2
Part 3
Part 4
Part 5

This is about software, written by a programmer primarily for other engineers. I am not going to try excessively hard