Stardate 20020728.1258

(Captain's log): Of the non-comprehensive list of major software categories I just listed (embedded, vertical apps, short-life commercial apps, long-life commercial apps and software infrastructure) OSS has had its greatest success in software infrastructure.

It's hardly surprising; it's what the OSS developers understand best, and they're most motivated to create products that they themselves want to use.

In addition, software infrastructure tends to not require timely delivery, and the fundamental problem is reasonably stable (compared, for instance, to the market requirements for the products that Carmack creates and sells).

The strengths of OSS apply well to this area, and most of its weaknesses are unimportant. But that's not true for any of the other major product areas. In most cases, an attempt to use OSS would be commercial suicide.

For nearly everyone, the possibility that some participants in the OSS process might be hostile and use the process for sabotage is a major concern. In fact, it alone may be a sufficient reason in most cases not to take the chance. However, there are many other issues involved for most of them.

There are two major conceptual ways OSS could be used commercially, which I'll typify by use of Apache and Mozilla as examples. The Apache project is almost entirely run at all levels by volunteers, whereas in the Mozilla project there is a rather large core of commercial developers who are relying on a halo of OSS developers to augment their efforts.

The Mozilla approach is much more likely to make sense to businessmen, because they want their own people doing the fundamental design of the system. Still, there are problems.

For embedded software, OSS has the following problems:

• It can't be scheduled; timely delivery can't be relied on.
• Debugging requires access to custom hardware which usually can't easily be accessed across the net.
• Active participation even for junior people requires substantial amounts of project-specific knowledge which isn't easily acquired, especially remotely.
• A great deal of proprietary information is usually involved in the process, and if that's released the company can be seriously harmed.
• It's nearly impossible to do embedded software without common impromptu face-to-face meetings with co-workers, either to ask questions or to brainstorm. Doing this electronically is sufficiently different as to not be practical.

For vertical apps, the objections are:

• Security, security, security. You want me to trust my billing system to code written by anyone who happens to come along and volunteer to work on it, without any kind of check of credentials or checks on trustworthiness?
• Recruitment: for most of the kind of people involved in OSS, vertical apps are boring. (Unless they want to figure out how to steal from it.)
• It takes a lot of knowledge of the specific aspects of the problem to make a significant contribution, which means things like observing the actual process of guests checking in at the front desk of the hotel.
• The industry is full of horror stories of vertical apps which ran badly over budget and over schedule; the idea scares the hell out of business people. They're unlikely to be very enthused by the use of a process which by its nature cannot be reliably scheduled. (Remember that Mozilla ran two years long.)

For short life apps:

• Schedule is everything. If you're six months late, you're dead.
• Secrecy is everything else. If you're on time but your competitor knows what you're doing a year ahead, he'll wipe you out.
• How do you make money selling what anyone can get for free from any developer? If your product was developed out in the open, who exactly buys it afterwards?

For long life apps:

• Will the participants be willing to work on what our marketing analysis says we need, or will they insist that they know what is required and try to add that instead? We don't need feature creep, or people trying to change the direction we're moving.
• There is major learning curve involved in making a reasonable contribution to these kinds of programs; you don't learn how a circuit board router works in a few days of study. In most cases you have to be conversant with the way that the package's customers do what they do, and most programmers don't know these things and can't easily learn them.

These are by no means complete; they mainly serve to show that there are going to be really serious misgivings on the part of businessmen, that irrespective of any perceived potential benefit to OSS there will be serious fears that attempting to use it will destroy their business entirely.

And it's not even clear that using OSS actually will solve the specific problem which kicked off this entire discussion, the increase in the rate of bugs as packages get bigger. For one thing, a lot of these packages are not growing as a result of Moore's Law; it's just that Moore's Law is making the more practical and affordable. They grow in the sense that they are more broadly adopted, but not in the sense that the code involved gets bigger.

But another problem is that OSS doesn't actually deal with the main source of increasing bugs in larger packages. OSS, being reactive rather than predictive, is not good at analyzing the problems of non-programmers, figuring out what they'll need, and then delivering it. There's little motivation by most OSS users to operate like that; they want to work on what they themselves want to use.

For OSS to apply to most of these problems, you'd have to use the Mozilla model, where a core group of paid engineers take care of the overall design and then provide a cadre for a larger implementation and debug effort which depended on volunteers. The main benefit that OSS would provide is in testing, providing a huge number of eyeballs looking at the code and using the product and reporting what they see.

And by that point it's too late, because the best way to eliminate bugs is through a clean design, and the design is done by the core group of paid engineers. By the time the army of OSS developers gets involved, all those interaction problems are already in the design and removing them by debugging and patching becomes extremely expensive. (Remember, each phase later in a project that a bug is found raises the cost to fix it by a factor of ten, cumulatively.)

So is there actually a solution to the problem of software scaling? Actually, there is. But it is a surprising one: software bloat.