Stardate 20020728.1258

(Captain's log): Can open source solve the problem of software scaling? I suppose it depends on what you mean by "open source", because different people mean different things by it.

There is an (overly) zealous branch of the OS movement which borrows a great deal philosophically from Marxism. To these people, it is a fundamental truth that software should not be owned. It should be created collectively, and be freely available to everyone. For this group, the only true "Open Source" is governed by the strong form of the GPL, and all other forms and licenses are harmful dilution of the concept.

Fortunately for me, Eric isn't among them, and since this discussion is with him, we'll adopt a much more moderate definition of Open Source. I cannot claim to be fully up to speed on Eric's attitudes towards OS so there's a risk that what follows is an unintended caricature. (If so, I hope Eric will correct me.) The essential characteristics of an Open Source effort are as follows:

• A substantial part of the labor is volunteer.
• Those working on it are physically distributed, communicating electronically.

This differentiates from the case of a commercial package which is distributed including its source; that isn't really the same. It also doesn't include parodies of Open Source such as claims by a certain company that its willingness to provide the source of its operating system to certain other companies under a non-disclosure agreement (NDA) is "open source". Likewise, Sun's efforts on Java only bear a passing resemblance to Open Source. (Usually the term is "Open Source Software" or OSS; an acronym I will use hereafter.)

Within this rather broad definition of OSS, there is no requirement for a Marxist attitude towards ownership. The OpenBSD effort, for example, permits companies to use OpenBSD as part of commercial efforts without being required to contribute back to the effort any changes they might have made (as the GPL requires), and indeed OpenBSD is being used in several commercial products now.

Likewise, there is no requirement for public disclosure. It is plausible that an OSS project would require each participant to sign an NDA before being given access to the source.

We can derive a large number of secondary implications from these basic features of OSS, which turn out to be critical.

The most important is that the volunteers contributing the labor are self-motivated. They work when they want, on whatever they want. They can quit at any time, and owe no obligation to the project. People being what they are, the kinds of motivations are enormously varied. Some are fascinated by the work itself. Some want glory and fame, seeing their name in the credits. Some are motivated by knowing that they've made a difference, even anonymously. Some are moved by missionary zeal; they're on a mission.

Some are even politically motivated. Cult of the Dead Cow is in the (slow) process of creating a meta-protocol which will operate on top of the web which will involve the ability to use anonymous distributed voluntary proxies as a way of interacting with web sites. The stated purpose of the tool is to make it possible for those in nations like China and Singapore to bypass national firewalls intended to prevent access to information which the governments there consider politically dangerous.

By the nature of the beast, OSS projects tend to be consensual rather than directed. Since no-one can be ordered to work on something, the only way anything ever gets done is by someone who thinks it should be done.

While it is theoretically possible to operate an OSS project with access to the code requiring a non-disclosure-agreement, in practice any large OSS project is public. It's impossible to prevent the source from being leaked, and once that happens it can never be recaptured. Projects which try to be draconian about this, which release only part of the source and keep critical parts secret, and which only release test builds which contain timeouts, will generally discover little enthusiasm by volunteers and will fail to recruit a large team.

At all levels, OSS is very susceptible to Brook's Law. In the book "The Mythical Man Month", Fred Brooks made the statement that adding people to a late project makes it later. Like all such rules this is not absolute, but his point was that new people have to be trained before they can be productive, and they have to be trained by those already on the project. Thus before any new hire can become productive, they will consume time of existing team members and may not contribute back as much as they consume, leading to a net loss and a longer schedule.

This is always a danger in any project. Most projects require participants to have a fair body of knowledge about the project and in some cases that knowledge can be quite extensive and very intricate. When I joined Qualcomm to work on CDMA cell phones, they sent me to classes to learn about CDMA at their expense. I also had to spend a great deal of time talking to others already in the engineering team to learn what was going on. None of that is free. Staffing up any project always has to be done with care to avoid this problem. On most OSS projects, the problems of Brook's Law are avoided by forcing new recruits to learn on their own, sink or swim.

OSS by its nature tends to be reactive rather than predictive. It doesn't look into the future, try to predict a problem which doesn't exist now but will exist then, and be ready with a soluti