"Linux Gazette...making Linux just a little more fun!"

The Cash and the Calling

By

September 1999

This paper analyzes a model of software development in which closed-source applications make use of open-source artificial intelligence parts. We begin by observing that AI has a huge potential but that problems limit the development of applications. We consider why there will continue to be closed-source AI applications and note that pure open-source development is limited by the number of people interested in starting open-source projects. We consider the possibility of closed-source applications based on open-source parts, both in a two-tier and a three-tier architecture. We look at the pool of talent available for open-source projects. We conclude that the use of open-source AI parts may significantly increase the development of AI applications and that this may be good for the state of the art of AI.

Table of Contents

Introduction - Potential Applications of AI
Closed-Source and Open-Source
Two-Tier AI: Closed Application, Open AI
Three-Tier AI: Application, Problem-Domain, AI
The Pool of Talent
Conclusion - The Potential of Open-Source AI Parts
Bibliography and Acknowledgements

1. Introduction - Potential Applications of AI

There seems to be more potential applications for artificial intelligence than actual. The untapped market for staff-scheduling alone is immense. The opportunities in the areas of materials scheduling, process optimization, expert decision-making and image interpretation seem endless. The development of AI applications would appear to be a field with a lot of potential for growth.

Presumably there is money to be made satisfying some of this potential demand. Why is this happening so slowly? Why are there so many potential products that people would pay for but that have not been developed?

There are a number of reasons why there are so many potential, as opposed to actual, applications of artificial intelligence:

AI software is expensive and risky to develop. It tends to be complex and subtle. It generally requires a large investment in problem analysis. More AI software would be developed if it were cheaper to do so.
AI applications tend to require a lot of support and they tend to have a lot of future-risk. It is particularly difficult for a tiny software venture to develop and market new AI software. More AI applications would be developed if getting involved with them was safer.
Some problems are sufficiently hard that no known AI technique is completely satisfactory. More AI would be used if trying to do so was more likely to be successful.

Artificial intelligence is limited by the expense and risk associated with trying to take advantage of particular opportunities.

2. Closed-Source and Open-Source

There will probably always be opportunities to develop closed-source AI software for rent. Companies identify potential applications, and where there is sufficient expected demand, they develop new products. This is expensive because of the amount of analysis and design required. The development only occurs because rents are expected. The source is closed to enable capture of those rents.

As we have noted, however, this sort of scenario is limited by the expense and risk involved.

What about open-source? Free should mean less expense. Open-source reduces future-risk. AI would definitely seem to be a good candidate for peer-review. Eric Raymond describes these benefits and how they come about in "The Cathedral and the Bazaar".

But each open-source project must be started by someone who does the initial analysis, design and development. There are a lot more potential AI projects than people interested in starting them. Open-source application development isn't likely to make a big dent in the pile of unexploited AI opportunities.

3. Two-Tier AI: Closed Application, Open AI

The expensive part of an AI application is not necessarily the AI. There are a variety of artificial intelligence techniques, tools, frameworks and engines available. The most expensive part of developing an AI application can be the problem-analysis and the design of how the AI is to be used.

It may well be reasonable for an application, based on expensive analysis and design, to be closed-source. But what if the application got its AI functionality from open-source AI parts? A staff-scheduling system could be based on open-source AI problem-solving parts. An image-recognition system could be based on an open-source neural-network.

In "The Magic Cauldron", Eric Raymond describes five discriminators that "push towards open source". The first four discriminators indicate that AI parts would be a good candidate for open-source:

Reliability is a major concern
Verification of design is difficult
AI applications can be critical to business processes
AI parts establish/enable a common computing infrastructure

The fifth discriminator, however, indicates the opposite. Artificial intelligence is not part of "common engineering knowledge". It is an area in which one would expect good proprietary techniques be able to generate good rents.

In practice, this can be difficult. The customers for software parts are developers of other software. Convincing a potential customer of the worth of a secret technique can be a tough sell. But more importantly, a company will not be interested in having its product dependent on a secret technique that may not satisfy future requirements.

Open-source software parts offer much less risk. They are easier to judge, they tend to be more reliable and customers always have the option of making their own changes.

4. Three-Tier AI: Application, Problem-Domain, AI

If open-source general-purpose AI parts are available, an interesting new product is possible. People can use the general-purpose AI to develop parts that are specific to a problem domain like staff-scheduling or courier-dispatching. This can make for a three-tier architecture -

The open-source, general-purpose AI parts
The problem-domain-specific AI parts
The application/interface

Much of the analysis and design goes into the middle tier - the problem-domain-specific AI. It can be expensive to develop and the market is much narrower than the market for general-purpose AI.

A middle-tier AI product might be developed by a company who will use it to develop an application for sale. In this case, the middle-tier would likely be closed-source.

A middle-tier AI product might be developed by a company or individual with the intention of offering application development services to narrow markets. The middle-tier might be closed to help capture the market or open to help sell the service.

The three-tier architecture provides more ways to take advantage of AI opportunities. The development of middle-tier AI products is encouraged by the existence of open-source general-purpose AI parts.

5. The Pool of Talent

Many programmers are interested in AI. It's an intriguing field - problem-solving, decision-making, remembering and recognizing... it's the ultimate challenge - software that thinks.

The vast majority of these programmers never apply their talents to AI - they have no opportunity in their jobs and they are not part of the academic AI community. The Open-source phenomenon provides a number of ways of tapping this pool of talent.

Programmers with a calling and/or a desire to make a name for themselves will do original research, write new open-source software and start open-source projects. Much unconventional thought will be brought to bear on various problems in artificial intelligence. Many thinkers will have a higher opinion of their thoughts than will later prove to be justified (your present author probably included). But the effect of the open-source movement on the state of the art of AI may be the next great thing that happens in the world of computers.

Open-source projects need participants - people who contribute time designing, developing, debugging and testing. The open-source culture that supports this participation is described in Eric Raymond's paper, "Homesteading the Noosphere". AI open-source projects should be particularly good at attracting participants.

Open-source AI that is used in commercial products should be particularly attractive to talent. There are a few reasons for this. One is that the the AI has proven to be useful - it is something worth working on. Another reason is that the work of the project is obviously important and the project is therefore an excellent place to make a name for oneself. A third reason is that money is involved, there is the possibility of paying work and the possibility of getting involved in new business ventures. Even people who aren't looking for work like the idea of acquiring knowledge that can be worth money.

If a commercial product uses open-source AI, there is the potential for paying work related to the AI. The product developer pays people to initially make use of the AI and this use may have to be maintained. Customers may require consulting, customization and integration services. The product developer and large customers may fund projects aimed at improving the open-source AI.

If money is being made on a commercial product that uses open-source software, there will be people trying to dream up ways of getting in on the action. People may start third-party consulting and integration services. People may launch a venture to develop a competing product. The possibility of acquiring an equity interest in some new venture has its attractions.

6. Conclusion - The Potential of Open-Source AI Parts

Open-source AI parts may significantly increase the development of commercial AI applications. Such development will become cheaper and less risky. Small companies that would lack credibility as developers or purveyors of closed-source AI could have adequate credibility as users of open-source AI.

Open-source AI parts may also significantly increase the development of home-grown AI applications. Many applications of AI in business are so specific that they will not be developed at all unless they are developed by, or at least for, an individual company for its own use. Development that would be too expensive and risky with closed-source AI products could be feasible with open-source AI.

As the open-source movement increases the application of AI, more time and money will be directed at improving the AI. As the state of the art of AI advances, more time and money will be directed at trying to apply it.

The open-source movement could have important effects on the application of AI.

7. Bibliography and Acknowledgements

"The Cathedral and the Bazaar" - Eric Raymond
http://www.tuxedo.org/~esr/writings/cathedral-bazaar/

"Homesteading the Noosphere" - Eric Raymond
http://www.tuxedo.org/~esr/writings/homesteading/

"The Magic Cauldron" - Eric Raymond
http://www.tuxedo.org/~esr/writings/magic-cauldron/

I owe much of my appreciation for the open-source movement to the writings of Eric Raymond. There are many good things to read at his web-site.

Brian Marshall's Home Page