"Linux Gazette...making Linux just a little more fun!"

AOLserver - a web development platform

By

Introduction and short history of AOLserver

A web server is not an end in itself - it's only a tool that allows creating web services. This article introduces an AOLserver, an open-source, standards-compliant web server extraordinaire (think Apache on steroids) that runs beautifully under GNU/Linux. We present its history and show how its capabilities make it an ideal web development platform.

First there was a company called NaviSoft founded by a couple of wizard Unix programmers. They've set out to create the best web publishing system on the planet and they were well on their way. It was 1994 and their product, a web server called NaviServer, was multi-threaded from the ground up (Apache is still trying to catch up with this particular feature), had a tightly integrated scripting language, good extension API and database connectivity built in. It was so good that AOL decided in 1996 to use it to power the core of their business - multiple AOL web-estate. But buying programs is not how AOL works - they've bought the whole company to make sure that the software will grow as fast as AOL's needs. Thus NaviServer has been renamed to AOLserver. In 1999 an MIT researcher, accomplished photographer and web-developer in one person, Philip Greenspun, convinced AOL that it should open-source AOLserver for the mutual benefit of AOL and public at large. So they did. After a few months of frantic code cleanup AOLserver 3.0 debuted as an open-source web server whose development is largely community driven. AOLserver 3.1 has been released in September 2000.

Qualities of a good web server

A web server is more like a C compiler than a word processor. It's a tool, a development platform that is used to build web services. Let's examine some of the desirable qualities that a web server should have and see how well AOLserver fits the bill.

Robustness, stability, scalability

I list robustness in the first place because if your web server crashes and your web pages are inaccessible to visitors then nothing else really matters.

AOLserver is robust, stable and scalable, after all it has been perfected for years by a tight group of programming wizards. Fortunately, you don't have to take my word for it. AOLserver has been battle-field tested in the most demanding environments. It is known to serve 30 thousand hits per seconds on AOL sites. ArsDigita, a Web development company, has built numerous web sites (www.photo.net, www.scorecard.com, www.away.com) that routinely serve millions hits per day. The bottom line is: AOLserver proved to be extremely stable and scalable by serving some of the most popular sites on the Internet today.

Features, a lot of them

For serving static pages AOLserver is just as good as any other fast, efficient and reliable web server. Today, however, publishing static content is not thrilling anymore. The focus is on dynamic, collaborative services. AOLserver has been built from the ground up for such purposes and it shows. Here's a list of features that a web server should have, why it should have it and a description of what AOLserver offers.

You want web server to support four of the most popular programming paradigms. You can think of a web service in object-oriented terms: a web server is an object whose methods are URLs. Users invoke methods by requesting an URL from within their web browser. The simplest way a web server can respond to such request is to fetch a static HTML file from the file system and send its content to an user. Doing anything more complicated means running a computer program which will generate an HTML page. One way to do it is by extending web server itself using its C extension API. Every popular web server (Apache, IIS, Netscape) has some implementation of this idea. AOLserver provides well thought out C extension API that makes writing modules that extend its functionality really easy. As an example: the source code embedding PHP in AOLserver is half the size of the similar code that embeds PHP in Apache (this "benchmark" should be taken with an appropriate grain of salt but it gives a rough idea of extension API quality). Having praised the API it should be noted that this is a very slow and error prone way of writing web services (on any web server). You have to code in a very low-level language (C), debugging is a nightmare and smallest mistake can crash the whole server.

To remedy those shortcomings a CGI protocol has been created. This protocol is supported by all major web servers including AOLserver. The idea is very simple: upon page request a web server will execute a program and whatever this program will send to its standard output will be sent back to a browser. The greatest advantage is that it can significantly shorten development time since programmer can use the best tool for the job (which usually means a scripting language like Perl, Tcl, Lisp or simply a language that he is most familiar with). The disadvantage is that it's slow (for each request a program needs to be executed and it's very expensive operation) and thus doesn't scale well.

To improve performance web servers started to directly embed scripting languages. The most popular examples are mod_perl in Apache and PHP. Since interpreter is linked with the web server executable it's no longer necessary to fork an external program to execute the script which saves a lot of time. This approach conserves the fast development advantage of CGI scripts but the cost is that it limits you to one particular scripting language (which may or may not be an disadvantage depending on how well developer knows this scripting language). AOLserver's story in this department is very compelling: it is the only major web server that comes with a tightly integrated scripting language (Tcl) out-of-the-box. If for some reason you dislike Tcl you can use PHP, Python, Java (and there is a work in progress to add Perl support).

The last programming paradigm is server-side includes ie. code embedded in HTML pages. When a page is server it is parsed by the server, HTML code is left untouched, embedded code chunks are executed and replaced by their output and resulting page is sent to browser. The most popular example of this paradigm are ASP pages in IIS. AOLserver provides developers with a similar feature: ADP pages that allows you to embeded Tcl code inside HTML.

You want your web service to be fast and efficient. AOLserver's multi-threaded architecture gives you the performance advantage over process based web servers (eg. Apache 1.3.x, Apache 2.0 is being rewritten as multi-threaded server but hasn't yet reached maturity). If a web server is based on processes it has to create a new process to serve each http request. AOLserver only has to spawn a new thread and it's much faster.

You want to have ability to share data between scripts. There are many uses for such feature, the simplest example would be to count how many times a given page/script has been called. It's not easy to achieve this in a process based web server because processes do not share dynamically allocated memory. As an example, if you use mod_perl scripts in Apache, you can have a global variable and increment it but if you think that this will tell you the total number of times the script has been executed you're in for a surprise: this value is actually per-script-per-process and since Apache pre-forks multiple processes and (as far as programmer is concerned) unpredictably assigns them to execute scripts your counter won't give you an accurate number. It's possible to overcome this using shared memory (or by storing the data in a database) but it is so cumbersome and non-standard that it's not popular among Apache developers. In AOLserver it's a child's play thanks to the fact that threads share dynamic memory and excellent built-in nsv interface.

Since dynamic web services usually have to store data in a database and retrieve data from a databases you want your web server to have an ability to talk to databases. AOLserver comes with an exceptionally good, standardized and fast database connectivity API. It's fast because it uses connection pooling, ie. database connections are opened when the web server starts and subsequently reused among scripts. An alternative (used eg. in PHP3) is to open a connection on the beginning of a script and close it at the end. This approach is much slower.
Standardized means that you use the same API to send SQL commands regardless of the database server used (to contrast, in PHP each database has its own set of APIs). It's easier to port the code when switching databases (SQL statements still need to be ported, but that just shows how standard an SQL standard is).
To top it off drivers exist for most popular databases: Oracle, PostgreSQL, MySQL, Informix, Interbase, Solid, DB2.

AOLserver provides developers with more basic blocks for building dynamic web services than any other web server. Most web services have to solve many similar problems and provide:

user and session tracking
permissions system
content management (publishing, templating)
e-commerce
banner management
site-wide search
and many, many more

Even given all the wonderful features of AOLserver you might get tired reinventing the infrastructure needed for supporting collaborative web service. Instead of re-inventing the wheel you might take a look at open-source, Oracle-based ArsDigita Community System (ACS) (or at OpenACS, port of ACS that uses open-source PostgreSQL database). You'll have to spend some time in learning about this very extensive toolkit but after an initial investment you'll be able to quickly build high-quality web services that require advanced collaboration features.

Documentation

Good documentation is essential. More often than not it's also the most frequently neglected aspect of open-source projects. AOLserver, thanks to its commercial roots, has a very good documentation, available both on-line and off-line. It consist of an administrator guide, detailed guides to C and Tcl interfaces, a guide to developing database drivers.

Vibrant community

An active community of users and developers is important for steering programs into new waters, providing suggestions and bug fixes, helping new users etc. AOLserver has entered open-source land relatively recently and has already gained an attention of external developers. There are projects in full swing that push AOLserver into realms it has never reached before. The following modules are being worked on by community members (independent of the core developers):"

quickly maturing PyWX project embeds a Python interpreter inside AOLserver so that it enjoys the same high integration as Tcl
nsxml module contributed by ArsDigita adds XML processing capabilities by wrapping libxml in a Tcl interface
nsjava enables calling Java from within AOLserver
nstomcat enables running servlets inside AOLserver
first stab at embedding Perl has been made

Conclusion and links

Web is changing rapidly. Static pages and simple CGI scripts are not sexy anymore. Anything worthwhile that can be achieved on the Web today involves building web pages from dynamic data sources, collaboration among visitors, tracking user's behavior etc. AOLserver's features make it an ideal development platform for building such advanced, collaborative web services.

To get more information check those sites:

www.aolserver.com is the best place to get more information about AOLserver and participate in its development
Philip Greenspun has written an excellent introduction to AOLserver. Here's part one and part two of his article
this paper shows how AOLserver is used to power some really heavy-weight sites
when you get past simple web sites you might want to read Philip and Alex's Guide to Web Publishing to find out what designing web services is about