The $50 Per Hour Search Engine

Diablo

New member
SearchEngine.jpg


Imagine doing a Google search and getting a bill for $50 an hour — or even $300 an hour. Even worse, that’s $50 for every hour you were even connected. Typing a query, reading output, or even grabbing a soda ran the clock. Want to print something? That might cost extra, too. If you aren’t careful, a simple research project could really add up.

That was the world of DIALOG, one of the earliest commercial online information retrieval services. Long before Google, AltaVista, or even the World Wide Web, DIALOG allowed people to search enormous collections of technical papers, patents, news, and other information from a computer terminal. It wasn’t cheap, and it wasn’t particularly easy to use. But for researchers who needed information, it was revolutionary.


From Rockets to Research​


The story starts, surprisingly enough, at Lockheed. In the early 1960s, the aerospace company faced a problem familiar to anyone doing research: how do you find out what someone else has already discovered?

[Roger Summit], who worked at Lockheed, recalled that it was sometimes easier to repeat scientific research than to find out whether someone had already done it. Libraries had card catalogs and elaborate indexing systems, but searching through mountains of technical literature was tedious.


DIALOG databases lived on tapes in a room with a raised floor (Byte Magazine)

Computers seemed like an obvious answer. But in the 1960s, computers mostly worked in batches. And they didn’t have a global network to draw data from, either. [Summit] and others began experimenting with a better approach. Lockheed established an information sciences laboratory in 1964, and by 1966 the team had a small working prototype of an interactive search system.

The hardware was big for its day but modest by today’s standards. The early system used an IBM System/360 Model 30 with just 32 KB of memory. It could search large collections of references on disk and tape and respond interactively.

NASA became an early customer. Lockheed’s work led to NASA’s RECON (Remote Console Information Retrieval Service) system, which allowed researchers to search documentation produced by the space program.

In 1972, Lockheed introduced DIALOG as a commercial service. The idea was simple: organizations could pay to search large databases without building and maintaining their own information retrieval systems.

Having a DIALOG​


Using DIALOG was nothing like using a modern search engine. You needed a terminal, or, eventually, a personal computer with a modem, and a connection to the service.


An image of the DIALOG computer from Byte Magazine

Connecting to the service could be tricky. In the 1970s and 1980s, that often meant dialing into Telenet (not related to the Internet protocol Telnet) or Tymnet, two commercial packet-switched networks that were, in some ways, precursors to the Internet. Instead of placing an expensive long-distance call directly to Lockheed’s computers, you could dial a local access number with your acoustic coupler or modem. The network would carry your session across the country. At 300 baud, about 30 characters per second, you weren’t going to get much for your $50. After all, even if you could saturate your modem (which, of course, you couldn’t), the most data you could pull in an hour was about 108 KB. Of course, everything you typed got an echo, and there were other considerations, so real throughput would never go that high.

Connecting was a multistep affair. You called the local network node, identified your terminal, supplied the address or name of the service you wanted, and eventually reached DIALOG’s login prompt. Then you entered your account credentials and could begin searching. By the mid-1980s, DIALOG also offered its own DIALNET network, and UNINET was another option. All of these networks charged for communications, typically by connection time, on top of the database charges. So your $50 an hour was really going to be more than that.

What these networks did, though, was allow providers to be in many major cities without having to install banks of modems across the country. Early online networks like CompuServe and the Source were accessible via these networks, too, sometimes with an extra charge tacked on.

The name DIALOG was appropriate because the system encouraged an actual dialogue between the operator and the computer. It was strictly a command line. A search produced a set of matching records, which could then be refined by additional searches.

Some of the important commands were BEGIN, EXPAND, SELECT, and TYPE. You could remember them as BEST. BEGIN selected a database. EXPAND displayed neighboring index terms, helping you discover what the database actually contained. SELECT searched for matching records, and TYPE displayed the results.

For example, a 1981 BYTE article shows a query like this:

SELECT PETROLEUM AND PRICES AND OPEC AND PY=1979

The system would find records containing the specified terms, restricted to the publication year 1979.

Boolean operators such as AND, OR, and NOT were essential tools. You could search for one concept, save the matching references as a numbered set, search for another concept, and combine the two sets.

There’s a video below from the Committee on Scientific and Technical Information that shows a 1968 DIALOG session (starting around 4 minutes in).


The Department of Education had a database, ERIC, that lived on DIALOG. Their manual has several versions that are still around online, including this one from 1981. This was sophisticated technology. In fact, some capabilities of DIALOG are still awkward or unavailable in ordinary web search engines.

Tick Tock​


DIALOG had one enormous drawback: cost. According to that 1981 BYTE article, database charges ranged from $15 to $300 per hour, with the most commonly used databases averaging about $50 per hour. Those were 1981 dollars. There could also be telecommunications charges. A 1983 PC World article quoted approximately $8 per hour to connect through networks such as Tymnet or Telenet.
The good news was that you didn’t necessarily need an hour. A competent searcher might find useful results in ten minutes or less. The bad news was that every minute spent figuring out how to formulate a query was a minute you were paying for.

Experienced searchers planned their queries before connecting. They knew which databases contained the relevant material and how to construct efficient searches. An entire profession of information specialists and search brokers developed around these skills.

Even printing was a consideration. At 300 baud, sending a long bibliography to your local printer could consume expensive connection time. DIALOG offered an alternative: have the results printed on its own high-speed printers and mailed to you.

You could even order copies of the original documents through the service. That’s important because DIALOG usually wasn’t delivering complete articles to your screen. Much of its value was in locating bibliographic records and abstracts that told you where to find the information.

In 1983, DIALOG introduced a service aimed at personal computer owners called Knowledge Index. It provided a smaller selection of databases at night and on weekends for $24 per hour, including communications charges.

Searching Isn’t All of It​


DIALOG also offered something resembling today’s automated search alerts. You could save a search strategy and arrange to have it run whenever a database was updated. If new matching material appeared, the service could print the references and mail them to you. Think Google Alerts, except the alert arrived in your mailbox rather than your inbox.

The databases themselves were impressive. In 1981, DIALOG offered roughly 130 databases covering everything from engineering and medicine to business and the social sciences. By 1983, the number had grown to more than 170.

Much of the underlying material was compiled and indexed by specialist publishers and information providers. DIALOG provided the searchable infrastructure that brought these collections together.

Google searches an enormous and often chaotic collection of web pages. DIALOG searched smaller, carefully organized collections of professional information. Neither approach is necessarily superior. They solve different problems.

The Changing of the Guard​


DIALOG ultimately became big business. Lockheed established it as a wholly owned subsidiary in 1981. In 1988, Knight-Ridder purchased it for approximately $353 million. The service subsequently passed through several owners, including The Dialog Corporation and Thomson, before ProQuest acquired it in 2008.
You might assume the arrival of the Web and inexpensive search engines killed DIALOG. After all, why pay to search when you could simply type something into Google? But the service didn’t disappear. Today, it survives as Clarivate Dialog, offering specialized research databases, including technical, pharmaceutical, and patent information. Its pricing has changed, and the old connect-time meter is no longer the central feature of the business.

The service still exists because carefully indexed, authoritative information has value beyond what a general-purpose search engine can provide.

Pluses and Minuses​


Modern search engines are extraordinarily powerful. We can search billions of pages from a device in our pocket, frequently getting useful results in a fraction of a second. Most of us never think about the computational resources or communications infrastructure required.

DIALOG required more effort, but it offered something interesting in exchange: precise, repeatable searches built from explicit operations on sets of information. The searcher controlled the process instead of relying primarily on an opaque relevance-ranking algorithm. We think of search engines as an Internet invention, but many features we associate with modern information retrieval were developed decades before the Web.

DIALOG’s greatest achievement wasn’t putting a library on a computer. It was demonstrating that finding information could be an interactive process rather than a batch job. Still, imagine if while reading this, you looked at the clock and realized you’d just spent $8 of connect time. Oops.

Not all search engines are equal, of course. Before modern search engines, there were several things similar to DIALOG and, of course, bibliographies.
 
Back
Top