The Search for the Fastest Engine

Like most of us, John M. Lervik has a favorite number -- but his isn't anything as simple as 3, 7, or 9. From his harborside office in Oslo, Norway, Lervik admits a fondness for the figure 2,112,188,990.

It's not a random number: It's the colossal number of Web pages currently indexed by his search-engine company, FAST (Fast Search & Transfer). It's big enough to nudge FAST ahead of Google's 2,073,418,204, earning FAST the title of the Web's most comprehensive search service. And, if Lervik's devotion to numbers continues, he hopes to use a formula of bigger, fresher, and closer to put FAST ahead of its better-known search-engine rivals.

In the era of the post - dotcom meltdown, it's fashionable to downplay the Web's role in the economy. But with little fanfare, search engines continue to flourish as an online hot spot for several key reasons. First, information proliferation: More data has been produced in the past two years than in the previous 2,000 -- and the race to catalog it poses an extraordinarily complex, significant, and ultimately rewarding mathematical and linguistic challenge. Second, Web habit: For most daily users of the Web, relying on a search engine as a first step in navigation has become a matter of routine. Third, advertising: Advertisers have clicked to the idea of flashing their brands in ads that accompany relevant search results.

The upshot is a fast-paced race to the future among the best-known search engines. If counting eyeballs is the metric that matters, Google holds the pole position. Its successful combination of powerful technology and common-sense usage pulls in 58 million users a month. Nearly half of all searches globally go through Google, which, according to some sources, has the strongest brand on the Web. And Google enjoys almost fanatical support from its users -- all of which should make its lead unassailable. But coming up on the blind side is Lervik's low-profile, aerodynamic Norwegian machine, with a supercharged search engine and a well-tuned business model that may just leave Google in the dust.

"While others have been building pretty cars with shiny wheels and nice paintwork, we've been building the strongest engine," says Lervik, CEO of the $36 million firm he cofounded in 1997 with fellow research students at the Norwegian University of Science and Technology -- Norway's version of MIT. It bothers Lervik little that most users have never heard of Alltheweb.com, the not-so-public face of FAST. In fact, letting others put their stickers over his brand is part of Lervik's distinctive strategy: By forging behind-the-scenes partnerships with some of the biggest online players, such as Dell, IBM, and Lycos, Lervik hopes to win more users than Google, without having them know that they're a FAST user.

The FAST Formula Lervik's formula for overtaking Google is a simple three-part equation: bigger plus fresher plus closer. First, bigger. Simply put, when it comes to search engines, size matters. Lervik is determined to make FAST the Web's biggest engine -- which is why FAST searches pages in 49 languages and 225 different file formats, including more than 180 million multimedia files, 132 million FTP files, and 2 million MP3s.

Second, fresher. One of the biggest frustrations of Web searchers is stale or dead-end results. Lervik's team completely refreshes FAST's index every 7 to 11 days -- which makes its results much fresher than Google searches, which operate on a 28-to-30-day cycle.

Third, closer. The real test of a search engine: How closely can it deduce the real intent of a user's query? When it comes to relevance, Lervik acknowledges, FAST trails Google -- and that is one area where he's been pouring in resources. For the past 18 months, FAST researchers have been working on what Lervik calls "the third generation of search."

Here's what's at stake: Take the query "Where is Saturn?" A first-generation search engine looks only for pages that contain the word "Saturn." A second-generation search engine culls documents linked from pages about Saturn.

To perform a third-generation search, says Lervik, means getting closer to the user's real intention -- applying rules of grammar, syntax, and semantics to computer linguistics. "It's a morphological challenge," Lervik says, "to understand that words can be written in many different forms." About one in 10 queries, for example, is misspelled, so FAST's latest algorithms consider whether the spelling of "Saturn" is correct. Query analysis recognizes "Where is" as a request for a location, while dynamic clustering groups results together based on whether the user wants information on Saturn the planet or Saturn the car.