While the idea of the W3C was being formulated, different search engines began to spring up.
In 1993, Matthew Gray developed The World Wide Web Wanderer, also referred to as just the Wanderer. It was a Perl-based web crawler that was first deployed in June 1993 to measure the size of the World Wide Web. The crawler was used to generate an index called the Wandex later in 1993. Really, Wanderer was the first search engine crawler.
Also in 1993, six undergraduate students at Stanford University were looking for a way to use statistical analysis of word relationships to improve relevancy of searches on the Internet. The search engine Excite was born out of this research and launched in 1995. At its height, Excite was one of the most recognized brands online and by 2000 it was the fourth most visited website on the internet. At its most popular, Excite provided an assortment of content including news and weather, a metasearch engine, web-based email, instant messaging, stock quotes, and a customizable homepage for its users.
Not really a search engine initially, In 1994 Stanford University students Jerry Wang and David Filo created Yahoo! It was originally an Internet bookmark list and directory of interesting sites. At its peak, Yahoo! provided its users a web portal which included: a search engine, business directory, web based email, news, finance, groups, answers, the first major online advertising platform, mapping, video sharing, fantasy sports, and its own social media website. At one point Yahoo! was the most widely read news and media website – with over 7 billion views per month – ranking as the sixth-most-visited website globally in 2016.
In March 1994, the search engine called WebCrawler was born. It was the first search engine to provide full text indexing and search capabilities. By November 14, 1994, WebCrawler served its one millionth query. On June 1, 1995, a company called America Online (AOL) bought WebCrawler so that it too would have the capability to crawl and index the web.
Also in 1994 Lycos was formed. What was unique about Lycos was how quickly it could crawl and index the World Wide Web. Lycos had the largest index at the end of 1996 with 60 million documents.
In 1995 a search engine named Altavista entered the market and it was even faster than the rest. In August 1995, it conducted its first full-scale crawl of the web bringing back about 10 million pages. Over 300,000 visitors used Altavista on its first day. They had 19 million hits by the end of 1996, and 80 million per day at the end of 1997. By 2000 the value of Altavista had reached $2.3 billion. Altavista is also important because it became the first Internet search engine to offer image, audio, and video search in 2002.
In 1996, two Stanford University students test Backrub, a new search engine which ranks sites based on inbound link relevancy and popularity. Backrub would ultimately become Google in 1998. The students names were Sergey Brin and Larry Page.
In 1997 a search engine called “Ask Jeeves” came into existence. It was unique because it allowed searchers to use questions to find websites and it was very good at doing this.
Realizing that search wasn’t going away, Microsoft entered the fray in 1998 with its first attempt at a search engine called MSN search. MSN Search was renamed Windows Live in 2006, then renamed again to Bing in 2009
AllTheWeb was launched in 1999 and it had even more features and speed than its predecessors. It was later bought by Overture which was then purchased by Yahoo in 2003.