Parts of a Search Engine

Search engines have three major elements:
First is the spider, also called the crawler or bot (for robot). The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis, such as every month or two, to look for changes.

Everything the spider finds goes into the second part of a search engine;
The index.
The index, sometimes called the catalog, is like a giant book containing a copy of every web page the spider finds. If a web page changes, then this book is updated with new information. Sometimes it can take a while for new pages or changes to be added to the index. Thus, a web page may have been "spidered" but not yet "indexed." Until it is indexed --actually added to the index-- it is not available to those searching within that engine.

Search engine software is the third part of a search engine. This is the computer program which sifts through the millions of pages recorded in the index, to find you a match to your search term, then presents the results in what it believes to be the most relevant order.

All search engines have the basic parts described above, but there are many differences in how these parts are combined to produce results. All the major search engines follow this theme to some degree, in the same way a cook may follow a chili recipe, but cooks like to add their own secret ingredients. Search engines like to add spice to their ranking methods. Each engine has it's own set of rules. Nobody does it exactly the same way, and this is why the same search on several search engines often produces different results.

The term "search engine" is often used generically but inappropriately to describe any site where you can perform a search. Search engines and directories are not the same. The differences lie in how the listings are compiled.

 

Search Engines Vs. Directories

Search engines, such as Google; http://www.google.com create their listings automatically. If you change your web pages, search engines eventually find these changes. Page titles, body copy, meta tags, descriptions and outside links to a web site, plus many other elements, all play a role in how and where the site is listed within the engine.
Directories

A directory such as Yahoo! depends on humans for entering its listings. You submit a short description of your web site to the directory and an editor reviews your submission before adding it to the index. A good site, with good content, may be reviewed more quickly than a poor site. In a directory you can "submit" your site for review, but there is no guarantee that it will be included.
Here are two well known directories.
Yahoo! - http://www.yahoo.com
DMOZ - http://www.dmoz.org

 

Hybrid Search Engines and Metacrawlers

Some search engines maintain an associated directory and also have a part of their listings supplied by a third party. Some engines are known as metacrawlers because they can "crawl" through several other search engines all at the same time. Metacrawlers can be very useful when you are searching because you can compare results from several engines at once.

My favorite Metacrawler these days is EntireWeb.
http://www.entireweb.com/

 

Using a Search Engine

Imagine walking up to someone and saying, 'cookies.' They're going to look at you with a blank face. By typing a request as vague as 'cookies,' into a search engine you are bound to get a vague set of results returned to you. Search engines don't have the ability to ask questions to help you focus your search. You have to help them. They need to quickly find a match to your request of 'cookies', so it makes sense they will first present you with the most generic list of subjects from within their index. Making your request much more specific such as 'chocolate chip cookie recipes' will bring you back a much more relevant list of results. The more specific your search term, the better your chances are of finding what you want.

Once you feel you are getting good at finding what you want, try using the "advanced" search techniques offered by the engine. The advanced search functions offer a complete array of 'power searching' tips and techniques. As an example, by using a plus sign (+) to combine your search phrases, you can really hone in on better results. Something like this: "DVD players +Toshiba" will get you right to where you want to be, rather than using the term DVD players by itself. This is only one rather simple example. Advanced searching is a whole new world. When you're ready to become a power searcher check it out.

Good luck and have fun searching.