What are Search Engines & Directories?

What is a Search Engine?

In simple terms, a Search Engine may be described as an automated computer program that enables one to search information, documents, or databases available on the internet. Search Engines provide a smart and swift way of finding information on the internet. When a search term or a keyword is entered into a search engine, it returns the most relevant and matching WebPages that contain similar search terms and keywords. Google, MSN Search, Yahoo, AOL Search, AlltheWeb.com, Ask Jeeves, Excite, Lycos, Infoseek, Overture and Alta Vista are some of the famous search engines. Search Engines are probably the only forum on earth that provides information on almost everything and anything and that too in a couple of seconds on your desktop.

From the SEO perspective, search engines can be considered as huge databases containing website information which offer visitors the facility to query the database based upon the visitors search term. Search engines gather data about websites with the help of specially design software called the search engine spiders or search engine robots or crawlers or bots which scan the world wide web looking for new websites to add to the existing database of sites, as well as tracking changes to websites that the search engine have already indexed.

From the SEO perspective, we can look at search engines as intelligent software that have been programmed to report website information in a meaningful manner based upon what is being searched. Since the search engine is actually software, the search engine optimization professional can learn how it has been programmed to think, and accordingly take action on websites so that they are viewed favorably by the search engine while displaying the search results.

Various components of the SERPs (Search engine results pages) with special focus on Google.

As a search engine optimization professional, you must start looking at the search engine listings with an eye for detail. Since what is displayed in the SERPs is the moment of truth for the SEO professional who has been optimizing a website or webpage, it is all the more necessary to understand various components:

Below is a screen shot of the Google results page when the term “iq test” is searched for:




Notice the marked parts of the screen shot of the Google SERP (search engine results page) for the term "iq test". Google is arguably the best search engine with incisive algorithms that enable it to generate precise results that match-up with the search keywords.

Notice the following important aspects in the above Google screenshot:

1.The Title of the listing in the SERPs has been marked by A. The contents of the title come from the title of the webpage.
2.The Description of the listing in the SERPs has been marked by B. The contents of the description come from the Description meta tag of the webpage.
3.The searched term or parts of the searched term are in bold. Each SERP contains ten listings on each page.
4.Google displays up to 66 characters of a Title Tag of the webpage’s in the Title of the listing in the SERP.

Google ensures two things:

1.That there should be no more than 66 characters in the title line.
2.That the last word of the title line is a complete one, even if it means reducing the number of characters to less than 66.

5.‘Cached’ has been marked as "C" and refers to a snapshot of the complete webpage’s (matching the search term) last crawled by Google. The main feature of this page is that it highlights all or a part of the search term.

You can use cache to quickly see how many times the searched keyword is present within a webpage.

6.‘Similar Pages’ has been marked as "D" and provides a list of different pages indexed by Google that contains information relating to the searched keywords.

7.Google displays ‘Results 1-10 of about 30, 000, 000,’ which has been marked by "E". When you search for the term "iq test", it means that the results displayed (for the search term "iq test") on the current Google page are the first ten of a total of 8,780,000 incidences of the presence of "iq test" found by Google.

8.Sponsored Links have been marked by "F". Sponsored Links are the links that you find on the right-hand side column of each Google SERP. These are paid text advertisements for which Google charges a fee for every click.

Below is a screen shot of the Yahoo results page when the term “iq test” is searched for?



Notice that the elements of the results page are almost the same as Google. Yahoo also displays the category to which the website listing belongs within the paid Yahoo Directory - this has been marked as "C".

At the end of the day, all the SEO efforts will boil down to what people see in the search engine results pages. It is therefore essential that you know from where the search engine picks up various parts of their listings because this is the only part of your website that people will get a glimpse of while on the search engine.

The following table shows where various parts of the SERP listings come from for various search engines:

Search Engine Title in the SERPsDescription n the SERPs
GoogleFrom the web pages title for search engine optimizationFrom the description meta tags of the web pages for search engine optimization
YahooFrom the web pages title for search engine optimizationFrom the description meta tags of the web pages and from the important content in the body of the webpage for search engine optimization
MSNFrom the web pages title for  search engine optimizationFrom the description meta tags of the web pages and from the important content in the body of the webpage for search engine optimization


From where are the main search results, pay per click search results and Directory results drawn?

Search engines make use of information from several sources apart from their own database of information. As search engine optimization professionals you should know from where various search engines are picking up data that make up various parts of their search results. The following terms should be clear

The main search results: refer to the results that we see in the SERPs and constitute the bulk of the results for any search engine.

Pay per click search results: Search results that appear due to advertisers paying the search engine to get a listing in the results, or having a textual advertisement displayed next to the search results.

Directory results: Many search engines have a directory of websites apart from a regular search engine feature. The listings within the directory are referred to as directory results.

The following table depicts the relationship between the various search engines and directories:

Search EngineMain Search ResultsDirectory ResultsPay Per Click Results
1GoogleGoogleDMOZGoogle
2YahooYahooYahoo DirectoryYahoo Search Marketing
3MSNYahooDMOZYahoo Search Marketing
4AOLGoogleDMOZGoogle


Do Search Engines search the whole web while performing a search operation?

No, as discussed above. Contrary to the misconception, Search engines do not search the internet; instead they search the documents that they have indexed and which lie in their databases. Search engines generate information about only those titles, phrases, keywords, subjects, or topics that are stored in their respective databases. As per the February 28, 2004 estimates, the Web has around 15 - 20 billion pages. The biggest search engine, Google, boasts of the largest Search Engine Index. Google’s Index contains around 6 billion pages in its Index.

Types of Search Engines:

Crawler-based Search Engines: As the name suggests, the Crawler-based search engines use a Spider or Crawler to search for new WebPages (or changes in existing pages). Google and Yahoo are the most powerful Crawler-based Search Engines.

Human-Powered Directories: Human-Powered Directories rely on humans for reviewing and selecting the WebPages. Open Directory Project, Yahoo Directory, Google Directory, LookSmart, and Euro Seek are some of the most popular Human-Powered Directories.

Hybrid Search Engines: Hybrid search engines use both crawlers as well as a directory for generating relevant results. Google, ExactSeek, Lycos and AltaVista are hybrid search engines which use a Directory to supplement their own search engines.

Meta Search Engines - Meta Search Engines are those search engines that search other search engines and directories. These search engine extract the best of the searches from various popular search engines and directories and include the information in their own search results. Dog pile, WebCrawler, Excite, MetaCrawler, and Ixquick are some examples of meta search engines.

How Search Engines work and how they rank websites based upon a search term?

As mentioned above Search engines consist of huge databases of websites that are queried based upon the search term entered by a user in the search engine search box. The job of the search engine optimizer (or SEO professional) lies in understanding how the search engine ranks websites in the results pages, and then taking necessary action to improve the websites current rankings. Every search engine makes use of a search algorithms (or formula) to query its database of websites. This algorithm defines the factors which determine the relevancy of a website while a search is being conducted for a search term. The search engine algorithm differs from search engine to search engine and is a carefully developed formula that allows the search engine to display websites that are most relevant to a search term in order of relevance. Most companies including Google keep the search algorithm a closely guarded secret.

Search engines are programmed to seek out evidence that point to a connection (or relation) between the searched term and the webpage’s existing in their databases. Some of the important factors that are included within the algorithm of most major search engines (including Google), and make up part of the evidence that points to relevancy between the search term and a webpage are.

•Whether the Page title of the webpage is includes the searched term
•Whether the header of the webpage is includes in the searched term
•Whether the body of the webpage includes the searched term and how many times it appears within the webpage
•The presence of the search term within pages of other sites that link to the webpage.

The above list is in no way exhaustive. Search engines like Google have several dozen parameters upon which their search algorithm is based. The role of the Search engine optimization professional is to understand the well known factors that make a considerable difference to the search results of the webpage and then to use optimization techniques on the webpage so that it is viewed favorably by the search engine algorithm and as a result the webpage’s gets a better ranking in the SERPs (Search Engine Result Pages) for a keyword/search term entered in the search engine search box. Always keep in mind that it will never be possible to know the exact logic behind any major search engine since it is never made public, and secondly because it is always evolving.

What are Directories and how do they differ from Search Engines?

Directories are sorted listings of websites that are maintained and approved for listing by human editors. Directory databases comprise of websites that have been manually submitted by the website owners or website editors, unlike search engines which make use of automation through spiders and bots for searching new websites, directories make use of human intelligence to ensure that each new site submitted to the directory meets certain standards and is placed in the appropriate category within the directory. Some of the important directories from the point of view of the search engine optimization are
•DMOZ ("http://www.dmoz.org/")
•Yahoo directory ("dir.yahoo.com")
•Looksmart directory ("www.looksmart.com")

A good site for locating directories is "www.allwebdirectories.com" which lists several directories by topic. Since most directories offer free listings, it is always a good idea to get listed in as many as possible since it is viewed favorably my most major search engines while ranking the site in the search results.

Difference between Search Engines and Directories

Search Engine Directory
Search Engines uses programs called spiders and crawlers which crawl the links and collect the data about the webpage’sDatabase of webpage’s is nothing but Directory. Webpage’s are classified by humans after examined thoroughly.
Search Engines list the results according to keyword relevanceDisplay of directory listings will be in alphabetical order.
In a Search Engine, Spiders or Crawlers locate the webpage’sWebpage’s are submitted to directories by humans.
After entering search terms in search engines searcher get the required information.Directories contains in folders should be browse through folders to obtain directory but no by keywords.
Due to broader search offered by search engines the pages found in search results are large.Due to limited search offered by directories as topic specific, website specific results obtained are less.
Websites are not categorized by search enginesIn a directory, a website is submitted under a particular category.


Searches can be conducted by using two different modes – search engines and directories. However, both are entirely different.

What is the Open Directory Project and how can a listing in DMOZ boost my SEO efforts?

The open directory project is the world’s largest volunteer based directory of websites organized, maintained and reviewed by chosen directory editors who work as volunteers. The Open Directory Project is available at “ http://www.dmoz.org/” and is one of the first places you should submit your website. Sites are submitted to DMOZ free of cost at “http://www.dmoz.org/add.html”, and are placed in an appropriate category after review by an editor who is a chosen specialist in a category within DMOZ. The importance of DMOZ can be judged from the fact that it supplies its directory listings to leading search engine such as Google, Altavista, AOL, Go.com, iwon, MSN, Netscape.

The DMOZ directory listing is organized alphabetically, so a website named eqtest.com will figure above a website called iqtest.com. Since DMOZ is such an important place to get listed, many website owners actually choose domain names beginning with numbers such as 1eqtest.com so that they list high within the directory. This trick may not always work since each site is reviewed by a real human being who may be able to see through the technique.

What is the importance of a Yahoo Directory listing?

Yahoo has its own paid directory listing at “http://www.dir.yahoo.com/” which allows websites to get listed for one year at $299.99. Yahoo editors review all submissions within one week. The position of the site within the directory changes according to the number of people clicking on it and it slowly move up the ranks as more and more people click on the listing . The Yahoo Directory listings are used within the yahoo search results, but spending the $299.99 has some other benefits that are usually over looked:

Getting a Yahoo Directory listing ensures that the website will quickly become available in the Google search results (within days) due to the agreement between the two Mega search engines that allows Google to use the Yahoo Directory listings information. This technique can save new sites months of waiting to see their site in the Google SERPs.

The Yahoo Directory passes PR (Google page rank) to the listed sites. Most Yahoo directory pages have a PR of 6 or 5. This is an excellent way of buying PR for your site from the world’s number one visited site. Due to this reason Google usually gives a PR boost to websites listed in the Yahoo Directory. You will read about the concept of Google PR later on.

What happens if Yahoo places you in the wrong category of its directory or on a Non-PR page?

If you had planned to get a Yahoo Directory listing for generating more traffic from Yahoo as well as the ranking benefits from Google, you may come up with a situation where your site is listed in a category which does not match your sites objectives or you may encounter a situation where you expected to get a listing on a PR6 page of the directory, but actually get one on a PR2 page. In such an eventuality, you can appeal to Yahoo within 30 days of getting listed and stand a good chance of being heard.

Do placements in (PPC) Pay per click search engines help to increase the website links popularity?

No, PPC search engines such as Overtune & Google Adwords do not offer direct links to websites from their search engines paid listings or on websites that offer advertising space to PPC search engine advertising partners. When people click on a PPC search engine listing, it first goes to the search engine for tracking the click through popularity, and then is re-directed to the advertisers website. The same is the case with affiliate networks such as Commision Junction.

How do you submit your site to search engines/directories?

You can either manually submit your site to search engines such as Google or Yahoo, or let the search engine spiders automatically find your website. If your website already has a link back from an existing spidered webpage from another website, it will most likely, get indexed when the webpage linking to you is re-visited by the spider. Below are some of the places where you can submit your site to various search engines yourself.

Google – “http://www.google.com/addurl”
Yahoo – “http://docs.yahoo.com/info/suggest/”
MSN – “http://beta.search.msn.com/docs/submit.aspx”
Altavista – “http://www.altavista.com/addurl/default”
DMOZ – “http://dmoz.org/add.html”

While submitting pages to search engines keep in mind that most engines have a maximum limit to the number of webpages from any website. It is a good idea to break up the page submission over a few days. If you feel the webpage has not been indexed even after 6-7 weeks, you can re-submit the URL. No search engines penalize websites for re-submission of webpages unless it is done excessively. Keep in mind that it could take search engines like Google or Yahoo upto 6-7 weeks to index a website that has been submitted for inclusion. You can speed up the initial indexing of the site by one or more of the following techniques:-

Get a directory listing in DMOZ (which itself could take weeks).
Get a link back from a quality site that has been indexed. Websites that have news or Blogs on them are visited often by spiders. Try to get links from such sites to speed up the indexing process.
Get a paid directory listing on Yahoo.
Submit your site to several free online directories
Always remember to be patient while the search engines index your site. Don’t resort to any desperate measures such as re-submitting the site over and over again. The search engines will always take their time.

Factors to keep in mind while submitting a site to the search engines

Ensure that the keywords and description metatags have been made carefully. You will learn about metatags later.
Ensure that the pages are optimized, and the keywords are placed throughout.
Add the site to the most relevant category. You can conduct a search of your keyword or check where your competitipon is listed in the directory.
Ensure that your website is working properly. Some directory editors will check to see if the site is real.
Ensure that there are no broken links.

How do you monitor the performance of your website in the search engines?

To check the ranking of your site in the SERPs, simply search for the keyword or keyword phrase in the search box of the search engine and check the rank of your optimized webpage. Keep working on your webpage till it reaches the first 10 positions, as most people don’t go beyond the first page in the search results. If you have a large number of keywords, you can consider using a software which will give you reports on the ranking of your keywords. Web Position (“http://www.web-positiongold.com/”) is popular software that analyses the ranking of pages in the search engines. Google does not approve of such software, but is not taking any active steps against people who use them or companies who make them.