How Search Engine Spiders Work

There are hundreds of search engines available today, but some are much more complex than others. This article will give you an idea of ​​how some of the most popular ones work.

 

Let's start with a small engine: InfoSeek. They point to only 200 words in your webpage, so it's important to make sure you have a meta tag on your site, and that the most important things are listed first. The information you enter in meta tags will be used to reflect your site description, and most meta tags may contain up to 200 characters of text. Meta tag keywords, however, can contain up to 1,000 characters.

 

These simple rules are important to remember for all search engines. Most importantly the information is accurate, it should be very close to the beginning of your meta tag or the beginning of the content of your site. Most search engines will not even affect your meta tags so it is important that you have the same body information that you have in your meta tags (although obviously you cannot simply list lists of keywords as this can be detrimental to your site content).

 

AltaVista search engine will send a Scooter, its spider, to explore your entire area. The scooter can take up to three months to spider and fully identify your location - a typical spider only takes 6-8 weeks. The scooter will usually enter somewhere between two and ten pages from your site each week. This means that the longer your site lasts, the better it will be shown for example of how search engines use Darwin's Theory in their thinking.

 

Joy used to be a powerful search center, but now it has been abandoned as an AOL provider and Netscape search provider, so it is less important than before. The algorithm he uses to determine the correlation of a keyword is very complex: it identifies your pages and tries to summarize them by selecting only the most important sentences. Expect your pages to be updated almost every two weeks. Keep in mind, though, that with meta tags it does not have a descriptive meaning when it comes to rank, or it will use your descriptive tags as long as the words match the content of your pages.

 

Let's move on to Lycos. Lycos fully integrates Open Directory Project (ODP) into their standard results pages, and uses search results from AllTheWeb. Lycos also uses clicks on its sister site HotBot. Lycos is one of the most difficult search engines, as their landing pages mean the same thing but refer to your site in a completely different way. As a general rule of thumb, your site will be hosted on Lycos in a timely manner as long as it is hosted on ODP and AllTheWeb.

 

Although WebCrawler is Excite, it still has a search engine and index. If you are likely to be listed on the WebCrawler list, you should try to stay on the list with it, as it is not a simple search engine to be listed. Its hit and miss rates combined with occasional targeting methods make the shipping process difficult, though not impossible.

 

The biggest player, of course, is Google, who use the page rank system as the centerpiece of their index. It used to be almost impossible to trick this page rank system to increase your ranking, but people soon realized that the more links they could generate on their site across the net, the better Google would be. Google is not considered to be using content sensitive standards. Sensitive content for content is used by Yahoo, Looksmart and ODP, however, and Google often looks at those sites when re-identifying its own database.

 

MSN is another important search engine. The holy trinity of search engines are currently Google, Yahoo !, and MSN. These three search engines come together to give you the most traffic you will find in search engines. MSN will usually be the first search engine to target your site and will likely rank more pages faster.

 

While no one can tell you when to search for any search engine, it is best to check back at least once a week. Whatever you do, however, you can re-post your site more often than two months or more - you may not get any reference if you do this.

Enjoyed this article? Stay informed by joining our newsletter!

Comments

You must be logged in to post a comment.

About Author