A spider is a program used by a search engine to index the
world wide web content. Spiders captures specific information
about a page some capture the title and the first 1,000 characters
of content, some captures the title and "description" meta
tag, some look for only the keywords and some use the combination
of all of these methods.
|