To avoid undesirable content from the look for indexes, webmasters can instruct spiders not to crawl certain files or directories through the normal robots.txt file in the basis directory of the area. Also, a site might be explicitly excluded from a internet search engine's databases by making use of a meta tag precise to robots (commonly ). When a