What is Robots.txt?

The robots exclusion protocol (REP), or robots.txt is a text file .Which is creat by the Webmasters to give order  to robots (typically search engine robots) how to crawl and index which  pages on  website.


Block all web crawlers from all content
User-agent: * 
Disallow: /

Block a particular web crawler from a particular folder
User-agent: Googlebot 
                                       Disallow: /no-google/

Block a particular web crawler from a particular page of site
User-agent: Googlebot 
Disallow: /no-google/blocked-page.html

Allow a particular web crawler to visit a particular page of a site
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html 
User-agent: rogerbot 
Allow: /no-bots/block-all-bots-except-rogerbot-page.html

Sitemap 


User-agent: * 
Disallow: Sitemap: http://www.example.com/sitemap.xml

0 comments:

Post a Comment