What is Robots.txt?
The robots exclusion protocol (REP), or robots.txt is a text file .Which is creat by the Webmasters to give order to robots (typically search engine robots) how to
crawl and index which pages on website.
Block all web crawlers from all content
User-agent: *
Disallow: /
Block a particular web crawler from a particular folder
User-agent: Googlebot
Disallow: /no-google/
Block a particular web crawler from a particular page of site
User-agent: Googlebot
Disallow: /no-google/blocked-page.html
Allow a particular web crawler to visit a particular page of a site
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html
User-agent: rogerbot
Allow: /no-bots/block-all-bots-except-rogerbot-page.html
Sitemap
User-agent: *
Disallow: Sitemap:
http://www.example.com/sitemap.xml










0 comments:
Post a Comment