Robots.txt: Difference between revisions
Jump to navigation
Jump to search
(Created page with "== robots.txt == [http://www.google.com/webmasters/start See what pages aren't crawled by Google. Try Google Webmaster Tools.] [http://www.robotstxt.org/wc/norobots.html The...") |
(No difference)
|
Latest revision as of 15:00, 18 April 2016
robots.txt
See what pages aren't crawled by Google. Try Google Webmaster Tools.
# robots.txt for http://www.example.com/ User-agent: * Disallow: /cyberworld/map/ # This is an infinite virtual URL space Disallow: /tmp/ # these will soon disappear Disallow: /foo.html
# robots.txt for http://www.example.com/ User-agent: * Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Cybermapper knows where to go. User-agent: cybermapper Disallow:
# go away User-agent: * Disallow: /
# allow all User-agent: * Disallow: