https://aznot.com/index.php?action=history&feed=atom&title=Robots.txt 2026-05-09T07:02:12Z Revision history for this page on the wiki MediaWiki 1.41.0 https://aznot.com/index.php?title=Robots.txt&diff=3257&oldid=prev 2016-04-18T15:00:10Z

<p>Created page with &quot;== robots.txt == [http://www.google.com/webmasters/start See what pages aren&#039;t crawled by Google. Try Google Webmaster Tools.] [http://www.robotstxt.org/wc/norobots.html The...&quot;</p> <p><b>New page</b></p><div>== robots.txt ==<br /> [http://www.google.com/webmasters/start See what pages aren&#039;t crawled by Google. Try Google Webmaster Tools.]<br /> <br /> [http://www.robotstxt.org/wc/norobots.html The Web Robots Pages]<br /> <br /> # robots.txt for http://www.example.com/<br /> <br /> User-agent: *<br /> Disallow: /cyberworld/map/ # This is an infinite virtual URL space<br /> Disallow: /tmp/ # these will soon disappear<br /> Disallow: /foo.html<br /> <br /> # robots.txt for http://www.example.com/<br /> <br /> User-agent: *<br /> Disallow: /cyberworld/map/ # This is an infinite virtual URL space<br /> <br /> # Cybermapper knows where to go.<br /> User-agent: cybermapper<br /> Disallow:<br /> <br /> # go away<br /> User-agent: *<br /> Disallow: /<br /> <br /> # allow all<br /> User-agent: *<br /> Disallow:<br /> <br /> == keywords ==<br /> <br /> [[Category:Web Development]]<br /> [[Category:Programming]]</div>

Kenneth