Robots.txt

From Omnia

Revision as of 15:00, 18 April 2016 by Kenneth (talk | contribs) (Created page with "== robots.txt == [http://www.google.com/webmasters/start See what pages aren't crawled by Google. Try Google Webmaster Tools.] [http://www.robotstxt.org/wc/norobots.html The...")

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to navigation Jump to search

robots.txt

See what pages aren't crawled by Google. Try Google Webmaster Tools.

The Web Robots Pages

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space

# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:

# go away
User-agent: *
Disallow: /

# allow all
User-agent: *
Disallow:

keywords

Retrieved from "https://aznot.com/index.php?title=Robots.txt&oldid=3257"