# Updated: 04-05-05 # http://www.robotstxt.org/wc/norobots.html # SAMPLES --------------------------------- # User-agent: Googlebot # Disallow: # Sample Says nothing is disallowed # User-agent: lycra # Disallow: / # Sample Says EVERYTHING is disallowed # The first two lines, starting with '#', specify a comment # The first paragraph specifies that the robot called 'webcrawler' has nothing disallowed: it # may go anywhere. # SAMPLES --------------------------------- #--------------------------------------------------------------------------- User-agent: * Disallow: /tmp Disallow: /logs