Robot file generator code, tool to generate robot file code, robots file generator code, contempo technologies services

This robots file code generator tool helps to develop robots.txt file to exclude the pages that you don’t want to index. This tool has fields to enter the robot to exclude and the directories to be excluded from indexing. On clicking create the robots.txt file will be created which you can use directly. Robots file

Entry	Meaning
User-agent: * Disallow:	The asterisk (*) in the User-agent field is shorthand for "all robots". Because nothing is disallowed, everything is allowed.
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /private/	In this example, all robots can visit every directory except the three mentioned.
User-agent: BadBot Disallow: /	In this case, the BadBot robot is not allowed to see anything. The slash is shorthand for "all directories" The User Agent can be any unique substring, and robots are not supposed to care about capitalization.
User-agent: BadBot Disallow: / User-agent: * Disallow: /private/	The blank line indicates a new "record" - a new user agent command. BadBot should uts go away. All other robots can see everything except the "private" folder.
User-agent: WeirdBot Disallow: /tmp/ Disallow: /private/ Disallow: /links/listing.html User-agent: * Disallow: /tmp/ Disallow: /private/	This keeps the WeirdBot from visiting the listing page in the links directory, the tmp directory and the private directory. All other robots can see everything except the tmp and private directories. If you think this is inefficient, you're right!

Robots File Generator Tool