To create a Robot Text file that will allow only the good* spiders access to your site simply leave the box below empty and hit enter. If you wish to restrict access to certain areas of your site then simply add the name of the folders(or files) you want to stop robots spidering to the box below (seperate each by a comma eg - /folder/, file.html, file_two.html ). More information regarding robots.txt files can be found here.
Folders should be entered inside a set of backslashes (eg /folder/).
Files should be entered with the complete file name (eg file.html).
Please ensure that you have set the correct paths to files/folders you wish to protect.
To complete your robots.txt file simply cut and paste the information generated on the next screen into a text file and upload that onto your server (in the root). It is advisable to validate your robots.txt file and this can be done here http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
Any comments or problems please email to info at robottext.co.uk
* - the definition of what constitutes a good spider is a matter of debate, but i have based this list of "good spiders" on information taken from http://www.searchengineworld.com/robots/robots4.txt
Users use any information generated from this site at thier own risk and whilst every effort has been been made to ensure that all information is up to date no gaurantee is given and the site owners cannot be held responsible for any errors that may arise from its use.