Forum Moderators: goodroi
I've recently include this line in my robotx.txt:
Disallow: /example
Because I need to exclude some files like example.htm, example.php ...
Unfortunatly these file are store in the directory /example_cars/
Do you think , the bot will not spider the entire directory or only the files named example.php, example.htm...