Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Robots.txt question excluding every folder expect one JVB_Mktg msg:1529021 7:22 pm on Mar 2, 2003 (gmt 0) How can I have a bot ignore every directory in my site except one?
Do I have to disalow all of them 1 by 1 ie.
Disallow: /folder4/ etc.. or is there an easier way.
I only need it to check www.mydomain.com/midi and exclude it from every other place.
wilderness msg:1529022 7:33 pm on Mar 2, 2003 (gmt 0)
Javi, Welcome to Webmaster World.
It is imperative that you keep in mind that robots.txt is a SUGGESTION to bots of compliance.
Many bad bots don't even read robots.txt.
With the above in mind. . .
denies all bots.
I seem to recall that Allow is not valid protocol in robots'txt.
So your alyernatives are either to deny all or list each "1 by 1"
JVB_Mktg msg:1529023 7:46 pm on Mar 2, 2003 (gmt 0)
Thanks wilderness :) weesnich msg:1529024 7:47 pm on Mar 2, 2003 (gmt 0)
User-agent: * Disallow: /folder1 Disallow: /folder2 Disallow: /folder3 Disallow: /folder4 [empty line here]
It is better to disallow "/folder", as some SE assume that they are allowed to try "/folder" if you only disallow "/folder/", especially if you have some internal or incoming links pointing to "/folder".
JVB_Mktg msg:1529025 7:54 pm on Mar 2, 2003 (gmt 0)
Thanks weesnich, it looks a lot better now :)