Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo robots txt

nedd to disallow all urls with?

         

cheaperholidays

2:16 pm on Nov 14, 2006 (gmt 0)

10+ Year Member



Hello everyone

If we put this in our robots.txt would this block slurp from indexing all urls with a query string

User-Agent: Yahoo! Slurp
Disallow: /*?

THanks

Pete

ps i did accidently post this in the yahoo forum but got blocked

goodroi

4:54 pm on Nov 16, 2006 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



hey cheaperholidays,

glad you reposted here :)

yes your robots.txt line is correct and should block all? urls.

since yahoo has recently started handling wildcards in the robots.txt me and my paranoid brain would suggest that you monitor your log files to make sure your site is still indexed properly after you start using wildcards for slurp. honestly i would even suggest you monitor those log files whenever you touch your robots.txt (just my paranoid 2 cents rather be safe than sorry :))