Forum Moderators: goodroi
the above is how we have our robots.txt set up to block yahoo from spidering, however yahoo is still spidering and placing pages in their index. We have another site with pages that could be seen as duplicates and want to avoid problems. What can we do differently to stop yahoo?
User-agent: Slurp
Disallow: /
See Also: Yahoo! Help / Search Help / Yahoo! Slurp - Yahoo!'s Web Crawler [help.yahoo.com]
I have specifically restricted ALL robots from visiting certain pages on my site like:
User-agent: *
Disallow: /rtv/ktv-l.htm
but Yahoo Slurp still comes and requests those pages. I think Yahoo does not obey the robot rules.
I've never seen Yahoo dissobey robots.txt, so either there is a problem with your robots.txt file, or what you are seeing is a bot that is spoofing the Yahoo User Agent.
You can check this by
1) seeing if Yahoo is actually listing those pages
2) checking to see what IP address these request are coming from and doing and doing a whois lookup to find out who owns that ip. e.g. www.dnsstuff.com
What is the code to get them in certain pages?
I am sorry I know this has been asked before and I did look for a search on this site to try to find this info.
"How do I prevent my site from being crawled or prevent certain subdirectories from being crawled?"
...then try including the Slurp-specific instructions in your robots.txt.
You can also search Google using your search term(s) plus the following:
site:webmasterworld.com
You'll find robots.txt info galore here on WW in all of this forum's posts, and the official basics here [robotstxt.org], a.k.a. robots.txt.org.
