Page is a not externally linkable
webhound - 9:05 pm on Sep 21, 2005 (gmt 0)
What I am wondering is, is it normal for slurp to keep crawling the site aggressively even though they have a manual ban on the site? And why am I seeing these other yahoo spiders never seen them in logs before.
Seriously though I can relate to this topic as I have been trying to get the company's site that I've recently started with "unbanned". Haven't had any success thus far though. (the ban happened before i started so it's sooo not anything i've done.)
YahooSeeker/1.2
YahooSeeker/CafeKelsa
YahooSeeker/CafeKelsa-dev
YahooSeeker/1.0