physics - 10:06 pm on Oct 7, 2003 (gmt 0)
Had to ban YahooSeeker in .htaccess
I didn't have rules set up in robots.txt but this thing was *hammering* my site. Some dynamic pages on my site take a while to load so instead of waiting/giving up this thing just keeps going for it! Needless to say my little site couldn't handle it.
Checked yahoo's product search and couldn't see myself in there so why suffer the hammering if I'm getting nothing out of it? If anyone starts to see profits from YahooSeeker spidering effects please let me know though.