Forum Moderators: open
I decided to give them some very, very spammy looking pages, just in case they are up to what i would be doing, were the situation reversed...
Seriously, though, they are the biggest no show...why spider and say you're putting together an engine, if you aren't actually going to do it?
Cheers,
Han solo
below is the referral line from the log file for petersnews:
news12.petersnews.com - - [04/Jan/2001:15:22:43 -0500] "GET /robots.txt HTTP/1.1" 404 216 "-" "user-1.petersnews.com"
All the other referral lines from petersnews are similar.
They followed the links through the whole site....including trying to check some password protected folders. Rather interesting for a search engine that isn't.