| 12:30 pm on May 30, 2001 (gmt 0)|
Saw the same thing yesterday with their spider and was wondering what was happening myself.
Hit about 10 of my sites going after just the robots.txt file and the root html page. A few times it would request the robots file 10x in a row. I think the total amount of requests were in the 70-80 range per site. Just on those two files alone.
| 12:59 pm on May 30, 2001 (gmt 0)|
>Three excite spiders
I think that the 3 excite spiders are just trying to remember how to index. Given time, they may figure it out.
| 6:20 pm on May 30, 2001 (gmt 0)|
just trying to remember how to index
LOL... true. They are a bit out of practice, aren't they?
Nice to know I wasn't the only one who site was being thusly molested though... Would have seemed awfully sinister if they were leaving everyone else alone.
| 8:25 am on May 31, 2001 (gmt 0)|
I have this problem as well. This is an overview of the access attempts on robots.txt on May 30 ONLY:
.....43 (14 others under 10)
which adds up to 13424 requests on a single day, 50 times more than any other file on the site. This number has increased every day since May 22. I've tried to find an e-mail address at excite.com where I can notify them about the problem but have been unsuccessful. Help would be appreciated.
| 6:14 pm on May 31, 2001 (gmt 0)|
They eased off after their first assault on my site, so I let it slide... you could always try firing off a letter to a likely "generic" email address like "email@example.com" or something of the like...
| 6:53 pm on Jun 1, 2001 (gmt 0)|
"just trying to remember how to index"... rofl
yeah, been visited, but not hard. they were looking for the robots.txt but also grabbed the index pages
| 2:00 am on Jun 10, 2001 (gmt 0)|
Mivox, did you ever get in?
| 12:51 pm on Jun 10, 2001 (gmt 0)|
Add me to the list of questors, wanting to know if mivox got into excite.
Anxiously awaiting the results, just in case I see Architext knocking at my robots.txt...
| 7:54 am on Jun 11, 2001 (gmt 0)|
A couple months ago Excite started listing the same pages from our site that all the Inktomi sites are showing... for a while it was just our index page, and then (last week or so?) the exact same four out-of-date pages (all either 404 or redirects now) that were apparently dredged up out of Ink's dustbin files appeared in Excite...
Excite hasn't a) spidered my site itself, b) sent me any traffic, or c) shown any pages differing in any way from Ink results for our company name. I've pretty much written them off, but the sudden assault on my robots.txt definitely threw me off.
| 5:39 am on Jun 13, 2001 (gmt 0)|
Thousands of fast requests for non existant urls on my logs too.. Seem like the mindless spider confused my sites whit others, or the requests comes semi-random generated, or maybe this is a super-intelligent multi-level scan eh..
I remember this excite spider behavior happened also 2 years ago.