Welcome to WebmasterWorld Guest from 50.19.53.104

Forum Moderators: bakedjake

Message Too Old, No Replies

What is excite up to now?

slamming my robots.txt all weekend...

     

mivox

9:10 pm on May 29, 2001 (gmt 0)

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Three excite spiders hammered my robots.txt file all weekend...
Never requesting anything else:

jung.excite.com.......57 requests for robots.txt
daal.excite.com.......46 requests for robots.txt
tympani.excite.com....29 requests for robots.txt

Anyone else see anything like this? Any ideas what excite might stand to gain by requesting a single robots.txt file over 100 times in the course of a weekend?

msgraph

12:30 pm on May 30, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Saw the same thing yesterday with their spider and was wondering what was happening myself.

Hit about 10 of my sites going after just the robots.txt file and the root html page. A few times it would request the robots file 10x in a row. I think the total amount of requests were in the 70-80 range per site. Just on those two files alone.

Mike_Mackin

12:59 pm on May 30, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Three excite spiders

I think that the 3 excite spiders are just trying to remember how to index. Given time, they may figure it out.

mivox

6:20 pm on May 30, 2001 (gmt 0)

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member



just trying to remember how to index

LOL... true. They are a bit out of practice, aren't they?

Nice to know I wasn't the only one who site was being thusly molested though... Would have seemed awfully sinister if they were leaving everyone else alone.

erikt

8:25 am on May 31, 2001 (gmt 0)

10+ Year Member



I have this problem as well. This is an overview of the access attempts on robots.txt on May 30 ONLY:

...4221 marcuse.excite.com
...4191 pascal.excite.com
...4163 pierce.excite.com
....243 jung.excite.com
....227 daal.excite.com
....126 tympani.excite.com
.....88 rorty.excite.com
.....81 dosa.excite.com
.....41 triangle.excite.com
.....43 (14 others under 10)

which adds up to 13424 requests on a single day, 50 times more than any other file on the site. This number has increased every day since May 22. I've tried to find an e-mail address at excite.com where I can notify them about the problem but have been unsuccessful. Help would be appreciated.

mivox

6:14 pm on May 31, 2001 (gmt 0)

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member



They eased off after their first assault on my site, so I let it slide... you could always try firing off a letter to a likely "generic" email address like "support@excite.com" or something of the like...

dogboy

6:53 pm on Jun 1, 2001 (gmt 0)

10+ Year Member



"just trying to remember how to index"... rofl

yeah, been visited, but not hard. they were looking for the robots.txt but also grabbed the index pages

littleman

2:00 am on Jun 10, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Mivox, did you ever get in?

jeremy goodrich

12:51 pm on Jun 10, 2001 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Add me to the list of questors, wanting to know if mivox got into excite.

Anxiously awaiting the results, just in case I see Architext knocking at my robots.txt...

mivox

7:54 am on Jun 11, 2001 (gmt 0)

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member



A couple months ago Excite started listing the same pages from our site that all the Inktomi sites are showing... for a while it was just our index page, and then (last week or so?) the exact same four out-of-date pages (all either 404 or redirects now) that were apparently dredged up out of Ink's dustbin files appeared in Excite...

Excite hasn't a) spidered my site itself, b) sent me any traffic, or c) shown any pages differing in any way from Ink results for our company name. I've pretty much written them off, but the sudden assault on my robots.txt definitely threw me off.

gekoviola

5:39 am on Jun 13, 2001 (gmt 0)

10+ Year Member



Thousands of fast requests for non existant urls on my logs too.. Seem like the mindless spider confused my sites whit others, or the requests comes semi-random generated, or maybe this is a super-intelligent multi-level scan eh..
I remember this excite spider behavior happened also 2 years ago.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month