Welcome to WebmasterWorld Guest from 54.161.53.213

Forum Moderators: bakedjake

Message Too Old, No Replies

What is excite up to now?

slamming my robots.txt all weekend...

     
9:10 pm on May 29, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 6, 2000
posts:3928
votes: 0


Three excite spiders hammered my robots.txt file all weekend...
Never requesting anything else:

jung.excite.com.......57 requests for robots.txt
daal.excite.com.......46 requests for robots.txt
tympani.excite.com....29 requests for robots.txt

Anyone else see anything like this? Any ideas what excite might stand to gain by requesting a single robots.txt file over 100 times in the course of a weekend?

12:30 pm on May 30, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 29, 2000
posts:1425
votes: 0


Saw the same thing yesterday with their spider and was wondering what was happening myself.

Hit about 10 of my sites going after just the robots.txt file and the root html page. A few times it would request the robots file 10x in a row. I think the total amount of requests were in the 70-80 range per site. Just on those two files alone.

12:59 pm on May 30, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 7, 2000
posts:1720
votes: 0


>Three excite spiders

I think that the 3 excite spiders are just trying to remember how to index. Given time, they may figure it out.

6:20 pm on May 30, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 6, 2000
posts:3928
votes: 0


just trying to remember how to index

LOL... true. They are a bit out of practice, aren't they?

Nice to know I wasn't the only one who site was being thusly molested though... Would have seemed awfully sinister if they were leaving everyone else alone.

erikt

8:25 am on May 31, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


I have this problem as well. This is an overview of the access attempts on robots.txt on May 30 ONLY:

...4221 marcuse.excite.com
...4191 pascal.excite.com
...4163 pierce.excite.com
....243 jung.excite.com
....227 daal.excite.com
....126 tympani.excite.com
.....88 rorty.excite.com
.....81 dosa.excite.com
.....41 triangle.excite.com
.....43 (14 others under 10)

which adds up to 13424 requests on a single day, 50 times more than any other file on the site. This number has increased every day since May 22. I've tried to find an e-mail address at excite.com where I can notify them about the problem but have been unsuccessful. Help would be appreciated.

6:14 pm on May 31, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 6, 2000
posts:3928
votes: 0


They eased off after their first assault on my site, so I let it slide... you could always try firing off a letter to a likely "generic" email address like "support@excite.com" or something of the like...
6:53 pm on June 1, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 23, 2000
posts:494
votes: 0


"just trying to remember how to index"... rofl

yeah, been visited, but not hard. they were looking for the robots.txt but also grabbed the index pages

2:00 am on June 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


Mivox, did you ever get in?
12:51 pm on June 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


Add me to the list of questors, wanting to know if mivox got into excite.

Anxiously awaiting the results, just in case I see Architext knocking at my robots.txt...

7:54 am on June 11, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 6, 2000
posts:3928
votes: 0


A couple months ago Excite started listing the same pages from our site that all the Inktomi sites are showing... for a while it was just our index page, and then (last week or so?) the exact same four out-of-date pages (all either 404 or redirects now) that were apparently dredged up out of Ink's dustbin files appeared in Excite...

Excite hasn't a) spidered my site itself, b) sent me any traffic, or c) shown any pages differing in any way from Ink results for our company name. I've pretty much written them off, but the sudden assault on my robots.txt definitely threw me off.

gekoviola

5:39 am on June 13, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Thousands of fast requests for non existant urls on my logs too.. Seem like the mindless spider confused my sites whit others, or the requests comes semi-random generated, or maybe this is a super-intelligent multi-level scan eh..
I remember this excite spider behavior happened also 2 years ago.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members