Welcome to WebmasterWorld Guest from 54.205.126.164

Forum Moderators: goodroi

Message Too Old, No Replies

I've been looking at the WW robots.txt ...

...but why are these spiders disallowed?

     
11:49 am on Jan 27, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 30, 2002
posts:303
votes: 0


Hi

I can see in the WW robots.txt file that the following spiders are disallowed:

User-agent: scooter
User-agent: grub-client
User-agent: grub
User-agent: looksmart
User-agent: Copernic
User-agent: ia_archiver
User-agent: ia_archiver/1.6
User-agent: Alexibot

I can understand the reason for disallowing most of the other agents but not the above.

Can anyone clue me up on why WW don't allow these agents?

Cheers

11:54 am on Jan 27, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7249
votes: 0


They're a waste of bandwidth?

TJ

12:23 pm on Jan 27, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 30, 2002
posts:303
votes: 0


I thought it might be that but wasn't sure if there was another reason?

If bandwidth is the only reason then I will allow them.
The site doesn't struggle from too many visitors at the moment - so I think I can live with giving the excess bandwidth to the spiders... for now.. ;)

2:46 pm on Jan 28, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38070
votes: 16


> User-agent: scooter

Allowed AV in forever. They had 50k pages indexed and were spidering them 20-30 times a year. They sent us a total in 3 years of 350 visitors. They cost us a 1000 fold in bandwidth what they sent us.

>User-agent: grub-client
>User-agent: grub
>User-agent: looksmart

Homey-don't-play-dat. Worse numbers than AV.

> User-agent: Copernic

Why would we even consider it?

> User-agent: ia_archiver
> User-agent: ia_archiver/1.6
> User-agent: Alexibot

Security and liability risks.

5:49 am on Jan 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 1, 2003
posts:1203
votes: 0


> User-agent: ia_archiver
> User-agent: ia_archiver/1.6
> User-agent: Alexibot

Is it true that alexa ignores this?

6:22 am on Jan 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 16, 2001
posts:2007
votes: 0


> Security and liability risks.

Can you explain? Are we talking about deleted posts that may be archived, etc?

1:32 pm on Feb 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 18, 2002
posts:3069
votes: 1


Are we talking about deleted posts that may be archived, etc?

I think so. Also, any threads that may need to be deleted at a later date.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members