homepage Welcome to WebmasterWorld Guest from 54.161.192.130
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I've been looking at the WW robots.txt ...
...but why are these spiders disallowed?
elgumbo

10+ Year Member



 
Msg#: 255 posted 11:49 am on Jan 27, 2004 (gmt 0)

Hi

I can see in the WW robots.txt file that the following spiders are disallowed:

User-agent: scooter
User-agent: grub-client
User-agent: grub
User-agent: looksmart
User-agent: Copernic
User-agent: ia_archiver
User-agent: ia_archiver/1.6
User-agent: Alexibot

I can understand the reason for disallowing most of the other agents but not the above.

Can anyone clue me up on why WW don't allow these agents?

Cheers

 

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 255 posted 11:54 am on Jan 27, 2004 (gmt 0)

They're a waste of bandwidth?

TJ

elgumbo

10+ Year Member



 
Msg#: 255 posted 12:23 pm on Jan 27, 2004 (gmt 0)

I thought it might be that but wasn't sure if there was another reason?

If bandwidth is the only reason then I will allow them.
The site doesn't struggle from too many visitors at the moment - so I think I can live with giving the excess bandwidth to the spiders... for now.. ;)

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 255 posted 2:46 pm on Jan 28, 2004 (gmt 0)

> User-agent: scooter

Allowed AV in forever. They had 50k pages indexed and were spidering them 20-30 times a year. They sent us a total in 3 years of 350 visitors. They cost us a 1000 fold in bandwidth what they sent us.

>User-agent: grub-client
>User-agent: grub
>User-agent: looksmart

Homey-don't-play-dat. Worse numbers than AV.

> User-agent: Copernic

Why would we even consider it?

> User-agent: ia_archiver
> User-agent: ia_archiver/1.6
> User-agent: Alexibot

Security and liability risks.

Kirby

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 255 posted 5:49 am on Jan 31, 2004 (gmt 0)

> User-agent: ia_archiver
> User-agent: ia_archiver/1.6
> User-agent: Alexibot

Is it true that alexa ignores this?

sun818

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 255 posted 6:22 am on Jan 31, 2004 (gmt 0)

> Security and liability risks.

Can you explain? Are we talking about deleted posts that may be archived, etc?

sem4u

WebmasterWorld Senior Member sem4u us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 255 posted 1:32 pm on Feb 4, 2004 (gmt 0)

Are we talking about deleted posts that may be archived, etc?

I think so. Also, any threads that may need to be deleted at a later date.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved