Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: goodroi
I can see in the WW robots.txt file that the following spiders are disallowed:
I can understand the reason for disallowing most of the other agents but not the above.
Can anyone clue me up on why WW don't allow these agents?
If bandwidth is the only reason then I will allow them.
The site doesn't struggle from too many visitors at the moment - so I think I can live with giving the excess bandwidth to the spiders... for now.. ;)
Allowed AV in forever. They had 50k pages indexed and were spidering them 20-30 times a year. They sent us a total in 3 years of 350 visitors. They cost us a 1000 fold in bandwidth what they sent us.
Homey-don't-play-dat. Worse numbers than AV.
> User-agent: Copernic
Why would we even consider it?
> User-agent: ia_archiver
> User-agent: ia_archiver/1.6
> User-agent: Alexibot
Security and liability risks.