Welcome to WebmasterWorld Guest from 54.204.106.194

Forum Moderators: goodroi

Message Too Old, No Replies

banning "vscooter" but allowing "scooter"

     
9:01 am on Nov 20, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7576
votes: 245


Like many of you, I don't wish my images indexed and placed in some pic gallery for homepage builders to harvest. Alta Vista's "vscooter" is just such an image bot (imo). I found it does follow robots.txt disallows - but - the other AV website crawler "scooter" would also not index. This one I do want to get pages.

Solution: I removed "vscooter" from robots.txt and banned it in .htaccess without the starting anchor (^); due to the other preceeding UA stuff. The other "scooter" now does not see anything resembling the word "scooter" in robots.txt so it goes on and crawls my site.

Disclaimer - this is how I have successfully dealt with the issue. Others may have alternate approaches or not even have had the problem in the first place.

9:04 pm on Dec 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


keyplyr,

In your original setup, did you have separate and distinct "User-agent" lines for both scooter and vscooter, or were you relying on one of them to 'fall through' and hit a default "User-agent: *" record?

Just curious,
Jim

9:10 pm on Dec 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 19, 2003
posts:695
votes: 0


>>Like many of you, I don't wish my images indexed and placed in some pic gallery<<

Yea, I felt that way prior to the ‘update’. Now I want them indexed so that it is a few more places someone can find us. But what I did was I wrote in the images, in big letters and using a color that did not take away from interpreting the image, ‘Sample www.mydomain.com’ Sure they could remove it, but it probably isn’t worth their time. Just an alternative.

9:15 pm on Dec 25, 2003 (gmt 0)

New User

10+ Year Member

joined:July 24, 2002
posts:22
votes: 0


i have just posted some interesting findings and (my) conclusions about altavistas multimedia crawler at:

[webmasterworld.com...]

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members