Welcome to WebmasterWorld Guest from 54.145.136.73

Forum Moderators: goodroi

banning "vscooter" but allowing "scooter"

   
9:01 am on Nov 20, 2003 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Like many of you, I don't wish my images indexed and placed in some pic gallery for homepage builders to harvest. Alta Vista's "vscooter" is just such an image bot (imo). I found it does follow robots.txt disallows - but - the other AV website crawler "scooter" would also not index. This one I do want to get pages.

Solution: I removed "vscooter" from robots.txt and banned it in .htaccess without the starting anchor (^); due to the other preceeding UA stuff. The other "scooter" now does not see anything resembling the word "scooter" in robots.txt so it goes on and crawls my site.

Disclaimer - this is how I have successfully dealt with the issue. Others may have alternate approaches or not even have had the problem in the first place.

9:04 pm on Dec 9, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



keyplyr,

In your original setup, did you have separate and distinct "User-agent" lines for both scooter and vscooter, or were you relying on one of them to 'fall through' and hit a default "User-agent: *" record?

Just curious,
Jim

9:10 pm on Dec 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>Like many of you, I don't wish my images indexed and placed in some pic gallery<<

Yea, I felt that way prior to the ‘update’. Now I want them indexed so that it is a few more places someone can find us. But what I did was I wrote in the images, in big letters and using a color that did not take away from interpreting the image, ‘Sample www.mydomain.com’ Sure they could remove it, but it probably isn’t worth their time. Just an alternative.

9:15 pm on Dec 25, 2003 (gmt 0)

10+ Year Member



i have just posted some interesting findings and (my) conclusions about altavistas multimedia crawler at:

[webmasterworld.com...]

 

Featured Threads

Hot Threads This Week

Hot Threads This Month