homepage Welcome to WebmasterWorld Guest from 54.226.213.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
banning "vscooter" but allowing "scooter"
keyplyr




msg:1527583
 9:01 am on Nov 20, 2003 (gmt 0)

Like many of you, I don't wish my images indexed and placed in some pic gallery for homepage builders to harvest. Alta Vista's "vscooter" is just such an image bot (imo). I found it does follow robots.txt disallows - but - the other AV website crawler "scooter" would also not index. This one I do want to get pages.

Solution: I removed "vscooter" from robots.txt and banned it in .htaccess without the starting anchor (^); due to the other preceeding UA stuff. The other "scooter" now does not see anything resembling the word "scooter" in robots.txt so it goes on and crawls my site.

Disclaimer - this is how I have successfully dealt with the issue. Others may have alternate approaches or not even have had the problem in the first place.

 

jdMorgan




msg:1527584
 9:04 pm on Dec 9, 2003 (gmt 0)

keyplyr,

In your original setup, did you have separate and distinct "User-agent" lines for both scooter and vscooter, or were you relying on one of them to 'fall through' and hit a default "User-agent: *" record?

Just curious,
Jim

jim_w




msg:1527585
 9:10 pm on Dec 9, 2003 (gmt 0)

>>Like many of you, I don't wish my images indexed and placed in some pic gallery<<

Yea, I felt that way prior to the ‘update’. Now I want them indexed so that it is a few more places someone can find us. But what I did was I wrote in the images, in big letters and using a color that did not take away from interpreting the image, ‘Sample www.mydomain.com’ Sure they could remove it, but it probably isn’t worth their time. Just an alternative.

zooros




msg:1527586
 9:15 pm on Dec 25, 2003 (gmt 0)

i have just posted some interesting findings and (my) conclusions about altavistas multimedia crawler at:

[webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved