homepage Welcome to WebmasterWorld Guest from 54.204.58.87
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Is it a spider?
topher




msg:1525446
 5:22 am on Jun 8, 2004 (gmt 0)

Hi all,
I'm new at this, so pls forgive...
I noticed my server logs showing "hits" to an image that only is displayed when a customer's credit card is rejected by the gateway. In fact it is "hit" as much as main navigation images, and that might suggest a HUGE amount of rejections, but I doubt it.
Can spiders crawling my /images dir be the cause of this? Should I be diallowing spiders from /images?
thanks!

 

jdMorgan




msg:1525447
 5:38 am on Jun 8, 2004 (gmt 0)

> Can spiders crawling my /images dir be the cause of this?

Yes.

> Should I be disallowing spiders from /images?

It would seem a prudent thing to do. Better yet, organize your images into two directories: Those images that you'd like to use to get traffic from image searches, and those that wouldn't serve well as 'ads' for your site.
Disallow robots from spidering the non-beneficial images to save bandwidth and prevent problems interpreting your stats.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved