homepage Welcome to WebmasterWorld Guest from 54.221.175.46
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Deny or Not to Deny /images
Denying /images folder
quebrada




msg:1527512
 2:32 am on May 1, 2004 (gmt 0)

Is there a reason not to deny spiders access to the images folder?

Q

 

Stefan




msg:1527513
 2:36 am on May 1, 2004 (gmt 0)

You get hits via image searches, at least we do, probably about 10% of our traffic. Some of them take a look around the site.

Of course, you get hot-linking too...

jim_w




msg:1527514
 4:45 am on May 1, 2004 (gmt 0)

I had a problem with competitors using the image search to steal our ideas, so I banned the image robots. Our rankings did drop. So now I only filter out from the bots the images I donít want to be searched for. I do this by using a script to send the requested image. If the request is not from our domain, another image get send that says it is stolen. And sure enough, after the S L O W (G) bot finally got around to getting most stuff updated, our rankings went up. However, (G) is now getting back to the same old, same old, by not knowing how to handle 301ís. After getting our link pages 301ís straight, I noticed in my log today, they are back to trying to fetch the old went pages, getting a 301, and then not coming back. Not very professional in my opinion.

As a matter of fact, I would be embarrassed if I was (G) this isnít a small glitch, it has been going on for more than 6 months now and it should not be a hard problem to fix if they wanted to. At least in 6 months they should have fixed it. That should reflect on their IPO value, oh, Iíd say about 6 months later as investors start seeing what is going on. DUH!

cabowabo




msg:1527515
 2:37 pm on May 3, 2004 (gmt 0)

Jim,

You denied the images folder and your rankings dropped in the SERPS? Did you test this? I often modify client's robots.txt files to exclude the images folder from the spiders to keep hi-jackers off the site as much as possible. Plus, I have even seen issues with slightly corrupted image files causing the search spiders to feeze.

Cheers,

CaboWabo

jim_w




msg:1527516
 4:46 am on May 4, 2004 (gmt 0)

Sorry this is soooo loooong :-(

Well that's the way it looks. No I did not test it due to the fact that (G) takes too long to index pages, blah, blah, blah, and since I'm not an SEO, I have too many other things on my plate to figure it out and remember verbatim what I did. With (G) 'playing' with their software, can anything really be certain with or without testing. It could be a perception that something that may test true one week, will not the next true the next. (I wonder if they do that on purpose to confuse SEOís? ;-)

However, that was one of about 5 things I changed after the infamous FL update, none that were anything major. It took me from the bottom of the 2nd page to the top for a specific 2KW search, one word being a common word, six. Of course it could also have been DMOZ updates, etc., etc., etc. I would be at the top of the 1st page if it werenít for Amazon taking 2 places and a portal that link-loaded DMOZ to get on top having 2 pages. That is 2 results of the 10 total on the default page that are the same domains. Or 1/5 of the SERPS on (G) for those KWs are the same domains. Remember when you would get 100 pages from the same domain on a search. It looks like that. (I think (G) came full circle.)

It may be moot anyway because I have several visitors a day that show up via the image search. Maybe some end user finds their SERPs are better if they search images? So if I am getting traffic from the images, then I want some of them there.

>>to exclude the images folder from the spiders to keep hi-jackers off the site as much as possible

Yea, that was my way of thinking as well. But, it may be just a waste of time in the long run. I use to try to keep the images out, but I have found that putting our URL in the images, using a gray see-thru font, works the best. Locks ONLY keep honest people out. And it seems to be a human trait that if humans can get away with something they will. It is indeed sad.

If they want to invest the hours it would take to get our URL out of a line or bar chart without messing it up, well, then there isnít anything Iím going to do on the server that will stop them.

On some pages though, I still make them work as much as possible to get them, including .htaccess, perl script, and JavaScript. But if they really want them there isnít anything you are going to do, so you might just as well try to increase your traffic with them, or at least that is the way I look at it. Theory being make it as much work as possible till they decide they have spend too much time doing it. Of course if their time is only worth $.50 an hour, then one would have to do a lot more to pile the labor on.

Krapulator




msg:1527517
 4:43 am on May 5, 2004 (gmt 0)

Denying the spider access to the images folder will have no negative effect on the ranking of your pages.

jim_w




msg:1527518
 4:57 am on May 5, 2004 (gmt 0)

Krapulator

In theory. However, based on what I have seen with (G) since the FL update, I personally would NOT make such a bold statement.

In theory, a 301 redirect should not have a Ďnegative effect on the ranking of your pagesí, but it did and still does because of either a reoccurring bug, or because they keep going back to old code or something else.

I posted about by 301 problems with (G) in several messages here. Theory and practical application does NOT always go hand-in-hand.

Monkscuba




msg:1527519
 6:41 am on May 5, 2004 (gmt 0)

images.google.com is the 7th best referrer in our logs. No way I would want to ban any bots! There are a lot of pictures in our site so plenty of image bot food.

If there are certain images you want to protect, you can always ban bots from only those images, while allowing access to the rest.

Krapulator




msg:1527520
 12:19 am on May 6, 2004 (gmt 0)

Jim,

I have blocked the imagebot for the last two years and enjoy healthy rankings.

GG has stated in the past that it does not have any impact at all.

I just can't see any logical reason why they would punish sites for not wanting to be included in the image search.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved