Removed index.html - does Google spider contents of folder?

[Lame_Wolf]

Ages ago I noticed people could see my folders, so I added an inedex.html to every folder. It didn't say much, but it stopped people going through folders that I didn't want them to go through.

Recently, I was in (index manager) cPanel, and turned every folder to "no indexes" and deleted all index.html files.

This was around the time Google Images changed into the way they are now. I lost a lot of visitors (according to sitemeter - because they were going directly to the image, therefore bypassing the sitemeter code.)

A couple of weeks ago, my hosting provider upgraded their servers - and it was far from smooth. My site was offline for hours at a time, and continued for a week or so.

The dust seems to have settled now, and has given me a bit of time to see what is going on. I was doing 10k visitors a day, but lucky to get 4200 now. (Peek season is 18k)

Every page on the sites that I manage, haven't been cached since 25th of March (just before the upgrade)

I have been using "Fetch as Google" on individual page, and was successful.
But when I try to "Fetch as Google" on a folder, it comes back with "Error"

Google says "HTTP/1.1 403 Forbidden"

If I place a dummy index.html file in there, then I get "HTTP/1.1 200 OK"

Was I blocking Google from spidering those folders once I removed the index.html files ? (I know they can be spidered/cached by inbound links etc.)

Should I put back the dummy index.html files ?
Thank you for any help on this.
LW

[TheOptimizationIdiot]

Shouldn't be an issue. I run the same thing and have for years over a number of sites. When you "turn indexes off", rather than Apache serving a default "index" with a list of all the files in a directory it doesn't allow access to dir/ without an index.ext (or whatever you have set as directory indexes) actually being in the directory. It serves a Forbidden error instead.

It does not effect an entire directory, only requests made for dir/ and should be no big deal as long as all the files that should be accessed are linked from somewhere. With the issues "stub" pages can cause I think it was likely a good move.

[Lame_Wolf]

Thank you, TheOptimizationIdiot. I thought as much, and wanted to make sure. With all the hassles with Google Images and my hosting provider, my brain has turned to jelly.

The structure is like this...

example.com/folder/

/folder/ will contain html files, along with 2 or more folders which hold images.

May be I am paranoid atm, but I started to look deeper when I saw that Google hadn't cached anything from the day before I moved servers.

Thanks again. It has put my mind at rest.

[TheOptimizationIdiot]

NP and I definitely understand this:

May be I am paranoid atm, but I started to look deeper when I saw that Google hadn't cached anything from the day before I moved servers.

There's something "goofy" I've been seeing over the last couple of months with the caching and site: searches too.

This might not be exact on the time line, but it was this "odd". I saw a preview of a page with content from a couple of weeks prior, but the template was from something 6 or 8 weeks earlier. The content was updated after the template was changed, so there's no way it was "just out of date", because the preview showed the updated content.

The caching has been very slow to update for me too.

The other day I had them send visitors to pages I could not find in the site search. I launched a new directory on the site, so there's no way it was "missed" or "some obscure page not returned" for my site:example.com/dir/ query, because I got "no results found" for the query. But Google referred two people to pages in that directory before I went and checked to see how many of the pages they had.

I've often seen pages returned for a site: query before the show in the SERPs or send traffic, in fact that seems fairly normal to me, but the other day is the first time in almost a decade I've seen them send visitors (return pages in the actual SERPs) before I could find even a single page in a directory with the site: search.

[Lame_Wolf]

There has been some odd instances with Google's caching. Someone posted a message in WebmasterWorld before. I tried to find an answer on Google for him. Google said it was cached (IIRC) 4 hours ago, yet the person hadn't long posted it to the group.

With my problem, it was partly my .htaccess file, google image change, host screwing up the upgrade. Even though it has remained up for longer, there's been some strange errors in my error log.

I suffered badly when Google changed their Image Search. So I lost a number of visitors because of that. I knew my .htacces was messy, but it did kind of worked. Then I found that it was allowing hotlinking, even though I thought I had stopped it.

So I rebuilt my .htaccess and forgot to include the image side of google to cache them. Then, my host upgraded and I lost a huge amount of downtime. WMT had messages saying they couldn't access the site etc.

In a panic, I reverted to the older .htaccess with a little modification. It seems to be working, but when I check site:www.example.com in images, there is less and less each day... another reason why I dug further into it.

[lucy24]

OK, what am I missing here? Why on earth don't you simply say "Options -Indexes" in your top-level htaccess? Or rather: Why wasn't it there all along? Auto-indexes are for things like long lists of publicly downloadable files that you've dumped out there for everyone to paw through.

But when I try to "Fetch as Google" on a folder, it comes back with "Error"

Google says "HTTP/1.1 403 Forbidden"

If I place a dummy index.html file in there, then I get "HTTP/1.1 200 OK"

Was I blocking Google from spidering those folders once I removed the index.html files ?

NO. You were blocking google-- and everyone else, good and bad, human and robot-- from seeing an auto-generated file called "index.html" which lists every single thing in the directory.

There's one huge question you didn't answer. How do visitors-- whether human or robot-- know what's in those directories? I have to assume you don't expect your normal human visitors to blunder into a raw index and click on filenames. If an individual file has a link to it, that link is completely independent of anything else in the directory. That is, physically independent.

I've always assumed that once a search engine has seen /directory/filename.html they will routinely try for /directory/ and see what turns up. I may be wrong about this, because I've got quite a few directories that contain accessible pages but no index file, and have never seen the googlebot getting redirected. But wouldn't you expect them to try?

[Lame_Wolf]

OK, what am I missing here? Why on earth don't you simply say "Options -Indexes" in your top-level htaccess?

Because you were assuming I knew .htaccess. I don't, and I find it hard to understand it.

Or rather: Why wasn't it there all along?

Well, when I started the site, I barely knew html and was self-taught. It was on a windows server, so .htaccess wasn't used.

It was years later that I moved to an Apache server. I was using a reseller, and it was they who made the .htaccess file. They also said that it was done by hand, rather than in cpanel because cpanel was buggy.

Auto-indexes are for things like long lists of publicly downloadable files that you've dumped out there for everyone to paw through.

Before I made my dummy index.html files I spotted some results that were example.com/folder/ and needed to block them from being shown, as they displayed the file structure of the site - something I do not wish to have.

NO. You were blocking google-- and everyone else, good and bad, human and robot-- from seeing an auto-generated file called "index.html" which lists every single thing in the directory.

It wasn't an auto-generated file. I used my template - which is header, menu, footer, with a message in the content section to ask people to use the menu to navigate.

If you came to my site and browsed it, you will only see URL's that end in .html
I don't use URL's like example.com/folder/

There's one huge question you didn't answer.

That's easy, unless there was a post that was removed and I didn't see it, then it wasn't asked.

I can only answer questions if they are asked. I have looked at the remaining posts above, and I do not see any questions, let alone one that wasn't answered.

How do visitors-- whether human or robot-- know what's in those directories?

Like I said in my original post, I can "fetch as Google" on individual files that are inside those folders, and comes back as a success.

"Fetch as Google"...
example.com/folder/an-article.html = successful
example.com/folder/ = error

I have to assume you don't expect your normal human visitors to blunder into a raw index and click on filenames.

I have had people who have dissasembled my URLs. I even do it myself on other people's sites. And they could have blundered into it if they came from a search engine, because they were showing up in the SERPs - but rare.

[g1smd]

It was years later that I moved to an Apache server. I was using a reseller, and it was they who made the .htaccess file. They also said that it was done by hand, rather than in cpanel because cpanel was buggy.

Yes, cPanel and other such systems generate awful htaccess files, with broken syntax and many errors.

It wasn't an auto-generated file.

If you have no DirectoryIndex directive and you allow indexing, the server auto-generates a list of the filenames that are within the folder when the bare folder URL is requested. If there is a filename defined in the DirectoryIndex directive, users see the content of that file when the bare folder URL is requested. If you turn indexing off, users see a 403 error if there is no DirectoryIndex directive and no index file and the bare folder URL is requested.

[Lame_Wolf]

Yes, cPanel and other such systems generate awful htaccess files, with broken syntax and many errors.

You would have thought that after all these years, cpanel would get it correct. Both .htaccess and cpanel have been here for years, so they should have ironed out any bugs by now.

If you have no DirectoryIndex directive and you allow indexing, the server auto-generates a list of the filenames that are within the folder when the bare folder URL is requested.

That is exactly what I was seeing before I put my dummy index.html files into every folder.

If there is a filename defined in the DirectoryIndex directive, users see the content of that file when the bare folder URL is requested.

That is exactly what I was seeing after I put my dummy index.html files into every folder.

If you turn indexing off, users see a 403 error if there is no DirectoryIndex directive and no index file and the bare folder URL is requested.

That is exactly what is happening now.

[g1smd]

The combination of defining a filename in the DirectoryIndex directive - or not - and turning the Indexes function on or off gives complete control over what vistors see when the bare folder URL is requested.

[lucy24]

I can only answer questions if they are asked.

Nonsense. You can anticipate questions and prepare the answers ahead of time. In fact you won't last long in business if you don't. "Does this come in a Large?" "Gee, I dunno, nobody ever asked that before, let me go find out." And then while you're checking the inventory, the customer wanders off and finds a site that lists available sizes on the order form.

How do visitors-- whether human or robot-- know what's in those directories?

Like I said in my original post, I can "fetch as Google" on individual files that are inside those folders, and comes back as a success.

That wasn't my question, though. The question is: How do people other than yourself know that the files exist?

I have had people who have dissasembled my URLs.

Sure. And that's what your custom 403 page is for. "Sorry, passing human, there's nothing here." But you can only disassemble an URL if you have an URL to disassemble.

If you came to my site and browsed it, you will only see URL's that end in .html
I don't use URL's like example.com/folder/

But search engines do, so you've now got Duplicate Content all over the place, unless you're forcibly redirecting from /folder/ to /folder/index.html. That's "redirecting", not "quietly rewriting" via mod_dir or IIS equivalent. And if the only function of those named /index.html pages is to prevent people from seeing the auto-generated index, you've got some pretty thin pages in high-profile locations.

Before I made my dummy index.html files I spotted some results that were example.com/folder/ and needed to block them from being shown, as they displayed the file structure of the site - something I do not wish to have.

I know that's what it looks like. But what you're actually seeing is a page constructed on the fly-- again, by mod_dir or IIS equivalent-- that shows the entire content of the directory, generally excluding pages with leading . dot. That is, if you enable auto-indexing for just one directory, human visitors will still not be able to see the .htaccess file that enabled the auto-indexing. (This part is, or should be, part of every host's config file.)

[TheOptimizationIdiot]

Why the lecture?

Lame_Wolf didn't know what to do and finds this stuff confusing, so they went with what the host offered thinking it would be adequate. They found another way and wanted to make sure it would work. Nothing more, nothing less.

Since the index.html pages were dummy pages and it's stated all actual pages end in .html:

1.) "That wasn't my question, though. The question is: How do people other than yourself know that the files exist?"

Obviously visitors found the locations with "dummy index" pages in place, so your question does not need to be asked or answered.

It's already implicitly answered as "there is a way" or the contents would not have been indexed or accessed in the first place, so they could not be "less indexed" now than they were with the "dummy pages" in place. There was/is very obviously a way to find the contents of the folder(s) for both people and search engines, since previously there were "dummy index pages" preventing exposure of the entire directory contents.

Since the "dummy pages" did not expose the contents of the directory, which is why they were put up, obviously they were not necessary for the locations within the directory to be found.

2.) "But search engines do, so you've now got Duplicate Content all over the place, unless you're forcibly redirecting from /folder/ to /folder/index.html."

Who cares about duplicate content in this situation since duplicate URLs are grouped together and "the best one" is picked by Google?

It's not a penalty issue, it's simply a matter of Google showing the location you prefer to visitors in the results, but Lame_Wolf doesn't want the pages in the results any way, so no big deal on which Google decides is the canonical or "best" if they're both available, especially since they were "dummy pages" that won't likely rank for anything anyway.

3.) "And if the only function of those named /index.html pages is to prevent people from seeing the auto-generated index, you've got some pretty thin pages in high-profile locations."

The only way they're "high-profile" locations is if they're highly linked, but they're "dummy pages", which would implicitly not be highly linked, so they're not high-profile.

Only a webmaster or someone who knows the web would ever "deconstruct" a URL to see what's on the index. Ask around among people who don't do this for a living and see how many have ever even though to do that instead of clicking the links on a site. And, it's very likely anyone who would think to do it would understand they should click the links if they got a generic Forbidden error.

Also, there's no "additional weight" given to a page based on being dir/index.html or dir/ otherwise we would all just end every location in name/ and get some extra credit. Links, mentions, and things along those lines make a page high-profile, not ending in a dir/ location or being dir/index.html

4.) "I know that's what it looks like. But what you're actually seeing is a page constructed on the fly"

It's 6 of one, half dozen of the other. The "page" or "exposing of the structure" was the issue, it really doesn't matter what the mechanism was or what it's technically called, all Lame_Wolf asked is if the issue was solved adequately with the new procedure put in place.

[Lame_Wolf]

I can only answer questions if they are asked.

Nonsense. You can anticipate questions and prepare the answers ahead of time. In fact you won't last long in business if you don't.

For starters, the site has been online for 9 years. Secondly, I am not a business. I retired in my 30's.

And FYI, I did anticipate questions, questions that *I* thought were relevant to the problem.

I have had people who have dissasembled my URLs.

Sure. And that's what your custom 403 page is for. "Sorry, passing human, there's nothing here." But you can only disassemble an URL if you have an URL to disassemble.

I know, and people had a URL - most of the time, valid ones. Like the URL for this thread... webmasterworld.com/google/4562872.htm
So, someone has arrived on this thread and removed 4562872.htm

If you came to my site and browsed it, you will only see URL's that end in .html
I don't use URL's like example.com/folder/

But search engines do, so you've now got Duplicate Content all over the place, unless you're forcibly redirecting from /folder/ to /folder/index.html. That's "redirecting", not "quietly rewriting" via mod_dir or IIS equivalent. And if the only function of those named /index.html pages is to prevent people from seeing the auto-generated index, you've got some pretty thin pages in high-profile locations.

Seriously, I am not bothered if Google see dupilcate content. They all say something like... "use the menu on the left to see the site"

I added them because I didn't want to see directory listings in the search engines. Nor did I want people to breadcrumb the URLs.

I have only seen them in the SERPs when I do a site:example.com but there are other people other than myself who had done it. And I do not produce thin pages thank you very much.

[Lame_Wolf]

Why the lecture?

Luckily I had my flame-retardant pants on, but it was rather harsh.

Lame_Wolf didn't know what to do and finds this stuff confusing,

'tis true. Along with a lifetime of sleep deprevention, I have 2 medical conditions that make it hard to take things in.

Obviously visitors found the locations with "dummy index" pages in place, so your question does not need to be asked or answered.

In most instances I found that the visitor arrived on a valid URL and then breadcrumbed the site. I know I saw them when I used the site operator, and I am sure I saw one in the regular serps for something obscure. (The real page was above it, so it wasn't a thin content page.)

But, if anyone saw the results in Google showing the directory listing, when they arrived, they would have seen the contents of my dummy index.html instead. It was then a waiting game for G to come round again, and let them slowly disappear from the search engines.

Who cares about duplicate content in this situation since duplicate URLs are grouped together and "the best one" is picked by Google?

Exactly. Who cares. I'd rather rank for "click here" ;)

The only way they're "high-profile" locations is if they're highly linked, but they're "dummy pages", which would implicitly not be highly linked, so they're not high-profile.

Very true.

It's 6 of one, half dozen of the other. The mechanism creating the "page" or "exposing the structure" was the issue, all Lame_Wolf asked is if the issue was solved adequately with the new procedure put in place.

Thank you for understanding the question asked, and for answering it in a professional and politeful way.

I have been checking site:example.com a number of times a day, both for normal SERPS and Images. The number of URLs have remained around 10300. The images have dropped from 17500ish to 7,400ish but have crept up a bit in the last day or so.

[TheOptimizationIdiot]

Thank you for understanding the question asked, and for answering it in a professional and politeful way.

Np again, and I'm glad I could help. I thought the question you asked initially coupled with the information provided was very easy to understand and answer.

I also think it's impressive you've overcome the challenges you have to not only teach yourself to write HTML, but to use mod_rewrite at all. Mod_Rewrite is a very difficult "thing" to understand and use.

I know people who don't have the same challenges as you outlined who throw their hands up and say "Ahhhhh, help! I can't do this!" when they try to use it, so I think it's fantastic you've been persistent and determined enough to overcome the obstacles in your path to even use begin to use it.

[Lame_Wolf]

Np again, and I'm glad I could help. I thought the question you asked initially coupled with the information provided was very easy to understand and answer.

Thank you. I tried to give a clear picture of things, so it was clear for others to understand, even if I didn't use the correct terminology.

I am one of these people who has to picture things before I can understand them. When I read from a book or web page, I often have to read it umpteen times, and even then it isn't always clear. Sit next to me and show me, then I am away with it, with may be the odd question here or there.

I also think it's impressive you've overcome the challenges you have to not only teach yourself to write HTML, but to use mod_rewrite at all. Mod_Rewrite is a very difficult "thing" to understand and use.

Thank you, it hasn't been easy in life. I thought I was the normal one, and others had a problem. I am only coming to terms with it now and understand myself better.

Learning HTML only became easy when my ISP at the time had a tutorial about it.
They told it in a very basic way, using notepad, and they altered colours so you knew which part they were on about, and explained what the tags did etc.

Another problem I have is memory. I can do something repeatedly for months on end. Then, if I haven't done it for a while, I am lost because I have forgotten.

I haven't done any mod_rewrites though, and for php, I only got as far as "hello world"

I know people who don't have the same challenges as you outlined who throw their hands up and say "Ahhhhh, help! I can't do this!" when they try to use it, so I think it's fantastic you've been persistent and determined enough to overcome the obstacles in your path to even use it at all.

Believe me, it can be extremely stressful at times, especially if I cannot grasp things when I know I should be able to do so. I am not thick, but if I cannot create a picture, then the brain melts into jelly.

I'd like to learn .htaccess but I haven't found a site yet where I can understand it in the way I need it. I need something that explains things in a simple manner, what things like [NC] mean, when to use them/not to use them, why you do things before others, etc etc

When I do find something about .htaccess and has the answer, someone pipes up and says "it is better if you do it this way" or "you should do it this way instead" If they are both correct, then where's the problem. If one is better, then why? One day i'll get there.

Thanks for your support and kindness. I bow my humble head.

[TheOptimizationIdiot]

I'd like to learn .htaccess but I haven't found a site yet where I can understand it in the way I need it. I need something that explains things in a simple manner, what things like [NC] mean, when to use them/not to use them, why you do things before others, etc etc

[webmasterworld.com...]

[webmasterworld.com...]

If you need more or something different, let me know and how it could be better... If you do I'll see if I can find JD01 to post another one, and if he's too busy or something, then I'll see what I can come up with. Not sure if I can post the way he did though. Funny how he just disappeared, isn't it ;)

Thanks for your support and kindness. I bow my humble head.

Oh, no... I don't have the same challenges to overcome you do. All my respect for being able to persevere through adversity. I think it's awesome to see someone overcome challenges they face and I'm inspired by those who do. So, Thank You!

[Lame_Wolf]

If you need more or something different, let me know and how it could be better... If you do I'll see if I can find JD01 to post another one, and if he's too busy or something, then I'll see what I can come up with. Not sure if I can post the way he did though. Funny how he just disappeared, isn't it

Thank you for the links. I have had a quick look, and I think I will be able to understand some of it. I did wonder what the "chinese hat" meant. That is one thing that is a little clearer. :)

Oh, no... I don't have the same challenges to overcome you do. All my respect for being able to persevere through challenges and adversity.

Oh you do. We all have challenges. Either it be physical, mental or financial. Mine are all three. :/

I think it's awesome to see someone overcome challenges they face and I'm inspired by it. So, Thank You!

I haven't overcome them, but I understand them, and myself better. It doesn't make things easier to learn, but makes it easier for me to understand why I find it hard to learn. If you know what I mean.

Back on topic (in case someone complains) I have just checked my images with the site operator and there has been yet another slight increase, so the .htaccess seems to be working albeit rather messy. I will work on it again once the dust has settled with the site move, and my head is clearer. Thanks again for everything. If I can ever return the favour, I will.

[TheOptimizationIdiot]

...so the .htaccess seems to be working albeit rather messy. I will work on it again once the dust has settled with the site move, and my head is clearer.

If you posted it in the Apache Forum [webmasterworld.com] you might get some help with it.
(Especially if you let me know when you do.)

Thanks again for everything. If I can ever return the favour, I will.

Don't even worry about it. I've gotten a Ton of help and information from here over the years as a reader, so I'm glad I can "give back" a bit to at least one of the contributors. I don't post questions often here, because I try to give back answers out of appreciation for those who contribute to making this place the authority for information and best place to visit when I need to know something.

Back on topic (in case someone complains)

If someone complains about the conversation we're having and the mods listen, then there's something really wrong, but just in case...

I'm glad you're getting reindexed again. I'd definitely make sure everything is available on an xml-sitemap and the <changefreq> (change frequency) is set to daily (EG <changefreq>daily</changefreq>) reference: [sitemaps.org...]

[lucy24]

make sure everything is available on an xml-sitemap and the <changefreq> (change frequency) is set to daily

If you have evidence that google takes this line at face value and schedules its crawling accordingly-- not just on a first visit but every time-- I'm sure a lot of people would like to see it.

[TheOptimizationIdiot]

Well, I can't post it, but I do... If you compared the WMT crawl rate to the changes I've made in updated frequency via xml-sitemap you would too.

If you really want some proof change your update frequency to monthly for a while in your xml-sitemap and see what happens to your crawl rate. Feel free. Go ahead. But, don't complain to me if it takes Google a month to pick up changes to your pages.

Found that one out the hard way, btw. If you'd like to learn that way too, be my guest. Oh, and it doesn't change when you take the update frequency out of the xml-sitemap. You have to specify a more frequent update schedule to override what they have in the system.

Maybe some sites are small enough they ignore it, but the smallest of the ones I work on regularly is 15,000+ pages and Google listens to that little line in the xml-sitemap.

[Lame_Wolf]

Funny how he just disappeared, isn't it

I forgot to reply to this bit. Yes, it is. He could be dead. His site is parked, so it is possible.

If you posted it in the Apache Forum [webmasterworld.com] you might get some help with it.
(Especially if you let me know when you do.)

Thank you. I will do, but I do not know when it will be. Eventually, i'd like it to be able to show watermarked images and stop certain search engines from directly linking to the larger images. But for now, i'll settle for a working .htaccess which I think I have. It will have to do for now.

Don't even worry about it. I've gotten a Ton of help and information from here over the years as a reader, so I'm glad I can "give back" a bit to at least one of the contributors. I don't post questions often here, because I try to give back answers out of appreciation for those who contribute to making this place the authority for information and best place to visit when I need to know something.

I too answer far more than I ask. In fact, I think I have had more posts removed than I have asked for help. :)

I'm glad you're getting reindexed again. I'd definitely make sure everything is available on an xml-sitemap and the <changefreq> (change frequency) is set to daily (EG <changefreq>daily</changefreq> reference: [sitemaps.org...]

I have dropped a lot in the SERPS for certain keywords. Mainly due to being offline and DNS issues. Neither of which were my fault. (So far, the hosting provider have said that there's been DNS issues, problem with the container, problem with the datacenter, and a problem with a node) One long nightmare.

I have never used an xml-sitemap as I have never found the need. It's not as if the information is updated, or a comments section. It's not that kind of site. Anyway, isn't there a security risk using xml-sitemap? I am sure I read something somewhere about a site was hacked by injecting something into the map. I could be wrong, but it rings a bell.