Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google Indexing Difficulty for Directory Level URLs

Has anyone else noticed this?

         

jtoddv

8:31 pm on Sep 28, 2006 (gmt 0)

10+ Year Member



What I am referring to exactly is a URL such as:
A) http://www.example.com/directory/

vs.

B) http://www.example.com/directory/index.htm

Is anyone noticing a difference or difficulty in getting the A URL indexed vs. the B style URL. This is the second time I have noticed this and was wondering if it was a trend, but to me it seems A is not getting indexed nearly as effectively as B. In the past it never seemed an issue, but recently it appears it has become one. This mainly refers to newer A style URLs, not ones that have been around for sometime.

Wondering if anyone else has noticed such a trend?

jtoddv

[edited by: trillianjedi at 3:25 pm (utc) on Oct. 7, 2006]
[edit reason] Please use "example.com" for example domains - thanks :) [/edit]

tedster

11:01 pm on Sep 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I can't say I see a trend.

In fact, I'm splicing another post below that describes the exact opposite -- where a new directory-style url replaced the established page-style url, effectively throwing away backlink influence for the more established url.

Reilly

7:01 pm on Sep 28, 2006 (gmt 0)

10+ Year Member




System: The following message was spliced on to this thread from: http://www.webmasterworld.com/google/3101153.htm [webmasterworld.com] by tedster - 7:02 pm on Sep. 28, 2006 (EDT -4)


Ok,

first hello guys,

i have a website, lets call that website www.example.com

on sub pages i link always to www.exmaple.com/index.html and everything ist going fine - for now 4 years - never got problems or something.

Now, i increased my website and i have a folder called

www.example.com/folder/ and in this folder i have a index.html - and

all sub pages link to the www.example.com/folder/index.html
everything is going fine for 2 years now - never got a problem with google :(

but some days ago i got a ODP/DMOZ Deeplink on the folder www.example.com/folder/ and since that time my www.example.com/folder/index.html is LOST in Space and was kicked - and google has spidered the www.example.com/folder/

Now, my question, what can i do now? i'm really frustrated and i don't know what to do? :(

And what happens to all the Universities and so on which linked on my www.example.com/folder/index.html - is this pr lost?

tedster

11:06 pm on Sep 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



So we see in this second case, Google jumps on the new directory-style url (because it comes from DMOZ?) and in the first case difficulty in getting the directory style url to show up in Google.

I suspect, but cannot prove, that it comes down to backlinks -- their authority and PR, more than their age and history.

Bewenched

3:23 am on Sep 29, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I've seen in my logs where googlebot attempts to find a base page like

www.mydomain.com/somedirectory/

but my links are like this
www.mydomain.com/somedirectory/somepage.html

it was happening so much in the logs that I built a default page that 301's to where the bot should go.

natural number

3:41 am on Sep 29, 2006 (gmt 0)

10+ Year Member



I too feel G might have trouble indexing subdirectories... but why I can't quite put my finger on it.

tedster

4:40 am on Sep 29, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The principal issue here is that Google will spider and index whatever urls it can. If two different urls point to the exact same content, it will try to filter one out of the search results. It is up to the website to direct Google (and every search engine) about which url is preferred.

The best practice has been discussed in many of our threads recently -- Use a 301 redirect to point /dir/index.html to /dir/. Then only /dir/ will be indexed by Google eventually, but any backlink influence from links that point to /dir/index.html will still "count". It is also important to be discplined about anchor tags within the site -- make sure they all use "Form A".

There is apparently an issue here for sites on shared Windows hosting who do not/cannot have admin access. So far, every attempt to handle this redirect in such a case that has been posted here seems to create an infinite loop. The server sends the redirect right back to "Form B", then the redirect back to "Form A" nd so on.

I've got no shared Windows hosting sites (in fact, I shudder at the idea) so I never had to wrestle with this issue -- but still I am trying to discover a solution. If there is one, most likely it will use VBscript or PHP, which many Windows environments also support. But it also may require a change at the admin level of configuration, or even be "impossible".

There are lots of people looking for such a solution, so if anyone has the fix for this you will be helping many folks.

tedster

4:51 am on Sep 29, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The only "solution" I ever saw -- and really it's more of a capitulation to the frustration of Windows servers -- is never to use a directory index page.

First, disallow directory index display whenever a user agent asks for it. Second, always name your "top level page" something other than the obvious file names that are configured into the list of index files on the server. That is, don't ever use default.asp, index.htm and the like. In this one case that I just remembered, they named the directory's top page with the same character string as the directory's name:

/dir/dir.asp

This convention is easy to remember and it does avoid search engine craziness. It also gives you a target page for the 301 redirect that will not go into a loop -- but it's pretty ugly, no? Still, I do know one Windows webmaster who did this out of sheer frustration and succeeded.

Reilly

7:58 am on Sep 29, 2006 (gmt 0)

10+ Year Member



hello,

ok today all seems a bit better - the www.example.com/folder/
replaced all positions in serps of www.example.com/folder/index.html with some little positions lost (from 9 to 14 keyword kombo)
(59 to 63 in main keyword).

i never used rewrite rule conditions :( and i do not know what to do ... but i will now change all my internal link structure from index.html to / - and after that i try to learn something about the rewrite module. I have read so many times about rewrite here and rewrite there but it was never necessary to use it :(

my main question is, when i change consiquently my linkstructure to / instead of index.html and i do not use a 301 redirect from index.html to / is the pr from sites outside linking on index.html lost?

sorry, but this site is my main income site and i'm afread to make big mistakes.

jtoddv

1:51 pm on Sep 29, 2006 (gmt 0)

10+ Year Member



I could see how a DMOZ link could overide what you had previously determined to be /dir/index.htm with /dir/. That link from DMOZ would probably hold more weight in the engines eyes.

But in the instances I am talking about, the URLs do not have backlinks from outside sources. Just internal links that all lead to the same exact URL with no crossover between /dir/ and /dir/index.htm. Main sites PRs are 4 or 5 (according to toolbar). And the troublesome URL are directly linked from the homepage, not burried deep in the site.

Maybe it is just the fact that the sites don't have enough value yet?

Reilly

2:30 pm on Sep 29, 2006 (gmt 0)

10+ Year Member



man,

i swear - that ODP Link changed all - i have a lot of Universities linking on http://www.example.com/folder/index.html and Google kicked that index.html for the http://www.example.com/folder/ and replaced all Serps Positions of the index.html with the folder version.

The ODP Deeplink destroys all my 10K Sites big Website Linkstructure.

Today i changed all the internal Linking from all 10K Sites from
index.html to / . I hope that will carry my PR through the folder and push me where i was.

That was a lession to me - next i gonna link only to folder and never to index.html.

Reilly

11:01 am on Oct 7, 2006 (gmt 0)

10+ Year Member



guys, 8 days passed and now Google is showing the / and the index.html Version on the Serps :(

Title
www.example.com/folder/index.html - 24k - Cache - Similar Pages

Title
www.example.com/folder/ - 25k - Cache - Similar Pages

First the index.html and then the folder /

what is happening :( i changed all the Site Structure to /

please need advice

g1smd

1:13 pm on Oct 7, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



They will sort it out, but it will take at least a few weeks.

The 301 redirect would make it happen more quickly, if installed.

Reilly

3:07 pm on Oct 7, 2006 (gmt 0)

10+ Year Member



no , i didn't have installed a redirect 301

i don't understand nothing about rewrite module and rewrite conditions - i never used them and i hope never to use them at all

ok i will wait - thanks a lot

The lonely thing which makes me frightend is that a lot of external Links of Universities link on the index.html :(