Folder Structure To Blame For Lagging Category Pages

I suspect my client's poor folder structure is hurting his site

[RankNFile]

Could having an overly broad folder structure full of single category articles cause main category pages to see less traffic than expected? I theorize yes, but I'm still an SEO toddler. Here's the situation:

I work with a blogger who doesn't put things in category folders. So instead of:

www.catwonderfulness.com/toys/best-toys-of-2014

I see:

www.catwonderfulness.com/best-toys-of-2014

He does this with all his new content and has potentially hundreds of these single category pages out there, with more showing in Google Analytics each week.


These pages are unoptimized, yet they beat the pants off his main category pages (the ones in his universal menu), which ARE optimized. I theorize that because the single category items are always fresh content, Google is indexing them faster; I also theorize that because there is no folder hierarchy to say where to crawl first, Google may even be burning up crawl budget on these single category pages and never even seeing the main category pages.

A stretch? I want to convince him that putting those items in the appropriate category folder will make the spiders recognize their importance - as well as provide them with new content!

I hope I explained this well...I also welcome ways to test my theory! (All I can think of is trying to check the cache to see if those category pages have been refreshed...maybe looking in WMT to see if a crawl rate rise correlates with a spike of new single category articles...)

[tangor]

If it hasn't been said: Welcome to Webmasterworld!

Any SE will crawl the root folder first... so no harm no foul in this regard... but at some time there will be heck to pay trying to managed a folder with potentially 10s of thousands filenames.

And yes, fresh content gets more interest.

There may come a time that robots.txt disallows might become important so some file/folder structure could be useful.

I have one site set up that way with about 1,300 pages, all in one folder (root). Never had any problems.

I don't think we should give too much weight to site performance regarding the ENTRY page as that is how the user gets there. What we DO want to accomplish is make sure that page is sufficient in interest that the user will investigate OTHER pages (including the home page, category pages, etc.)

It's very nice to get the direct hit on the domain url and have folks stay and go find all the other goodies offered, but I'll take my traffic any way I can get it.

[lucy24]

Never mind the URLs: what does the navigation look like? Is there a single front page with 10,000 links extending into infinity-- or are there in-between pages? Again, it's not what's in the browser's title bar. (Does anyone but you and me even look at those? ;)) It's what you click to get there.

This doesn't only apply to humans. There is a type of robot that crawls all pages linked from the front page. When one of these visits me, it's ecumenical in its requests: doesn't matter if it's
example.com/aaa/
or
example.com/abc/def.html
All that matters is that they found the link on the front page. Search engines work the same way, only on a bigger scale.

Besides, it's a blog. Aren't those all produced by some type of CMS that makes your life a misery if you want the URL format to be anything other than their flavor-of-the-week?

[Robert Charlton]

RankNFile - I'm not precisely following what you're asking, but I hope this will address what may be behind some of your questions.

To emphasize what lucy24 is saying... what matters in site structure is the navigation and what a page links to. It's not about the directory structure of the urls. If anything, those can get in the way.

Also, note that on site with the default home page of...
http://www.example.com/

...that having a page with this url...
http://www.example.com/best-toys-of-2014.html

...does not necessarily mean that the home page is linking to the file...

best-toys-of-2014.html

(I'm using the .html extension for clarity here).

I recommend that you take a look at this discussion, and pay careful attention to g1smd's suggestions, and also to deadsea's comments...

How important is it to organize pages into directories?
http://www.webmasterworld.com/google/4364322.htm [webmasterworld.com]

In particular...

Yes, it is good to have a hierarchical navigation structure, but it is not always a good idea to reflect that structure into the URLs for individual content pages.

You may have to puzzle this through and read it a few times, but once you get it, I think it will be worth the effort... because I think it explains the advantage of putting files in the root. It's not about spidering. It's about avoiding url confusion when you apply different navigation paths to your pages.

Also, take a look at the Matt Cutts video... and note that injecting folders into your filepath named for keywords is not going to help anything with regard to SEO.

IMO, you should use a hierarchical structure in your navigation... but that's got nothing to do with folders. And yes, IMO, that navigation structure does affect how pages are crawled.