Newbie SEO question: Site not indexed after 1 year

[MarkW]

Hi guys,

I own a hobbyist website <snip> and I'm having trouble getting all of my pages indexed on Google.

Actually, indexed anywhere would be more apt. It's been almost a year and I'm getting nowhere.

It's a built with Wordpress and utilizes several free and commercial plugins. I've also recently optimized it for mobile as it seems to be a major necessity with new websites these days.

One problem I've had is that in the early days I did chop-and-change descriptions, meta titles and keywords etc... Now I've settled on a design and I'm happy with what is infront of me.

Just really perplexed by Google and it's algorithms. I've read hundreds of articles scattered about the web on this issue and even taken a few courses (as a matter of desperation) from so-called SEO experts via Udemy on SEO optimization and building up brands.

Can someone take a look at the link above and offer me some more pointers and guidance regarding this. I'd appreciate it. Always eager to learn.

Regards,
Mark W.

[edited by: Andy_Langton at 9:43 pm (utc) on Apr 17, 2016]
[edit reason] No specifics, please - see stickymail [/edit]

[Andy Langton]

Hi MarkW, and welcome to WebmasterWorld [webmasterworld.com] :)

As a start, I think we should clarify some of the technical language:

If your site is not crawled after a year, it would mean Googlebot is not visiting your pages. This would usually mean you have no links to your site, you haven't used a sitemap, or your pages are robots excluded.

If your site is not indexed this would mean that Googlebot crawls your site, but pages don't appear in results at all (e.g. check with a search for site:example.com). This could be because of serious quality problems, robots exclusion via meta elements or HTTP headers or a few other possibilities.

If your site is not ranked you have slightly trickier problems to diagnose. You might check if your pages appear for exact phrases and similar checks to make sure there aren't serious underlying problems. However, it could be that the keywords your pages target are simply too competitive, if you're operating in an area with established sites and are not growing your site's profile quickly enough to keep up.

[dipper]

in addition to the helpful advice from Andy to clarify "robots excluded" that specific point could mean two things.

Excluded in the robots.txt file and/or excluded in the html source of the site/pages

re: robots.txt look for something like this
User-agent: *
Disallow: /

re: source, look for something like this
<meta name="robots" content="NOINDEX,NOFOLLOW"/>

[smallcompany]

For WordPress, under Settings/Reading, there's a setting called "Search Engine Visibility" with the checkbox for "Discourage search engines from indexing this site"

It should not be checked.

[aristotle]

Google webmaster tools [also called search console] can give specific definitively answers to your questions.
If you haven't enrolled your site, you should do so. You can use it to submit pages to google's search index.

Even more, you can find out exactly how many times your pages have appeared in google's search results over the past few months.. These are called page impressions. You can create a graph of daily impressions in the search traffic --> search analytics section. This will tell you if your pages have been appearing in google's search results, and how often they've been appearing.

People naturally tend to focus on clicks, but I also like to look at impressions. It's interesting to create a graph that only shows impressions and look for trends.

[frankleeceo]

It seems like you mean some pages get indexed and many are not. Not all of the pages get indexed, especially if you have pages that are very similar to one another. More often than not, getting all the pages indexed does not warrant more traffic, especially for a small niche site, so I would personally focus on the traffic and content, rather than the "# of pages" in the index.

However, to make sure Google correctly indexes all the pages. Make sure all the pages are as different from another as possible. From description, title, to content. The more similar the pages are, the less likely they get indexed or ranked, your own pages crowd your other pages out.

Wordpress has quite a bit of default canonalization features with all the archives and subpages. I would personally disable those or at the minimum noindex all of those as well if they are not necessary for your site's architecture.

I would next diagnose the site to see the pages that are getting search traffic, and figure out the keywords that the site is actually ranking for. Figure out the current competitions and what sites are ranking for your keywords. And compare that to your own site's offering.

I operate a few wordpress niche sites and I have not had any issues with my strategy. But I pick my battles and avoid highly competitive areas unless I believe I have a shot at beating out my competitors.

[MarkW]

Hi guys,

Thanks so much to all for taking the time to respond. Some fantastic points were raised.

I recently found a fantastic SEO tool that generates a real-time analysis of a given website. I put my site through the process and received a score of 57.1 out of 100.

My major stumbling blocks were keyword consistency, URL rewrite (I use taxonomies which adds a "?" into most URLs, not entirely sure how to combat this one), blocking factors "flash" (Adsense maybe?), mobile speed slow (runs very fast on mine and my wife's devices, not sure about that one), language (declared English, detected Croatian? WHAT?) and my backlinks are almost non-existent.

Also, my load speed is about 4.5 seconds on a desktop which isn't the best. On a mobile it loads within the 2 second fault line.

Within the SEO section of the analysis, everything seems to be okay though. My title length is good (50 characters), meta description (150), headings, alt attributes, in-page links, no broken links, www resolve, ip canonicalization, robots, sitemap, no underscores, domain registration (domain is 2 years old) and a blog exists.

Another thing is I use a hell of a lot of plugins. A security one in particular came with a firewall which adds lines to .htaccess. I disabled it and I'm now eager to see what effect that will have. I don't think there is any need for such a plugin while my site is still trying to build up it's branding. Does anyone agree?

[Andy Langton]

SEO tool that generates a real-time analysis of a given website

Be very careful with such tools. In all the years I've been doing SEO, I have never come across a tool that provided scoring etc. that is of any practical use. The data might be useful, and some tools are great time-savers, but you absolutely cannot trust scores/grades etc. Predominantly, this is because the implication is that this is how search engines work (a checklist or scoring system) and they definitely do not.

E.g.

My title length is good (50 characters), meta description (150)

You could just tell your website platform to automatically cut titles and descriptions to that length (and some do it without asking you). This does not tell you anything at all about whether your titles and descriptions are in any way suitable. If they're terrible, the automated approach is concealing problems that you might otherwise have uncovered.

keyword consistency

What does that even mean? If they're saying you don't use the exact same keyword throughout content, then you should seriously evaluate whether you could trust any of the info from such a tool.

My strong recommendation is that until you feel you totally understand what the various important factors are, automating things is a mistake. I say this as a happy to subscriber to a number of SEO-related tools.

[MarkW]

Hey Andy, thanks for that heads up. It's so easy to read too much into SEO tools like that because it's generally the only feedback we get in terms of optimisation. The results are right in front of us, the design looks professional and it's been recommended to me countless times.

I guess I'm just going to wait this out and see what happens with Google (and Bing for that matter) in regards to ranking. Hopefully, things will change for the better and it won't take too long.

[paranoid android]

Just a quick question - did you check the obvious thing? Did you make sure you have a robots.txt file that allows Google to crawl the site. Did you check Webmaster Tools to see if your site is being crawled, etc?

[Andy Langton]

I guess I'm just going to wait this out and see what happens with Google (and Bing for that matter) in regards to ranking

It's certainly going to help your sites to be as involved as possible - rather than waiting. If you mean that you can wait, then this is in your favour - you can be building up your site and its reputation in a way that will certainly add to your potential for future success. If you mean literally wait and see, I don't think this is likely to work out.

One point from your tool:

backlinks are almost non-existent.

If that's true, it would dramatically affect your site's potential and you should address it as a priority. Too few links will prevent you from ranking for almost everything except very specific keywords. One way of looking at it: who do you know that would vouch for your site? Do search engines know that they would vouch for your site?

[tangor]

Are you having the same problem in Bing?

We can't talk about your niche/site in public, but there are niches that simply do not generate much traffic, if any.

Secondary, are there any promotions you have done, such as radio, tv, newspapers, community/gov? These are not WEB solutions but real life solutions. If your niche and your real world connection can grow together you will eventually get those important links. But PEOPLE have to know you are there, first. Algorithms are NOT PEOPLE.

[flatfile]

Going by your first reply I'd say your biggest problem is links and unfortunately you can't just wait this one out. Since you say backlinks are almost non-existent, I assume this is a domain that hasn't been registered before and therefore unlikely to be penalized. If you don't know where to start, use SEO tools to look at your competitors' backlinks and take it from there. The beauty of SEO is that once you start showing up in search results, more sites will start linking to you.

[anallawalla]

MarkW, can you comment directly on the point made by others -- have you set up an account in Google Search Console? If so, what issues is it reporting?

[MarkW]

Thanks so much for all of these responses guys.

Just to pick up on a couple of points:

1) The niche is very popular. In fact, it's almost overkill in that chosen area but I offer a different perspective and I'm doing things slightly differently. I know what my main focal points are and what separates me from the crowd and I'm working hard to generate interest in them.

2) Regarding the backlink issue. Flatfile's response got me thinking. I had planned on waiting for them to build up naturally because I'm so confident that once the word is out about my site, the interest will peak. Having said that, what am I waiting for exactly?

So I did what was suggested and checked out the backlinks for my competitors. I visited a couple, found some contact information and composed an email:

Hello [site name],

General blurb about what my site offers...

The only thing that is against us at the minute is that we are fairly new. We need backlinks. Backlinks help us to generate visits and higher rankings within Google, Bing and the rest of the major search engines.

If we could set up some kind of collaboration between our two websites, I’m sure that eventually we could form a generous partnership with a healthy exchange of visitors, and maybe revenue, at some point down the line.

For the time being however, I propose an exchange of links in maybe the footer or on a separate affiliation page. That would help us enormously and we would be (me in particular) hugely grateful to you guys for doing this.

I’m sure you guys understand what it’s like for a new brand at the beginning and how difficult it is to become a recognisable face amongst the crowd.

If we could set up some form of collaboration with a reputable website such as yourselves, it would really help us along massively.

I appreciate you taking the time to read this. I await your response.

It's slightly brazen but I feel it's worth a shot.

I've also decided to open a YouTube channel and start creating videos sharing some of my content.

Also, this is the view of the search console: <snip>

Crawl errors:
3 Server errors
315 Soft 404
532 Not found

Search Analytics:
37 Total Clicks

Sitemaps:
4,496 URLs submitted
1,692 URLs indexed

I have a lot of 404 and "not found" errors from links that shouldn't exist so I'm not entirely sure why they're showing up in the console.

Regards

[edited by: Andy_Langton at 11:06 am (utc) on Apr 20, 2016]
[edit reason] Stats, not screenshots - image hosting can be temporary [/edit]

[Andy Langton]

From your Google data, is 4,500 around the number of pages you believe you have? You could try crawling it yourself with a tool like Xenu Link Sleuth.

>> 315 Soft 404

Do you have pages that say "no results found", have placeholder content or similar? Also, if you site is fairly new, why so many actual 404s?

>> 4,496 URLs submitted
>> 1,692 URLs indexed

That doesn't look particularly good. If you have too few links, sitemaps will not help your rankings (they just "force" Google to discover your content, potentially masking the underlying problem). Otherwise, the implication is that Google is no fan of your content.

Incidentally, the numbers confirm that you have pages indexed] (although you may still have indexing issues). Your problem is getting those indexed pages to rank.