Forum Moderators: Robert Charlton & goodroi
Far too many variations to allow me to set up exclusions in robots.txt so I ended up adding rel=nofollow to these links to stop them being spidered.
Of course some of these tags do lead to good pages and I have other hard-coded links around the site without the rel=nofollow so that Google can pick them up.
I am now in the position where a lot of these content pages are showing page rank (a PR 3 or 4 a lot of the time) but the page is not in Google's index.
E.g. the page mysite dot com / widget
shows a PR or 4 but searching Google for the resource shows page not found. Here is the weird bit though! If I search for mysite dot com / widgerr (i.e change one or two characters in the name of the resource) I get Google suggesting:
Did you mean mysite dot com / widget
Which suggests (as does the Toolbar PR) that it knows about the page so why is it not listing it fully?
I am at the point now where I am going to remove the visible tagging in the hope that Google will no longer see the nofollow links to good content pages and am looking to you guys to see if you have noticed anything similar. Also working on automatic tagging of content so only sensible content pages are output.
Any thoughts or suggestions greatly appreciated.
To push the point of the post (sorry for rambling initially, point not as clear as it could be) - anyone ever seen a page show toolbar PR but for the page to not actually be in the index but for typos of the page Google shows a suggestion of the page that is not in the index!
The other point of the post is do you feel it is possible that because Google is seeing a mix of internal links both with and without the rel=nofollow attribute that it is keeping these pages out of the index?
Did you mean mysite dot com / widgetWhich suggests (as does the Toolbar PR) that it knows about the page so why is it not listing it fully?
As for the nofollow'd page showing PR, I believe Matt Cutts as said this is possible.
mysite.com/widgt
expecting Google to come back with:
mysite.com/widget
I actually used widget here instead of this being a substitute word. Google actually came back with:
mysite.com/swing
And this is one of the pages that has visible PR but is not being included in the index.
I have now removed the tagging system so that these pages will no longer have internal links with the nofollow attribute and will also look to change the url rewriting to put a new directory in front of the dynamically generated pages to make sure we are away from any pages that had rel=nofollow pointing to them and will let you know if it works.
No results found for www.mysite.com/donney.
However, there are results from websites with similar addresses:
www.mysite.com/donkey
Similar pages
Again the donkey (not real result) page is a real page with toolbar PR but is not in the index.
I have now convinced myself that this is due to internal use of the nofollow :o(
Did you have any links without rel=nofollow pointing to the pages that aren't showing up? rel=nofollow shouldn't exclude the pointed-to page from the index, just the link, right?
Yes, a nice internal linking structure without the nofollow attribute was used to show Google the pages we did want indexed but I think the nofollow links on a lot of pages may have outweighed the ones without nofollow.
Have implemented changes to put the pages in a new location /widgets/page and Google is happily spidering these now. Have also 301'd the old pages to the new location and removed all nofollow tags. Will let you know what happens in a few days.
This is a historic domain with a good amount of backlinks. My experience with other sites of this nature suggests that new pages added to the site usually show up within days (a week or so at the most) yet I waited for a good two or three months with this one until I really started poking around to see what may be going wrong.
I have never seen pages with a visible PR that did not show up in the index before and the nofollow is the only thing I can think of that is different!
If you want to check the site out to make sure I am not losing my marbles then feel free to PM me :op
Looking promising so far. Before we killed the nofollow attribute around the site and put the pages in a new directory we had around 45 of the pages we wanted fully visible in the index.
At the time of writing we have 59 in the index (jumps around from 40 to 60 something is the most I have seen) but certainly more than we had before.
As for the www.mysite . com /wigget returning a suggestion of www.mysite . com /widget these are not showing up so I this kind of suggests Google did know about the other pages but did not want to include them in the index.
I am going to wait a few more days and see what else gets put in the index before fully making my mind up. We have around 250 pages used in an internal linking structure and some supporting external links to some of these pages so we have a way to go before calling it a result and adding more to the internal linking to boost the number of these pages that will be visible to Google. All I can say is that it looks promising at present.
I'm trying to figure out if I added rel=nofollow to some links I shouldn't have or if Google just generally likes more PR flow around the site.
rel=nofollow on internal links may be a flawed concept at present.
My non-supplemental pages are still down so I guess I'll remove the nofollow tags.
Have to remember though that Google had 15,000 internal pages that held nofollow content in its index when I removed the tags. As another way of trying to get the pages I want indexed I have stopped Google indexing these item specific pages to try and force it to index the static pages. A couple of weeks since doing that yet Google still holds 11,000 of these pages in the index.
All of these cached pages still contain nofollow links so even though they have been removed from the site Google will still be counting these in its internal calculations so cannot really say whether the nofollow removal has made a difference as Google still sees these in its cached pages.
I am starting to wonder if we haven't tipped some kind of penalty somewhere...
The odd one is still that the /doney page still returns a suggestion of /donkey and I now know that this is a suggestion for a page from the site rather than a spelling suggestion as I have a sitemap up with just the pages I want indexed and Google reports that all the pages in the sitemap have been indexed!
A real pain in the rear this one... I am just going to add a load more of these static pages and see what happens over the next month or so.
As all the static pages are template driven I am also looking to make changes to make more specific page titles and meta data to see if that helps. I am sure I will get over it at some point and while I cannot say for sure that nofollow has caused this problem (as there are so many other variables to take into account) I doubt I will use nofollow internally again in a hurry.
Good luck with your site, hope it picks up for you.
Yeah, traffic and orders are way up even with fewer pages in the non-supplemental index. With the recently announced changes to the mechanics of the supplemental index, I wonder if I'm chasing my tail here.
