homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Google and Yahoo Both Ignoring Robots.txt?

Msg#: 4261259 posted 4:30 pm on Feb 1, 2011 (gmt 0)

Hey All,

I am back for some more expert advice, thanks in advance!

I have a personal blog (wordpress) and have decided to disallow indexing of tags and categories through an SEO plugin in the CMS. I have also disallowed any comment pages, any feed pages and basically anything that's not the actual post in my robots.txt.

My problem is that for some reason both Google and Yahoo are indexing some of the tag pages as well as some of the feed pages.

For example:

I have these all disallowed in my robots.txt file. Is there any other reason these would be indexed as pages? They aren't linked from anywhere, either on my site or anyone else's.

Interestingly, Bing is the only one following my instructions...




WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4261259 posted 6:33 pm on Feb 1, 2011 (gmt 0)

A robots.txt disallow rule stops spidering - but URLs can be indexed even though the content isn't crawled. The information listed can come from internal or external backlinks, DMOZ etc.

However, seeing the feed indexed on a Wordpress site seems odd. Are you seeing these URLs only for a site: operator result - or are they ranking for some kind of regular query?


Msg#: 4261259 posted 6:46 pm on Feb 1, 2011 (gmt 0)

@tedster: Just the "site:" operator, nothing is returning in any SERPs. It's just annoying to see it, was wondering if there is any specific reason this could be happening.

And as I mentioned Bing's "site:" operator shows exactly what I want it to show but Yahoo and Google's does not. Weird.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved