homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Blogger, Google webmaster tools and Robots text with /search
Does this robots.txt blocking matter?

10+ Year Member

Msg#: 4152652 posted 11:44 pm on Jun 14, 2010 (gmt 0)

I have a google blog on one of my sites. I recently converted from FTP publishing to google publishing. The site is located at blog.example.com.

If I look in my crawl errors I see

URL Detail Detected
http://blog.example.com/search/label/widgets URL restricted by robots.txt Jun 8, 2010
http://blog.example.com/search/label/widgests2 URL restricted by robots.txt May 30, 2010

My robots.txt for the blog subdomain shows:

User-agent: Mediapartners-Google

User-agent: *
Disallow: /search

Sitemap: http://blog.example.com/feeds/posts/default?orderby=updated

Does this matter? I'm not sure exactly what it is blocking.. ie just the labels page or a search that results in labels.

Do I need to do anything with this or should I just let things lie peacefully



[edited by: tedster at 1:11 am (utc) on Jun 15, 2010]
[edit reason] switch to example.com - it can never be owned [/edit]



5+ Year Member

Msg#: 4152652 posted 2:05 pm on Jun 15, 2010 (gmt 0)

Change your robots.txt file.
This line:
Disallow: /search
needs to be edited to look like this:

Your robots.txt file is specifically preventing bots from crawling the /search folder level of your site and *everything* below it.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved