Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Modifying Robots.txt on Blogger possible?

         

silverbytes

6:20 pm on Oct 31, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Found 51 errors in robots.txt in a blogger site. I didn't know robots.txt existed there since I didn't make one.

However there it is and looks like this:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Noindex: /feedReaderJson

Sitemap: http://example/feeds/posts/default?orderby=updated

My errors looks like this:
http://example/search/label/example%20term

How do I modify that (since I have no access to robots.txt)?

[edited by: tedster at 6:42 pm (utc) on Oct. 31, 2008]
[edit reason] de-link the examples [/edit]

tedster

10:21 pm on Oct 31, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This is part of Blogger's default robots.txt and the user cannot modify it. The Disallow rule for the /search/ directory was introduced mid-2007, because the /search/ directory introduces duplicate content to search engines.

If you've got a Sitemap with some /search/ urls listed, then you need to remove them from the sitemap if you want those error messages to go away.