homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Google site maps and "robots text too long" error

10+ Year Member

Msg#: 34740 posted 2:20 am on Jun 13, 2006 (gmt 0)

When logged into my google site maps account, I tried to test my robots.txt and got error message

" Must be at most 5000 characters "

I'm blocking google from about 600 pages to avoid dup content filter. (tried meta tag block, but yahoo responds to "googlebot" meta tag also)

Is this just an error that google site maps is posting. Or can your robots.txt be too long?



WebmasterWorld Senior Member 5+ Year Member

Msg#: 34740 posted 3:25 am on Jun 13, 2006 (gmt 0)

I, too, also get the error. I have a long robots.txt file.
In the box that shows the contents of your robots.txt, clear half of it and then test it with the various user agents.
Then check the other half of the robots.txt file with the user agents.

If there are more then 5000 characters in that box (which only shows what is contained in your robots.txt, all edits are not saved to your robots.txt file) you will also get the error.

Hope that helps, your not alone on that message.


WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 34740 posted 12:38 am on Jun 14, 2006 (gmt 0)

I remember the Google URL Console (Removal Tool) has a limit of 50 lines for the robots.txt file.

It would be nice if all of the Google Tools had a consistent limit that was well advertised.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved