Welcome to WebmasterWorld Guest from 54.196.42.8

This page requires javascript to function properly. It seems that your browser does not have Javascript enabled. Please enable Javascript and press the Reload/Refresh button on your browser.

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google site maps and "robots text too long" error

Junior Member

joined:Sept 25, 2002
posts:106
votes: 0

When logged into my google site maps account, I tried to test my robots.txt and got error message

" Must be at most 5000 characters "

I'm blocking google from about 600 pages to avoid dup content filter. (tried meta tag block, but yahoo responds to "googlebot" meta tag also)
(<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">)

Question?
Is this just an error that google site maps is posting. Or can your robots.txt be too long?

Senior Member

joined:Apr 28, 2006
posts:1044
votes: 2

I, too, also get the error. I have a long robots.txt file.
In the box that shows the contents of your robots.txt, clear half of it and then test it with the various user agents.
Then check the other half of the robots.txt file with the user agents.

If there are more then 5000 characters in that box (which only shows what is contained in your robots.txt, all edits are not saved to your robots.txt file) you will also get the error.

Hope that helps, your not alone on that message.
Jake