Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Google maximum characters robots.txt ?

10:32 pm on Nov 19, 2009 (gmt 0)

10+ Year Member

Does Google only accept a robots.txt of a certain maximum length?

I have just looked through my robots.txt as displayed by Google webmaster tools 'Crawler access' and it is cut off approximately 40% through.

My robots.txt file is just 13kb

Many thanks

4:54 pm on Nov 20, 2009 (gmt 0)

10+ Year Member

This appears to be a bug whereby the validator tool accepts the whole robots.txt contents except when secondary bots are selected from the bottom drop down menu.