Welcome to WebmasterWorld Guest from 54.197.133.54

Forum Moderators: goodroi

Message Too Old, No Replies

Google maximum characters robots.txt ?

     
10:32 pm on Nov 19, 2009 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2005
posts: 211
votes: 0


Does Google only accept a robots.txt of a certain maximum length?

I have just looked through my robots.txt as displayed by Google webmaster tools 'Crawler access' and it is cut off approximately 40% through.

My robots.txt file is just 13kb

Many thanks

4:54 pm on Nov 20, 2009 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2005
posts: 211
votes: 0


This appears to be a bug whereby the validator tool accepts the whole robots.txt contents except when secondary bots are selected from the bottom drop down menu.