Msg#: 4535330 posted 9:52 am on Jan 15, 2013 (gmt 0)
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
Msg#: 4535330 posted 10:17 pm on Jan 15, 2013 (gmt 0)
You are looking in the wrong place.
The problem is not that google is crawling urls you don't want it to know about. The problem is that it is inventing urls that don't exist. If it only does it now and then, this is normal: It is testing for "soft 404" responses. But if it is happening very often, you need to figure out where it is getting these imaginary URLs. There are other threads discussing this problem.