Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt need help


lucy24 - 10:17 pm on Jan 15, 2013 (gmt 0)


You are looking in the wrong place.

The problem is not that google is crawling urls you don't want it to know about. The problem is that it is inventing urls that don't exist. If it only does it now and then, this is normal: It is testing for "soft 404" responses. But if it is happening very often, you need to figure out where it is getting these imaginary URLs. There are other threads discussing this problem.


Thread source:: http://www.webmasterworld.com/robots_txt/4535330.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com