Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt need help


g1smd - 10:20 pm on Jan 15, 2013 (gmt 0)


Disallow: /*%

will disallow crawling of URLs with a % in.

However, as stated above, that's the right answer to the wrong question.


If the requests are met with a 404 or 410 response then there is nothing further to do.

If your server is returning '200 OK' then you have a bigger problem to fix.


Thread source:: http://www.webmasterworld.com/robots_txt/4535330.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com