Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- paging google! robots.txt being ignored!
Dijkgraaf - 8:31 pm on Nov 9, 2005 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/782.htm
And those 35000 aren't in a seperate directory from the pages you do wan't crawled?
Brought to you by WebmasterWorld: http://www.webmasterworld.com