Forum Moderators: goodroi
Is there any chance crawlers could get bogged down with this amount of disallow instructions and leave the site altogether?
My company has never used a robots.txt file before and there are many old, bad pages out there showing up on the search engines. I am trying to play clean up.
Should I be concerned about using such a long robots.txt?
thank you,
gingerspice
[webmasterworld.com...]