Forum Moderators: goodroi

Message Too Old, No Replies

Do bots confuse .htm and .html files in robots.txt?

         

subgenius

4:29 pm on Feb 16, 2006 (gmt 0)

10+ Year Member



I was recently updating a client's robots.txt file after redesigning their site, and most of the pages I was excluding were .htm files because the original designer used FrontPage. When I create sites, I want the 'L' to receive the credit it deserves, so I tend to use the .html extension.

Several of the new pages share the same name as the old pages, so the only difference is the 'L'. Since I'm trying to block the old .htm files from being crawled, is there any chance that the bots will get lazy and ban the .html files as well?

sonjay

6:08 pm on Feb 16, 2006 (gmt 0)

10+ Year Member



Rather than disallowing the files in robots.txt, why don't you just nuke the files and put in 301 redirects from the old versions to the new?