Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

To robots.txt or not to robots.txt - Moved site, removed pharma urls

         

cmendla

3:18 pm on Dec 16, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




Here it the situation.

    Old site was hacked w/ generated urls

    I removed the urls from google via wmt. I added those urls as disallows in robots.txt

    Site has now been moved as a plain html site to another server



Question - should the robots.txt file on the new server have disallows for the urls that were generated on the old server given that it would be ALMOST impossible to have those urls generated on the new server.

There are probably still links out there pointing to the old generate URLS although I went through the disavow process using tactical nukess.

thanks

chris

lucy24

5:24 pm on Dec 16, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I added those urls as disallows in robots.txt

I'm seriously missing something here. Why do you want the search engine to think that nonexistent URLs are present on your site? You've blocked its only way of finding out that they're not really there.

cmendla

6:05 pm on Dec 16, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Lucy - Thanks.. It makes sense now.