mack
msg:4245768 | 2:48 am on Dec 24, 2010 (gmt 0) |
The sub domain will be viewed by a bot as a different site. It will therefore read the robots.txt file located at subdmain.example.com/robots.txt
If you disallow robots there, the sub domain wont be crawled.
Mack.
|
virtualreality
msg:4245791 | 5:02 am on Dec 24, 2010 (gmt 0) |
Thank you, Mack.
|
Joshb
msg:4248312 | 4:16 pm on Jan 2, 2011 (gmt 0) |
You could also twick the robots.txt file to control which sub-domain will be scanned. could be a powerful tool to 'trick' crawlers "last-scan" timers.
|
|
