| changing robots.txt
replacing robots.txt |
darex
msg:1529197 | 8:02 pm on Dec 9, 2003 (gmt 0) |
If you put up a robots.txt with a disallow in it and then you want to reallow spiders - Is it sufficient to simply delete the robots.txt file or do you need to put up a new robots.txt file which explicitly allows all spiders everywhere (E.g. User-agent: *
Disallow: Ie is there some latency in the way spiders remember the robots.txt even after it has been deleted - if you don't specifically countermand the robots.txt with a new version? (in particular I am interested in google)
|
jbinbpt
msg:1529198 | 8:24 pm on Dec 9, 2003 (gmt 0) |
Safe thing is to replace it. jb
|
|
|
