I have disallowed a new directory for a few days while I populated it with files. Is there a problem later with getting these files indexed after removing the Disallow?
Lord Majestic
3:55 am on Nov 3, 2005 (gmt 0)
It may well be - some search engines cache robots.txt for some time, so its possible they could come at the wrong time and your temporary change will be longer than expected.
bumpaw
4:07 am on Nov 3, 2005 (gmt 0)
I think I'm seeing a little of that and will probably discontinue the practice.
bumpaw
1:45 pm on Nov 3, 2005 (gmt 0)
Could this be a problem? Disallow: /duck/pig/
Am I disallowing other files in /duck/? The idea is to disallow only files in /pig/
Lord Majestic
1:50 pm on Nov 3, 2005 (gmt 0)
Am I disallowing other files in /duck/?
No - the path you use in disallow statement will disallow any URL that starts with that value.