| 8:15 pm on Jun 28, 2006 (gmt 0)|
This should work:
| 9:00 pm on Jun 28, 2006 (gmt 0)|
You have that backwards. All bots, including Slurp, msnbot and Googlebot will read and obey the first Disallow.
Slurp and Google bot will go on their merry ways when they see the first directive, happily gobbling down pages. Others will see the second and be off to some other bloke's site.
| 9:11 pm on Jun 28, 2006 (gmt 0)|
I never understood this. All I see are 'disallows'.
First 2 robots specifically, then a wild-card for the rest.
Why doesn't this disallow all of them? -Larry
| 9:16 pm on Jun 28, 2006 (gmt 0)|
What you're missing is the effect of what
comes after the :
(the above line says disallow nothing on site.)
(the above line says disallow everything on site.)
| 9:17 am on Jun 29, 2006 (gmt 0)|
Thanks KevinP: That makes perfect sense and explains everything. -Larry
| 12:30 pm on Jun 29, 2006 (gmt 0)|
Thanks guys, I will try that. It seems so simple, but I am always afraid to do something which will discourage googlebot. I didn't realize that the SE bots read robots.txt that way (sequentially I mean). Very cool, thanks again.