| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe and Support WebmasterWorld|
|Slurp/cat and Slurp/si|
What's the difference?
| 3:01 pm on Jul 11, 2003 (gmt 0)|
Noticed both crawling my site yesterday.
| 4:34 pm on Jul 11, 2003 (gmt 0)|
Slurp/si just reads robots.txt, and Slurp/cat does everything else.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2014 all rights reserved