Is slurp friendly?

doesn't follow 301's and ignores robots.txt

[wingslevel]

I've read about others having problems here.

Last week I disallowed slurp from a good portion of my site. It very politely checks my robots, then goes right where its not supposed to. my robots is formatted correctly.

I know that slurp doesn't get 301's, but will it eventually heed my robots? Maybe it caches robots data and hasn't refreshed yet?

[jdMorgan]

Slurp is very slow about everything.

There is a big time lag before it responds to robots.txt, and it continues to ask for 404-Not Found or 410-Gone pages for months or even years after they are removed from a site.

However, once it figures out your robots.txt, it seems to honor it -- At least, I haven't had any problems recently in that respect.

Jim

[wilderness]

Slurp attempted to read a folder today which has been disallowed in my robots for over two years.

For some reason their bot just goes haywire some times.