Who is Smith and Douglas 2.0

from 66.227.29.211

[Staffa]

Had three visits in the last week.
Grabs about a third of my pages including pages linked to form action=
and located in disallowed directories in robots.txt.

And of course ignores the robots.txt file.

Any ideas? Thanks

[edit_g]

The IP 66.227.29.211 resolves to a broadband ISP in Northern Colorado.

[Staffa]

Thanks edit_g, I know I did a whois already.

I just don't understand why an ISP would crawl part of my site and in an uncivilized fashion at that.

Any other ideas? Much appreciated.

[edit_g]

Could be some sort of testing? Hard to know really.

Send them an email and ask them, and remind them what robots.txt is for while you're at it.

[Staffa]

Thanks again, good idea I'll do just that.

[engine]

Might not be the ISP - likely to be one of their customers testing something or perhaps harvesting.