Forum Moderators: goodroi

Message Too Old, No Replies

SygolBot, from sygol.com: anyone seen this before?

just visited my site, and did NOT respect robots.txt

         

stapel

3:28 pm on Oct 2, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does anybody have any experience with the SygolBot "search engine" spider? I did a search, and couldn't find any reviews.

Thank you.

Eliz.

DanA

4:23 pm on Oct 2, 2005 (gmt 0)

10+ Year Member



It comes from an Italian search engine.
It doesn't request robots.txt.
It often uses HEAD requests.

stapel

12:37 am on Oct 3, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



So it's not necessarily malicious, just stupid?

Eliz.