GingerCrawler/1.0

[Mokita]

Not seen this in my logs previously:

User Agent: "GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)"

Came from a Savvis IP.

Robots.txt: Yes
Obeyed it: Yes

[GaryK]

I tend not to post stuff that reads/obeys robots.txt.

First saw this one on May 4, 2009. Last seen June 21, 2009. It's taken 632 files from an author's site I host. Obeyed robots.txt each time.

[Mokita]

GaryK wrote:

I tend not to post stuff that reads/obeys robots.txt.

To each his own. This forum is called "Spider Identification" not "Disobedient Spider Identification".

[GaryK]

Please don't think I was taking a swipe at you. I was just pointing out why I hadn't previously mentioned this bot and many others. If I posted every bot I see I'd be making dozens of posts every week. If that's a good thing let me know and I'll be happy to do it.

[enigma1]

Yes I also noticed that in my logs. The A records of that ip point to goddady and so it got kicked. I generally block spiders who come from known hosting providers as I expect them to have their own.

[JAB Creations]

enigma1, check again, looks like they amended it. :)

- John

[enigma1]

John, for one ip I checked the A records point to domaincontrol dot com.

[Pfui]

It's behaved until now. Not today. Took robots.txt -- where it gets a blanket Disallow: / -- then tried to grab an .html page. Tsk,tsk.

crawler6.gingersoftware.com
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)

[Pfui]

One by one, all are assimilated...

ec2-[yada-yada].compute-1.amazonaws.com
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)

robots.txt? YES

See also: amazonaws.com plays host to wide variety of bad bots [webmasterworld.com]

[frontpage]

We added this resources consuming bot to our Mod-security 2.x list.

SecRule HTTP_User-Agent "GingerCrawler"

[incrediBILL]

If I posted every bot I see I'd be making dozens of posts every week. If that's a good thing let me know and I'll be happy to do it.

I have to agree with Mokita here, the forum is called "Spider Identification" not "Disobedient Spider Identification" and the more information we share for all spiders the better off we are.

Personally, I rarely look at them anymore because whitelisting bots set me free.

Obedient or Disobedient, they all get bounced, out of site, out of mind ;)

[GaryK]

I already said I'll be posting all the new bots I see and I've been doing that. :)

Have you found a way to post my list of known bots yet without taking WebmasterWorld software down in a ball of flames? ;)