Forum Moderators: open

Message Too Old, No Replies

GingerCrawler/1.0

         

Mokita

3:13 am on Jul 10, 2009 (gmt 0)

10+ Year Member



Not seen this in my logs previously:

User Agent: "GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)"

Came from a Savvis IP.

Robots.txt: Yes
Obeyed it: Yes

GaryK

5:36 am on Jul 10, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I tend not to post stuff that reads/obeys robots.txt.

First saw this one on May 4, 2009. Last seen June 21, 2009. It's taken 632 files from an author's site I host. Obeyed robots.txt each time.

Mokita

5:07 am on Jul 13, 2009 (gmt 0)

10+ Year Member



GaryK wrote:
I tend not to post stuff that reads/obeys robots.txt.

To each his own. This forum is called "Spider Identification" not "Disobedient Spider Identification".

GaryK

2:39 pm on Jul 13, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Please don't think I was taking a swipe at you. I was just pointing out why I hadn't previously mentioned this bot and many others. If I posted every bot I see I'd be making dozens of posts every week. If that's a good thing let me know and I'll be happy to do it.

enigma1

10:50 am on Jul 14, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes I also noticed that in my logs. The A records of that ip point to goddady and so it got kicked. I generally block spiders who come from known hosting providers as I expect them to have their own.

JAB Creations

1:30 am on Jul 17, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



enigma1, check again, looks like they amended it. :)

- John

enigma1

1:03 pm on Jul 17, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



John, for one ip I checked the A records point to domaincontrol dot com.

Pfui

7:22 pm on Jul 31, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It's behaved until now. Not today. Took robots.txt -- where it gets a blanket Disallow: / -- then tried to grab an .html page. Tsk,tsk.

crawler6.gingersoftware.com
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)

Pfui

6:59 pm on Aug 11, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



One by one, all are assimilated...

ec2-[yada-yada].compute-1.amazonaws.com
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)

robots.txt? YES

See also: amazonaws.com plays host to wide variety of bad bots [webmasterworld.com]

frontpage

12:10 am on Aug 22, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We added this resources consuming bot to our Mod-security 2.x list.

SecRule HTTP_User-Agent "GingerCrawler"

incrediBILL

12:24 am on Aug 23, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If I posted every bot I see I'd be making dozens of posts every week. If that's a good thing let me know and I'll be happy to do it.

I have to agree with Mokita here, the forum is called "Spider Identification" not "Disobedient Spider Identification" and the more information we share for all spiders the better off we are.

Personally, I rarely look at them anymore because whitelisting bots set me free.

Obedient or Disobedient, they all get bounced, out of site, out of mind ;)

GaryK

4:18 pm on Aug 23, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I already said I'll be posting all the new bots I see and I've been doing that. :)

Have you found a way to post my list of known bots yet without taking WebmasterWorld software down in a ball of flames? ;)