Tutorial Crawler 1.4

TutorGig's Tutorial Crawler

[pendanticist]

New one for me.

216.40.225.** - - [21/Dec/2003:19:48:20 -0800] "GET /robots.txt HTTP/1.1" 200 1524 "-" "Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)"

Seems well enough behaved as it has only taken HTML pages and no images while holding within the 'no fetch more than one document each ten seconds' limits set on the page above.

I should also mention there was little out there in the form of search results other than previous visits to other sites.

Pendanticist.

[mack]

Yep had this one in my logs also over the past few days.

It has been well behaved on my site also. It obeyed robots.txt

They seam to only be looking to index tutorials. Not quite sure how they can index only tutorials when they seam to index all html docs?

Dont think I will ban it though.

Mack.

[pendanticist]

Hey, mack? They hitting your files evenly?

The logfiles from overnight show the return, but the crawl stops too early and at the same place it did last time.

It gets to the end of my Aboriginal Peoples section, then peels off without visiting the remaining 160+ indices, all of which are HTML...got no tutorials.

Other than what you mentioned, do you see it looking for anything specific at your place?

Pendanticist.

[mack]

It arrived on Saturday morning and took robots then index. It then started on a section called guides. (could in a round about way be classed as a tutorial)

I am wondering if it only index sections that have a keyword that interests them?

Possible I guess.

Mack.

[onlineleben]

It seems that they are only indexing sites having tutorials, guides, learning material on their site. Checked out their site some time ago, when I started to get visits from them.
IMHO visitors from tutorgig are very targeted

[pendanticist]

Seems like what I'm reading might just be the case as I do have an academic site.

<Still don't have any Tutorials tho...>

[marcs]

That bot has been active on one of my sites for some time. That site does have tutorials on it.

[GaryK]

I first saw it in my logs on 7/20/2003. There are tutorials on the site it crawled. So far it's been very respectful of robots.txt and very well behaved in general. I list it as a "General Crawler" in my browscap.ini file.