Forum Moderators: open
Travelbot came from 69.13.105.33 and Jetbot from 64.71.144.*
Any idea whether these bots are good/bad?
I first saw it on Saturday. It is grabbing robots.txt, and seems to obey it as well. Crawling is steady, but not aggressive. It likes to work its way through a directory before moving to another directory. (That is, it grabs pages only in the root dir, then moves to /DirA and grabs, then to /DirB...)
That's my limited experience with this bot. I have already banned it, based on personal criteria. Where TravelBot falls in the range of "good" to "bad," I don't know. In my eyes, it's leaning hard to "bad," given that it does not identify who is operating it, offers no contact information and the "website" of the ISP it is running from is nothing but an AUP for a homepage.
I'll note that although the term "travel" is in the name of the bot, this is not a travel website-centric crawler.
Searches for more info have turned up primarily (seemingly?) irrelevant fluff, but there were some academic references regarding software agents. Searches for "TravelBot software agent" will turn up the about half-dozen references I speak of.
volatilegx points out a good thread on JetBot. Too bad I never received a final reply, but there should be enough info in that thread for you to decide whether JetBot is "good" or "bad."
I'll note that although the term "travel" is in the name of the bot, this is not a travel website-centric crawler.
But interestingly my site is about travel photography, so a "travelbot" would be right on target. I thought initially that this bot targeted travel sites.
Coincidence?
TravelBot popped up in the logs for 5 sites I manage (this weekend), none of them travel related, leading me to the conclusion that the bot doesn't target travel websites.
I suspect it is a semantic issue: "travel" as in roaming/exploring/surfing/crawling the web, rather than vacation/trip/excursion. (Y!/Ink has never "Slurped" down my sites, but they do take steady, gentle "sips.")
