homepage Welcome to WebmasterWorld Guest from 54.205.168.88
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
noxtrumbot/1.0 (crawler@noxtrum.com)
Another new crawler?
GaryK




msg:897920
 5:48 pm on May 29, 2005 (gmt 0)

It appeared to obey robots.txt. It took all the pages at a more than reasonable pace of about one page per minutes. But I'm always leery of crawlers that don't supply a web page for me to look at. Extrapolating their e-mail address into a URL brought me to page that required a login. There are almost no log file entries for it in a search on Google.

 

larryhatch




msg:897921
 3:44 am on May 31, 2005 (gmt 0)

Noxtrum just hit my whole site. Files are taken at precise 1 minute intervals, looks well behaved
so far. I never heard of them though. -Larry

larryhatch




msg:897922
 3:56 am on May 31, 2005 (gmt 0)

A little more info on Noxtrum: I got the same login popup, and noted "zona restringida" (restricted area)
which is in Spanish. Now, a G search for just noxtrum.com brings up about 4 pages.
Maybe 3 pages are log listings online (why do people do that?) and about half of those are also in Spanish.

"Noxtrum" doesn't appear in my Spanish-English dictionary, and closest English word is Nostrum. -Larry

GaryK




msg:897923
 6:16 am on May 31, 2005 (gmt 0)

Thanks Larry. Since you've confirmed they really do seem well behaved I'll send them an e-mail in Spanish to ask what their plans are and suggest that they add a public URL to their user agent.

larryhatch




msg:897924
 7:18 am on May 31, 2005 (gmt 0)

Garyk: Do you have any Ssspanish language pages? I have one or two. That might explain their visits. -Larry

GaryK




msg:897925
 3:38 pm on May 31, 2005 (gmt 0)

Yes, I have pages in English, Spanish, French and Russian. I'm still waiting for a reply to my e-mail to them.

WebWalla




msg:897926
 6:51 am on Jun 30, 2005 (gmt 0)

It's the new Spanish search engine [webmasterworld.com]

keyplyr




msg:897927
 5:42 am on Jul 11, 2005 (gmt 0)

This bot is relentless. Visited 3 or 4 times in the last week, each time getting almost evey webpage. 1 minute intervals yes, but unless my logs have missed it, noxtrumbot has not requested robots.txt yet.

larryhatch




msg:897928
 5:51 am on Jul 11, 2005 (gmt 0)

Anyone who reads this thread probably got crawled by Noxtrum.
At least its going to be an actual search engine, and not some phony
crawler the public can't use. Maybe they will send some traffic. -Larry

larryhatch




msg:897929
 5:59 am on Jul 11, 2005 (gmt 0)

I just checked 2 days of access_log files. Noxtrum DID read robots.txt
once yesterday, then pulled in my index.html page and every (2nd level) page linked from there.
It didn't dig any deeper into my content pages (3rd level).
Oddly, it showed a referral from another site to my index page along with the noxtrum stuff,
so I know how they found me. -Larry

keyplyr




msg:897930
 8:31 am on Jul 11, 2005 (gmt 0)


Yes, it did finally request robots.txt (4th day of crawling the same 300 webpages.) Guess my expirary settings are working like they're supposed to.

Still, I'm wondering why all the attention with my site though. Their model seems to be local interest and there's no Spanish content besides a few references to Spanish event locations on a couple pages - guess that's enough.

larryhatch




msg:897931
 8:57 am on Jul 11, 2005 (gmt 0)

Keyplyr: My guess is they just followed a link from some other site to yours. -Larry

keyplyr




msg:897932
 8:01 am on Jul 12, 2005 (gmt 0)

Ya think?

AcsCh




msg:897933
 1:21 pm on Jul 25, 2005 (gmt 0)

Looks like the spider from a new SE from Telefonica, the main spanish phone provider. Dont ever expect any answer from them.. ;)
[google.#*$!.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved