homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

noxtrumbot/1.0 (crawler@noxtrum.com)
Another new crawler?

 5:48 pm on May 29, 2005 (gmt 0)

It appeared to obey robots.txt. It took all the pages at a more than reasonable pace of about one page per minutes. But I'm always leery of crawlers that don't supply a web page for me to look at. Extrapolating their e-mail address into a URL brought me to page that required a login. There are almost no log file entries for it in a search on Google.



 3:44 am on May 31, 2005 (gmt 0)

Noxtrum just hit my whole site. Files are taken at precise 1 minute intervals, looks well behaved
so far. I never heard of them though. -Larry


 3:56 am on May 31, 2005 (gmt 0)

A little more info on Noxtrum: I got the same login popup, and noted "zona restringida" (restricted area)
which is in Spanish. Now, a G search for just noxtrum.com brings up about 4 pages.
Maybe 3 pages are log listings online (why do people do that?) and about half of those are also in Spanish.

"Noxtrum" doesn't appear in my Spanish-English dictionary, and closest English word is Nostrum. -Larry


 6:16 am on May 31, 2005 (gmt 0)

Thanks Larry. Since you've confirmed they really do seem well behaved I'll send them an e-mail in Spanish to ask what their plans are and suggest that they add a public URL to their user agent.


 7:18 am on May 31, 2005 (gmt 0)

Garyk: Do you have any Ssspanish language pages? I have one or two. That might explain their visits. -Larry


 3:38 pm on May 31, 2005 (gmt 0)

Yes, I have pages in English, Spanish, French and Russian. I'm still waiting for a reply to my e-mail to them.


 6:51 am on Jun 30, 2005 (gmt 0)

It's the new Spanish search engine [webmasterworld.com]


 5:42 am on Jul 11, 2005 (gmt 0)

This bot is relentless. Visited 3 or 4 times in the last week, each time getting almost evey webpage. 1 minute intervals yes, but unless my logs have missed it, noxtrumbot has not requested robots.txt yet.


 5:51 am on Jul 11, 2005 (gmt 0)

Anyone who reads this thread probably got crawled by Noxtrum.
At least its going to be an actual search engine, and not some phony
crawler the public can't use. Maybe they will send some traffic. -Larry


 5:59 am on Jul 11, 2005 (gmt 0)

I just checked 2 days of access_log files. Noxtrum DID read robots.txt
once yesterday, then pulled in my index.html page and every (2nd level) page linked from there.
It didn't dig any deeper into my content pages (3rd level).
Oddly, it showed a referral from another site to my index page along with the noxtrum stuff,
so I know how they found me. -Larry


 8:31 am on Jul 11, 2005 (gmt 0)

Yes, it did finally request robots.txt (4th day of crawling the same 300 webpages.) Guess my expirary settings are working like they're supposed to.

Still, I'm wondering why all the attention with my site though. Their model seems to be local interest and there's no Spanish content besides a few references to Spanish event locations on a couple pages - guess that's enough.


 8:57 am on Jul 11, 2005 (gmt 0)

Keyplyr: My guess is they just followed a link from some other site to yours. -Larry


 8:01 am on Jul 12, 2005 (gmt 0)

Ya think?


 1:21 pm on Jul 25, 2005 (gmt 0)

Looks like the spider from a new SE from Telefonica, the main spanish phone provider. Dont ever expect any answer from them.. ;)

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved