Welcome to WebmasterWorld Guest from 54.196.175.173

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

noxtrumbot/1.0 (crawler@noxtrum.com)

Another new crawler?

     

GaryK

5:48 pm on May 29, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It appeared to obey robots.txt. It took all the pages at a more than reasonable pace of about one page per minutes. But I'm always leery of crawlers that don't supply a web page for me to look at. Extrapolating their e-mail address into a URL brought me to page that required a login. There are almost no log file entries for it in a search on Google.

larryhatch

3:44 am on May 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Noxtrum just hit my whole site. Files are taken at precise 1 minute intervals, looks well behaved
so far. I never heard of them though. -Larry

larryhatch

3:56 am on May 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A little more info on Noxtrum: I got the same login popup, and noted "zona restringida" (restricted area)
which is in Spanish. Now, a G search for just noxtrum.com brings up about 4 pages.
Maybe 3 pages are log listings online (why do people do that?) and about half of those are also in Spanish.

"Noxtrum" doesn't appear in my Spanish-English dictionary, and closest English word is Nostrum. -Larry

GaryK

6:16 am on May 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Larry. Since you've confirmed they really do seem well behaved I'll send them an e-mail in Spanish to ask what their plans are and suggest that they add a public URL to their user agent.

larryhatch

7:18 am on May 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Garyk: Do you have any Ssspanish language pages? I have one or two. That might explain their visits. -Larry

GaryK

3:38 pm on May 31, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, I have pages in English, Spanish, French and Russian. I'm still waiting for a reply to my e-mail to them.

WebWalla

6:51 am on Jun 30, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It's the new Spanish search engine [webmasterworld.com]

keyplyr

5:42 am on Jul 11, 2005 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



This bot is relentless. Visited 3 or 4 times in the last week, each time getting almost evey webpage. 1 minute intervals yes, but unless my logs have missed it, noxtrumbot has not requested robots.txt yet.

larryhatch

5:51 am on Jul 11, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Anyone who reads this thread probably got crawled by Noxtrum.
At least its going to be an actual search engine, and not some phony
crawler the public can't use. Maybe they will send some traffic. -Larry

larryhatch

5:59 am on Jul 11, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just checked 2 days of access_log files. Noxtrum DID read robots.txt
once yesterday, then pulled in my index.html page and every (2nd level) page linked from there.
It didn't dig any deeper into my content pages (3rd level).
Oddly, it showed a referral from another site to my index page along with the noxtrum stuff,
so I know how they found me. -Larry

keyplyr

8:31 am on Jul 11, 2005 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month




Yes, it did finally request robots.txt (4th day of crawling the same 300 webpages.) Guess my expirary settings are working like they're supposed to.

Still, I'm wondering why all the attention with my site though. Their model seems to be local interest and there's no Spanish content besides a few references to Spanish event locations on a couple pages - guess that's enough.

larryhatch

8:57 am on Jul 11, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Keyplyr: My guess is they just followed a link from some other site to yours. -Larry

keyplyr

8:01 am on Jul 12, 2005 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Ya think?

AcsCh

1:21 pm on Jul 25, 2005 (gmt 0)

10+ Year Member



Looks like the spider from a new SE from Telefonica, the main spanish phone provider. Dont ever expect any answer from them.. ;)
[google.#*$!.com...]
 

Featured Threads

Hot Threads This Week

Hot Threads This Month