Welcome to WebmasterWorld Guest from 54.163.142.67

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

noxtrumbot/1.0 (crawler@noxtrum.com)

Another new crawler?

     
5:48 pm on May 29, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


It appeared to obey robots.txt. It took all the pages at a more than reasonable pace of about one page per minutes. But I'm always leery of crawlers that don't supply a web page for me to look at. Extrapolating their e-mail address into a URL brought me to page that required a login. There are almost no log file entries for it in a search on Google.
3:44 am on May 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


Noxtrum just hit my whole site. Files are taken at precise 1 minute intervals, looks well behaved
so far. I never heard of them though. -Larry
3:56 am on May 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


A little more info on Noxtrum: I got the same login popup, and noted "zona restringida" (restricted area)
which is in Spanish. Now, a G search for just noxtrum.com brings up about 4 pages.
Maybe 3 pages are log listings online (why do people do that?) and about half of those are also in Spanish.

"Noxtrum" doesn't appear in my Spanish-English dictionary, and closest English word is Nostrum. -Larry

6:16 am on May 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Thanks Larry. Since you've confirmed they really do seem well behaved I'll send them an e-mail in Spanish to ask what their plans are and suggest that they add a public URL to their user agent.
7:18 am on May 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


Garyk: Do you have any Ssspanish language pages? I have one or two. That might explain their visits. -Larry
3:38 pm on May 31, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Yes, I have pages in English, Spanish, French and Russian. I'm still waiting for a reply to my e-mail to them.
6:51 am on June 30, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2002
posts:741
votes: 0


It's the new Spanish search engine [webmasterworld.com]
5:42 am on July 11, 2005 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6986
votes: 167


This bot is relentless. Visited 3 or 4 times in the last week, each time getting almost evey webpage. 1 minute intervals yes, but unless my logs have missed it, noxtrumbot has not requested robots.txt yet.

5:51 am on July 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


Anyone who reads this thread probably got crawled by Noxtrum.
At least its going to be an actual search engine, and not some phony
crawler the public can't use. Maybe they will send some traffic. -Larry
5:59 am on July 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


I just checked 2 days of access_log files. Noxtrum DID read robots.txt
once yesterday, then pulled in my index.html page and every (2nd level) page linked from there.
It didn't dig any deeper into my content pages (3rd level).
Oddly, it showed a referral from another site to my index page along with the noxtrum stuff,
so I know how they found me. -Larry
8:31 am on July 11, 2005 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6986
votes: 167



Yes, it did finally request robots.txt (4th day of crawling the same 300 webpages.) Guess my expirary settings are working like they're supposed to.

Still, I'm wondering why all the attention with my site though. Their model seems to be local interest and there's no Spanish content besides a few references to Spanish event locations on a couple pages - guess that's enough.

8:57 am on July 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


Keyplyr: My guess is they just followed a link from some other site to yours. -Larry
8:01 am on July 12, 2005 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6986
votes: 167


Ya think?

1:21 pm on July 25, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 11, 2003
posts:71
votes: 0


Looks like the spider from a new SE from Telefonica, the main spanish phone provider. Dont ever expect any answer from them.. ;)
[google.#*$!.com...]