Forum Moderators: open

Message Too Old, No Replies

Bigsearch.ca

new Canadian search engine currently Beta

         

Mokita

5:47 pm on Dec 13, 2006 (gmt 0)

10+ Year Member



Full UA:
Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; [bigsearch.ca...] info@enhancededge.com)

Only asked for robots.txt, which is fortunate, as I have all Nutch bots barred. ;)

From their About Us: "Bigsearch.ca is a search engine seeded from the dmoz.org directory. Bigsearch.ca uses open source technologies to create an open and fair index."

volatilegx

1:52 am on Dec 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Saw it coming from 72.0.207.162

Mokita

2:58 am on Dec 16, 2006 (gmt 0)

10+ Year Member



The bot has visited one of our sites four times (so far) in December, always from 72.0.207.162. Each time it has asked for robots.txt and default page, but it always gets a 403 for the default page as I have Nutch agents barred.

I have now tried disallowing it in robots.txt using "Bigsearch.ca" - this is just a wild guess, as their web site gives no clue if it obeys robots.txt or what User Agent might be required. I'll see what happens if they return.