New Full Spidering Search Engine - Giga Blast

GigaBlast.com

[Brett_Tabke]

I'd heard about this one awhile back and it appears they are up and running:

[gigablast.com...]

Real time spidering on the AddUrl page.

[gigablast.com...]

Feels like: Infoseek
Tastes like: Peppermint (minty fresh ;)
Looks like: squeaky clean.
Relavance like: who cares. it's a new se with instant addurl - done.

[mattdwells]

it will spider dynamic urls but i currently have that option disabled. i will probably enable it shortly.
matt

[papabaer]

I am seeing some good returns on various searches. In fact, I am suprised at the depth I see. This one will be interesting to watch... I'm gonna adopt it as a pet and see if it's housebroken! ;)

[Ove]

Thanks for the info Matt
this will be interessted to follow.

/Ove

[papabaer]

Hey Matt! How about "Here Giggy, Giggy!" Wait a second... Giggy? Yeah!

[Key_Master]

More power to you Matt. I've been testing your engine for a few weeks now and I must say I am impressed. In many ways I found it to be more relevent than Google (before the attention you recently started receiving). I know you'll get all that fixed in due time so it doesn't change my opinion any.

I have a few questions:

1. Have you figured out a user agent for your spider and if so, what will it be?

2. Robots.txt- will your spider obey it?

3. (Just for my amusement) What is your policy on cloaking?

Key_Master

[john316]

Hi Matt!

Sorry for jumping to conclusions!

Unfortunately on the web..I've used the "if it looks like a rat and smells like a rat" acid test for some time.

Obviously I was mistaken, please accept my apology.

I wish you the very best!

[holographic]

It would be nice to see a SE publish exactly where they stand with what they do and don't accept.

[Brett_Tabke]

Thanks Matt, good luck with it!

[papabaer]

Seriously Matt, it appears you are off to a nice start. The traffic will come...

Best of luck! And please, keep us informed - we love this stuff!

[mattdwells]

1. Have you figured out a user agent for your spider and if so, what will it be?

it will be gigabot.

2. Robots.txt- will your spider obey it?
yes, except top level pages.
it also obeys the meta disallow tags, too.

3. (Just for my amusement) What is your policy on cloaking?
if it's used to abuse the search engine and mislead searchers, then my policy is to ban that site. I index the meta keyword/description tags so use those instead.

[msr986]

I hope everyone realizes that the "last 5 queries" feature is exposing everyone's top keyword targets!

Be careful searching unless you want everyone at WmW to know what your pet keywords are!

[Key_Master]

>>>it also obeys the meta disallow tags, too.

Is this true for top level pages also?

[mattdwells]

>>>it also obeys the meta disallow tags, too.
>>Is this true for top level pages also?

no, i always index top level pages regardless of robots.txt or meta disallow tags.

[mattdwells]

NOTE: i've just added the option to set the number of summary lines in your search results on my advanced search page.

do you guys think this would be useful? or just confusing to users?

matt

[Alby]

Wow!

I added one of my urls and 3-4 seconds later it was indexed and ranked number one under targeted kw's! :-) One of my other sites were already indexed.

This is going to be a good one...

Alby

[jimbo_mac]

has anyone tried refreshing the page
indexes 200 to 350 pages per refresh. gigabot spider is busy.

[brotherhood of LAN]

The last 5 queries thing has turned into a forum-like tool

Such is the diversity of the web :)

[paynt]

Welcome to Webmaster World Matt.

Thanks for the thrill. I agree, it reminds me of the old Infoseek rush. Those were the days.

If you wanted to stir up a bunch of web'sters, you've certainly done that. Good luck on your venture.

[jeremy goodrich]

nicely done...good luck! (off to try and remember how to auto submit he he he)

[wharsono]

Nice, ... Matt will you make more little size your logo, its hard for me in Indonesia

[Napoleon]

Welcome Matt. A very nice system you have there as well. It deserves success.

[brotherhood of LAN]

Was playing around with the last 5 queries thing on the home page

I searched for

place link <here> for googlebot

and it sorta went a bit off track. Check it out

[gigablast.com...]

[wharsono]

I was test by search "java furniture", but i think java furniture is nothing relevant with sun java.

[bobriggs]

Ok, I'm lost. I admit it. A lot of unanswered questions.

First, littleman's UA and ip - I can't find it anywhere in the last week.

Compare this to the link in Brett's first post. Who is 'Guest from 208.254.87.133'? if not the spider and if that's really a cached page.

I can't tell if these pages are cached or not. I submitted a site about 8:00 am this morning, and the cached page shows an update 2 1/2 hours later. No sign of any spider in my logs, AT ALL.

I submitted 5 sites, they're all in, all saying today's date as having been spidered, but NO evidence that anything actually came around? The only spider that's been around lately is whizbang labs.

I'm glad the sites are in, and the SE looks promising. If I look at the data I have, it looks as if the sites were pre-spidered, and the cache (mirror) is showing the current page. I guess I could change a page and look at the cached copy to verify this. Has anybody seen evidence of a crawler? Or or my logs wrong?

---

continued [webmasterworld.com...]