Search engine - keyword search mechanism

program to search for key words in multiple URLs

[wangdy]

Hello,
I wasn't sure how to phrase the question that I have since I am not sure that it would be considered a search engine issue...

I need to find out if there is a "search engine" that can go through numerous URLs and search for keywords? Would I have to write a program? Is there a software package? How difficult would it be to create?
Thanks!
Dennis

[WebGuerrilla]

Hi Dennis,

Welcome to WmW.

I'm afraid I don't quite understand the question. Where are the numerous URL's? Do you have a list on your computer that you want to search through, or are you looking to find URL's with specific keywords in a particular search engine database?

[wangdy]

Thanks for the welcome!

The former... we are looking to search for keywords in URLS or search engine databases. Basically, to find out what other sites on the web are using brand names.

Thanks!

[agerhart]

I think this may be able to help you. I found this in my log.

http://www.netcraft.com [netcraft.com]

It returns site from its database that have the search word or phrase in the URL.

-A Gerhart

[WebGuerrilla]

Yes, Netcraft is very handy. You can also perform URL only searches at most indidual search engines. The results aren't presented in as nice of a format as Netcraft, but you will usually find more URL's due to the larger size of the databases.

You can search at AltaVista for URL's containing specific words by typing url:keyword into the box.

At Google, you can use their advanced search page [google.com] to search for phrases contained only in URL's.

[wangdy]

Thanks!

Another novice question... are search engines easy to write? Or are there software packages out there?

It seems, after thinking about what I needed to do, that this project is very similar to building a search engine.... any suggestions?
Thanks again...

[WebGuerrilla]

There are several companies that make search software. AltaVista sells an Enterprise version that can be used to run a search engine. It is typically used by ecommerce sites and company intranets, but it has all the bells and whistles of AltaVista's site.

Inktomi also offers similar software.

A great place to visit for info on search software is [url=http//:www.searchtools.com]searchtools.com.[/url]

[wangdy]

OK... on a related topic,
Am I correct to think that a spider does not crawl a site unless it is submitted to a search engine (Yahoo, google, etc)?

[Woz]

wangdy,

back to your first question, you might want to consider setting up a Search Engine Monitoring system. There are two companies I know of that can do this for you:- http://www.trackengine.com/ and http://www.trackengine.com/ (the latter of which came to light in these fora) and there may be more.

These are free systems that will monitor specific searches at specific Search Engines for you and report the results via advertisement supported email.

So you could setup these systems to monitors searches at SEs for your particular product/brand name and be notified if anything new comes up in the searche results.

To your second question,

Some search engines will only spider you if submitted, some follow found links. For some of the Engines, Google and Inktomi for instance, it is better if they FIND you rather than be submitted.

Directories on the other hand, such as Yahoo NBCI and the likes, are human operated and so will only look at your site after you submit it. There are exceptions of course but in the makors exceptions are in the minority.

So a course of well prepared and well targetted submissions is best to gain maximum results and avoind unintentional pitfalls. And the best way to get Well Prepared is to read as much of these fora as you can before starting.

Onya
Woz

[asuprun]

Hi WoZ,

You mentioned free tools that send emails when any specific SE results page is updated. Can you give any link?

Thanks a lot!

Asuprun.

[wangdy]

Thanks Woz,

That's some good info there... I've also been diving into what some companies call the "Deep Web"... the information that is served up in databases.

I first read about these in a magazine article in the Standard. It's really neat what they can find out there...