Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
I need to find out if there is a "search engine" that can go through numerous URLs and search for keywords? Would I have to write a program? Is there a software package? How difficult would it be to create?
You can search at AltaVista for URL's containing specific words by typing url:keyword into the box.
At Google, you can use their advanced search page [google.com] to search for phrases contained only in URL's.
Inktomi also offers similar software.
A great place to visit for info on search software is [url=http//:www.searchtools.com]searchtools.com.[/url]
back to your first question, you might want to consider setting up a Search Engine Monitoring system. There are two companies I know of that can do this for you:- http://www.trackengine.com/ and http://www.trackengine.com/ (the latter of which came to light in these fora) and there may be more.
These are free systems that will monitor specific searches at specific Search Engines for you and report the results via advertisement supported email.
So you could setup these systems to monitors searches at SEs for your particular product/brand name and be notified if anything new comes up in the searche results.
To your second question,
Some search engines will only spider you if submitted, some follow found links. For some of the Engines, Google and Inktomi for instance, it is better if they FIND you rather than be submitted.
Directories on the other hand, such as Yahoo NBCI and the likes, are human operated and so will only look at your site after you submit it. There are exceptions of course but in the makors exceptions are in the minority.
So a course of well prepared and well targetted submissions is best to gain maximum results and avoind unintentional pitfalls. And the best way to get Well Prepared is to read as much of these fora as you can before starting.