Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
I need to find out if there is a "search engine" that can go through numerous URLs and search for keywords? Would I have to write a program? Is there a software package? How difficult would it be to create?
Welcome to WmW.
I'm afraid I don't quite understand the question. Where are the numerous URL's? Do you have a list on your computer that you want to search through, or are you looking to find URL's with specific keywords in a particular search engine database?
You can search at AltaVista for URL's containing specific words by typing url:keyword into the box.
At Google, you can use their advanced search page [google.com] to search for phrases contained only in URL's.
Another novice question... are search engines easy to write? Or are there software packages out there?
It seems, after thinking about what I needed to do, that this project is very similar to building a search engine.... any suggestions?
Inktomi also offers similar software.
A great place to visit for info on search software is [url=http//:www.searchtools.com]searchtools.com.[/url]
back to your first question, you might want to consider setting up a Search Engine Monitoring system. There are two companies I know of that can do this for you:- http://www.trackengine.com/ and http://www.trackengine.com/ (the latter of which came to light in these fora) and there may be more.
These are free systems that will monitor specific searches at specific Search Engines for you and report the results via advertisement supported email.
So you could setup these systems to monitors searches at SEs for your particular product/brand name and be notified if anything new comes up in the searche results.
To your second question,
Some search engines will only spider you if submitted, some follow found links. For some of the Engines, Google and Inktomi for instance, it is better if they FIND you rather than be submitted.
Directories on the other hand, such as Yahoo NBCI and the likes, are human operated and so will only look at your site after you submit it. There are exceptions of course but in the makors exceptions are in the minority.
So a course of well prepared and well targetted submissions is best to gain maximum results and avoind unintentional pitfalls. And the best way to get Well Prepared is to read as much of these fora as you can before starting.
That's some good info there... I've also been diving into what some companies call the "Deep Web"... the information that is served up in databases.
I first read about these in a magazine article in the Standard. It's really neat what they can find out there...