Data/site search script

script for searching online database

[sam247]

I am looking for a script that I can use to search an online database and return data to me for inclusion in either a database or in an excel format.

Specifically, I am attempting to extract data from a public site that allows people to search for public data. For example, if I search for all 'widgets' in a particular state (i.e. California), I get back a .asp page that contains multiple records. Some of the data I need is contained in a clickable link on each record. I would like to be able to grab the data from all of the records in the initial page and also the data available from the clickable links. There are over 100,000 records so it would be almost impossible to do without a script.

Anybody have any thoughts or ideas on a product that is available to do this? Thanks.

[eelixduppy]

Welcome to WebmasterWorld!

We have a nice thread in our library on how to create a search in MySQL. Here's the thread: [webmasterworld.com...]

This is obviously vary if you are talking about different db technology, but the concepts are the same. Shouldn't be too hard to implement.

[sam247]

Thanks for the quick response and the welcome. I looked at the link to the topic you sent. Great info. I will be searching a public site so I won't be able to connect to the database. So I guess that I would need to query in the URL?

[eelixduppy]

>>that I can use to search an online database

Sorry, I misunderstood what you originally had access to. Yes, you are correct; you'd have to add the appropriate GET or POST information to the request to that page to get the information that you want. From here, you are going to have to sort the data depending on how their script actually returns the data to the browser.