Automating a datafeed import

How do you tell your database to automatically update?

[zander213]

I am integrating an external datafeed that is changing daily. How do I make my MySQL database pull this fresh data daily and update all my pages with new content?

Any help would be greatly appreciated!

[httpwebwitch]

The best solution is always to grab data directly from the feed (like, from an API service). If you can't do that, you need to make a local copy. How often you update your database is up to you.

If it's a website, you can use a spider... It depends what the data is.

You can create a PHP script that does this:
grab data > parse data > insert/update SQL

then run that script as often as you feel it's necessary. But if the data is always changing, your copy will always be inaccurate to some degree.

The solution will depend on what data are you grabbing, and whether it needs to be guaranteed as accurate. weather? stock quotes? ticket prices? Gross National Debt? population of Zimbabwe?

[lazydog]

You'll need to write a script that pulls the feed and updates your database. And then have "CRON" run it as per your schedule.

Lookup crontab, cron, crond in the linux manual.

If you write a php script to do this, then you have two options
1. Use php-cli
2. Use wget

Hope that helps,

Saurabh.

[zander213]

Thanks for the great info - I am pretty new at this ;)

One quick addition here:

I have a url that contains export feeds in 2 formats (.tab, .csv, and .pipe)...

will I be able to grab a .tab and import that directly or do I need to save it, import it, and then pull from it?

Any good resources out there on doing something like this?

Thanks again in advance!