Forum Moderators: coopster

Message Too Old, No Replies

Spider a site to update data, stock levels, prices

spider stock levels

         

kingkahn

8:54 am on Dec 12, 2007 (gmt 0)

10+ Year Member



How would I go about doing this? Is there a software package available, or a script?

Stock level, Item Number, Price compare, - has it changed!

So it would spider my site and index item numbers of products!, or I could add all my item numbers to a database.

Then it would spider the wholesalers site and compare the item numbers, to see if the Price, or the stock level has changed since the last spider.

Thanks.

creative craig

9:52 am on Dec 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not sure if a wholesaler would like you to spider there site in this way?!

If you are working closely with them surely they could provide you with the relevant information via a spreadsheet?

coopster

3:47 pm on Dec 12, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



Welcome to WebmasterWorld, kingkahn.

I agree, check with your wholesaler first to see if the data is available in a more manageable format. I'm guessing you will find that yes, it likely is. Next, it sounds like you do not have your data in any other format besides hard-coded HTML pages. I think you will find life easier if you were to employ a database, or perhaps even a flat-file or XML document at least.

ytswy

4:00 pm on Dec 12, 2007 (gmt 0)

10+ Year Member



For your own stuff, 100% use a database.

I deal with a number of suppliers who don't seem to know one end of a computer from another (as in the concept of an export from their database is simply impossible to explain), so I could believe it if spidering was the only way to get the required information - I've actually considered doing this myself but this is definately a last resort and may well not be worth doing anyway.

It would need to be a custom script. You'd read in the pages and parse them for certain blocks of text to indicate where the relevant data is and then pull it off into your own data structure. At any time even a slight design change on the target site could break your script.

kingkahn

11:49 pm on Dec 12, 2007 (gmt 0)

10+ Year Member



ah ok,

Well thanks for the info.

As it turns out, I can export the entire list into a csv file. And my items are in a mysql database.

Im not sure how I would go about having the two of them look at each other..?

What do you think?

kingkahn

11:50 pm on Dec 12, 2007 (gmt 0)

10+ Year Member



thanks for the help.

coopster

3:55 pm on Dec 14, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



As it turns out, I can export the entire list into a csv file. And my items are in a mysql database.

You would need to create a (temporary) table and import your csv file into this table. The table format would be much like your existing MySQL table. Once imported, you can run a LEFT OUTER JOIN and a RIGHT OUTER JOIN (or use NOT EXISTS) to determine items you do not have in your database yet as well as items in your database that are not in your distributor's database (perhaps the item has been removed as they no longer provide it). Lastly, you would compare the item descriptions, price, etc. of the products that did indeed match in the INNER JOIN.