Forum Moderators: skibum
I have signed up and had a play with perl/xml, but not gone live yet. I was wondering if anyone had got it up and running and got any results from it, good, or bad.
<url snip>
and another was linking it with my forums member profiles so people could select their favorite artists. They submit their artist name and it pulls their cd catalog from Amazon.
<url snip>
I've been thinking about releasing the code, but it's not quite ready for prime time yet.
[edited by: WebGuerrilla at 7:19 pm (utc) on Oct. 11, 2002]
[edit reason] TOS #10 [/edit]
That's your basic "browse" through the categories deal.
I've also got it integrated into my main site database so if you bring up a movie, sountrack, or actor, it bounces out all kinds of things for folks to buy.
I also believe I'm the first person to have a working "Third Party" and "Buy Used" feature up. You make a lot lower percentage on that, but I'm hoping it balances out in volume once I get the PR on that sub-domain up and the low prices start popping up in SERPS.
Also, the Third party stuff is pretty much buried deep no matter how you cut it, so I've been working on a front end so sellers (if they feel like it) can set up a storefront page with categories for their products (and the ability to browse ONLY products they have available. If a seller doesn't have a front end set up on my site (and I'd guess any other site unless someone is REALLY clever) the visitor would have to a) find a product the seller has and b) click to the actual details page for that seller's product that they found and c) click again to get a product listing for that seller.
(If any of you folks sell Marketplace stuff on Amazon, I'm looking for some Beta Testers on this - drop me a line!)
I'm still always tweaking things, but the basics are there. Now it's just a matter of streamlining it so there aren't so many clicks from my front page to the "buy button" for a specific product.
When I first set it up, sales increased immediately by about 500%. Since then there has been a nice steady increase and I haven't had more than 1 day per week without at least one sale in about a month. Average is probably 3-5 items a day. I'm not getting rich, yet, but it's starting and if I can just clean up the navigation I'll be golden. (Part of the problem there is that there's just so MUCH on my site and I never want to back a visitor into a dead end where their next logical choice is to close the window). <shrug> I'll get it right, eventually!
G.
Grumpus - that was the answer I was looking for, its worth spending the time developing some scripts for it then. I have a few sites that it could slot into rather nicely. Do you mainly do book/cd sort of products or a lot of their other range as well? We have one site where we would be using all of that - the site has been almost dead for a couple of years now due to time constraints and lack of imagination to squeeze some revenue out of it, but I think this could be pretty good. I'll dust her down and see how she does - its gonna need a large re-vamp though!
We also have a power tools site that I would be looking to use this on, hoping this could be quite lucrative. Any joy in this area - do Amazon have a revenue limit on power tools or did they recently lift it - I cant quite remember.
Cheers
Gotta make two XSL sheets, one for product listings, and one for product display. Shouldn't be too difficult to make them work together, given XSL's "template" capability. When I get it to work, I'll post it at the Amazon webservices board for those of us who don't have server-side capability, but want a virtual amazon store.
I took a look at your site Grumpus, nice set up, you've got that integrated smartly. What sort of development time did you spend getting it up and running?
Do you mainly do book/cd sort of products or a lot of their other range as well?
The main focus is videos and cds (soundtracks, really). Obviously, the web services really help out as, for example, if you're looking at an actor's page on my site, not only can I zap up a list of movies for sale that the actor was in, but if they are someone who makes records, as well (Jennifer Love Hewitt, for example) it can pop up albums. It'll even bring up books by and about people. Another cool thing with movies, I can even key in the "action figure" and "video game" tie ins. Pretty nifty all around.
I'm going to try the XSLT method with one of my more successful sites
If you aren't much of a programmer, XSLT is the way to go because you can just take the sample templates and go. If you know a bit more, or are a tinkerer, as I am, you'll want to work right from the raw XML. What I'm doing is just creating an XML object, loading the results into that and parsing all the data into my variables at the top of each page. From there, I can do pretty much whatever I want to with it. Much more versitile, really.
If you aren't familiar with XML and your chosen platform (ASP, PHP, whatever) there is a definite learning curve involved that is much more steep than XSLT, but worth it if you really want to do some fancy stuff (now, or down the road).
G.
Anyway, my script was built off some existing work called amazon_functions, just search for it on google. It lays down the basic work of grabbing and caching the Amazon feeds.
I suppose it's a matter of being able to tie everything in and it's a bit trickier if it doesn't have an ASIN. They tell me it's something that's in the works and that sometime not toooo far down the road, we should see all zShop items and probably even Auction items coming over the feeds. THAT is when it'll get really fun to develop stuff. :)
G.
This said, I've had occasional trouble with pages hanging, which I suspect is to do with the version of XML on the server . I've tried deleting all versions and reinstalling version 4 and still it happens - when it tries to create the XML object. When I stop and start IIS the same page works again. Anyone got any experience of this?
Yeah, the XML server hangs sometimes. It's been a LOT better lately. Last month at about this time, there were periods when it would hang your entire page load no matter what you did for 3-4 minutes! Now, it seems, that if the server can't handle your request at that moment, it sends you a proper error message that you can parse into something along the lines of "Server Too Busy, Try Back later" a la what DMOZ does. There HAVE still been occassional periods where it's hanging, though. It's MUCH less frequent.
To remedy this (well, sort of remedy this) I've built a giant switch on my site so that if the feeds start hanging, I can flip the switch and the AWS feed vanishes from my site completely. (It's just a table with a 1,0 field in it and each call to the AWS checks for a 1 in that table before making the call). This is a less than perfect solution, but it works.
To be honest, I think the whole AWS thing started out as some dude at Amazon just tinkering and one of the suits said "Hey, that's RFC! Release it!" Now that it's taking off, they're scrambling to upgrade servers and all of that. It's a slow, but steady process of improvement on both the dependability and features.
In Amazon's defense, this is, by far, the largest venture of this sort on the web. With less than a week of coding, I was able to increase the spiderable content on my site - oh - about 2 times. A month and a half later, that figure is closer to 50 times. In its current state, I'd say there have to be about 8-10 million spiderable pages there. Just the load on amazon's server during the Google crawl from my site alone is going to put it through its paces, not to mention if it's hitting 10 other sites at the same time.
As time goes by and Amazon dedicates more resources to this (and they already have come a LONG way since they first introduced the rudimentary functions this summer) it'll all get better.
G.
The code for the switch is simple. Make a table. Give it one recordset with one field that is a boolean (or just a 1 or 0). Call that table.
<% if KillAmazon.Fields.Item("onoff").Value = 1 then %>
<AMAZON CODE HERE>
<% end if %>
Then when you want to turn it off, just update the table with a 0 in there instead of the 1. Just remember that if you get the XML and parse it up top and then use a second run of code lated in the page to display your parsed data, you'll need two "IF / THEN"'s If you never create the XML object, but try to display it later in the page, it's going to give you an "object required" error.
Nothing really fancy, it just requires you to go in and surround all you amazon code with the IF/Thens...
G.
I started looking at the example perl scripts that come with the Amazon Web Services pack, had a look through it all and at first I thought, ok that'll work and we can build on that. But then I changed my mind.
The thing that bugged me the most was that the xml feed is by its very nature a strucutured document. The sample perl script basically then went on to parse the xml doc and put it into a different structure based on arrays of hashes. Why not take the strcutured xml, parse it and output it. I would think that by parseing it and storing in another structure the overheads are increased.
I'd be interested to hear if any of you think this sounds sensible. I decided to go down a different route - use php to request the xml doc and transform using an xsl stylesheet. I've not gone live with a final app yet but it seems to be working pretty well.
Cheers
As I said in a previous post, I've had best luck with just yoinking the XML, parsing out the data into an array of strings above the HEAD in my ASP, then just slapping the strings onto the page where they should appear. In reality for basic Amazon listings (there's more involved when you start working with Marketplace data and browsenodes), I've got two pages - the results list code and the details code. No matter whether it's a book or a dvd or a silly vibrating Harry Potter Broom, it's all processed by the same code.
This puts more work on my server than Amazon's, which is just fine. If something goes fishy with my server, I have some control over it. If it's on Amazon's server, I have to send a report and then, if I'm lucky, 24-72 hours later, I get a "We're aware of the problem and are working to remedy it" message.
Musicale's "Amazon Lite" code is REALLLY GOOD, and is a great place to start when trying to figure out how to do all of this in this way. (I know it was invaluable to me when I first started tweaking around). I haven't looked at his updated versions, but the original stuff (while there isn't much in the way of comments) really demonstrates exactly what's happening. If you have "intermediate" or better skills in programming, you should have no problem looking at his source and figuring out what's going on.
I haven't posted any of my code (except for the occasional snippet when asked) for the simple fact that for tracking and a few other things, I have it linked into my "user" database, and there's the whole security issue involved.
While the core idea of what I'm doing is based upon the methods Musicale is using, I have made a few changes that I think really help speed up everything. Here's the key one:
I'm getting the XML object open, parsed, and closed as quickly as possible. In the Amazon Lite example (at least the Version 1 that I looked at), the parsing of the XML and the creation of the page is all happening at the same time. I've created an array and assigned ALL the variables and then dumped the XML object before I even get to the HEAD portion of my code. (Amazon Lite 1.0 uses an array for the reviews portion, but other than that, it grabs the data, writes it to the page, grabs the next bit, and writes, and so on.)
Not only does the XML object stay in the server's memory longer, but if you want to change where any of the elements appears on the page, you've got to move entire blocks of code. With the array, I can just put the "<B>Product Title: </B> <%=ProductTitle(X)%>" code somewhere else and it still works. (Maybe Amazon Lite does that now, too - as I say, I haven't looked at the code for newer versions).
G.
I have a handful amazon links on my pages, and just changed them to add the item directly to the shopping cart, hoping that will improve my chances of a 15% commission. Before that, I had to establish a SOAP request, so that I can display life prices (I hate to get something added to a cart before I see what it costs myself).
The SOAP thingie was extremely simple to implement, using Mark Pilgrims amazon.py module, which reduces a request for the details from a row of ASINs to a simple function call from my Python module. My own script is then integrated into each page by SSI.
It would be trivial to cache the price information for the permitted hour, if my traffic actually justified the effort. But reading the information further above so far, caching might also be useful for when the amazon XML servers are overloaded or inaccessible for other reasons. Right now, I'm not displaying any books in this case.
So, the interesting question for me would be: Does it happen that the amazon XML servers go down, while the "normal" amazon site is still happily running along? If they tend to go down together, then caching would be pointless as a backup measure. If the XML feed goes down alone, then serving up my own backup data might secure me an additional sale every few months...
Any clues?
On the other point, thanks again for the response Grumpus, very useful. I can see that storing the info in an array has less overhead than actually having an open xml connection, hence better for later manipulation. Although with regard to the xsl, I am not actually sending it to Amazon to apply, rather Im keeping all of this done at my end and using php to apply the stylesheet. I guess there is still more bandwidth requirement doing it this way by I'm not sure how much - doing it locally seems great but naturally when i go live things will slow down - I'll let you all know how it goes - if its a total dog I guess I will have to admit defeat and go down the other route, I'll kepp you posted.
Thanks for all your input.
The problems seem to come in when the feed is slow. Originally, I had my code set up so that objects (both the XML objects and recordset calls for the database) were opened at the top of my code, processing was done, then everything got closed at the end of the asp code. Recently, I've had to move my close calls up into the code to the line after the last time I need that object.
What was happening (and still seems to happen on occassion, so there must still be pages I didn't fix) is that the ASP code would open a crapload of objects, wait for the Amazon feed, process everthing, then close everything. If the feed is slow, people get impatient halfway through the page process and hit reload. This leaves ALL the objects open and then goes and opens a whole batch of new ones. As you can see, before long - with 300-400 people an hour hitting reload - the server soon becomes overwhelmed with too many open connections and just hangs. Then, the site is effectively (though not completely) offline for the 5-10 minutes it takes to realize that it's in a pickle and that the IIS needs to restart. Then, it starts all over again.
The remedies that seem to help most:
Call the XML FIRST thing in the code. This means, that if it's slow to come and the person hits reload, it's only 1 object that's left in limbo.
Close everything as soon as possible. As soon as I'm done with a recordset, I get it out of there.
Process the XML all at once (before the HEAD tags in the document) and get it gone. A HEAVY XML document from Amazon can be huge with the product reviews and such.
These changes seem to really help a lot. My site still runs into trouble sometimes, but I'm not 100% certain that I've got it all fixed on every single page, yet. Gonna go through all of that once I finish my coffee this morning.
G.
it doesn't look to me like you really solved your problem yet, you seem to simply delay the symptoms. I know next to nothing about ASP, so I may be well off the mark here, but this is how I read your description:
Apparently, if you don't close an object (network or database connection) in ASP, then it will linger there forever. This is disaster prone behaviour for a scripting environment to start with (which should do automatic resource cleanup and garbage collection), but maybe you're just stuck with that. So, if this is really the case, then there's only one correct solution:
You need to encapsulate every such call in a timeout yourself, and close the connection whether the visitor got the page served or hit reload. This being a purely theoretical comment, I can't tell you how to do that or if it's actually possible in ASP. But if you don't to at least something aequivalent, then you're bound to hang your server every so often. The intervals will be longer if you only leave one object lingering per request instead of several, but in my world, every server hang is one too many.
Ok, enough ranting about stuff I don't know anything about... ;)
With my solution, that 120+ number changes to 20 and that leaves me room for growth to 6 times the traffic I'm currently getting before I end up in a new and stickier situation.
G.
