readfile("http://.") is slowing down my page downloads

How do I read contents of remote script without slowing down my page load

[etdebruin]

I have built a website that relies pretty heavily on doing a bunch of readfile("http://remotewebsite.com") for displaying content.

The problem is that the page load slows down tremendously and I need to reconsider how this is done. Here are a few questions:

1. Is it better to use cURL or readfile?

2. Can cURL be set up in such a way that it performs non blocking downloads?

3. Is there an HTML tag I can wrap my readfile in so that a my readfile can happen concurrently to page load? Similar to what happens when <img> or <script> tags are executed?

AJAX is not the answer for me because I need the content from the readfile to be visible to Search Engine Robots.

Any ideas would be greatly appreciated.

[coopster]

Welcome to WebmasterWorld, etdebruin.

readfile() should work fine, I'm not sure what performance increase, if any, curl would provide. You would have to set it up and test it. I'm not sure what you mean by non-blocking downloads but if you aren't meant to be downloading something from another site, perhaps you should be rethinking your page contents and site design. Whatever you seem to be accomplishing just doesn't sound like good site layout/design. Why can't you develop and read the content from your local server?

[NameNick]

Hi,

I suppose the remote access is the bottleneck. Instead of retrieving the file from the remote server every time the page gets called I would cache the content of the remote page. You could use a cron job for this.

NN

[etdebruin]

Actually, caching was the answer. I implemented the following caching scheme:

a. Generate an md5 of the filename and QUERY_STRING
b. Check if the file exists in the local cache
c. If not, do the readfile from [filename?QUERY_STRING...]
d. If true, do the readfile from CACHE/md5(filename+QUERY_STRING)

Works like a charm!

The only problem I had to solve was how to refresh the cache and instead of opting for some time based solution, I have my content server update the local site cache whenever the content changes on the server. This way content that hardly ever changes, will only get read remotely once.

Done.

[BarryStCyr]

You could always store a timestamp for the last time a particular page was downloaded into your cache and if a user requests the page and the timestamp is older than a certain time period, like an hour, day or week, you could then download a new page to cache and serve that page. Then only the people asking for an outdated page would wind up having to wait. Other requesting the page before the cache time period would get the cached page automatically.

Barry

[etdebruin]

This system would not work as my content server needs to expire cache items that have been updated by people who use our CMS.

So, if a page of content has been updated by our user, our CMS server updates that customer's site cache so that the next query is served by our CMS server and not by the client cache.

It works really well.