Updated to 5.6.x, now file get contents() isn't working

[csdude55]

I've been digging all day, but haven't found an answer to this.

I was running 5.3.x (I think, whatever EasyApache said was EOL), and updated to 5.6 last night. I had done some research and didn't foresee any major problems with the update.

But today I discovered that none of my pages using file_get_contents() are loading data anymore! This is how I generally grab XML content from remote sites (news, weather, etc), so I have several broken pages!

To my knowledge, updating PHP was the only change made. I do see, though, that they're all loading HTTP remote files, while my site is now HTTPS. I did a test run on my end using HTTP and it still didn't work, but still, it's worth mentioning.

My original code was basic:

$contents = file_get_contents("http://www.example.com/whatever.xml");

And today I tried this, with no notable impact:

$streamContext = stream_context_create([
 'ssl' => [
  'verify_peer' => false,
  'verify_peer_name' => false
 ]
]);

$contents = file_get_contents("http://www.example.com/whatever.xml", false, $streamContext);

That's based on what I read here:

[php.net...]

But both are coming back as "false", so it's definitely the file_get_contents() that's failing.

Any suggestions?

[csdude55]

Update to this...

For whatever reason, it turns out that some of the RSS feeds work, and some don't. I can't find any rhyme or reason to it; just SOME feeds that come up fine in the browser won't load through PHP. I took it down to the bare basics:

// This one worked yesterday, but doesn't now
// $file = "http://w1.weather.gov/xml/current_obs/KUKF.xml";

// This one does work, though
$file = "https://news.google.com/news/feeds?output=rss";

$contents = file_get_contents($file);
echo $contents;

The error message is pretty vague:

failed to open stream: HTTP request failed! HTTP/1.1 400 Bad Request

I thought that maybe the first one didn't work because it's not HTTPS while the Google feed is, but one of my other feeds is HTTP and it works, too. I would post that one, but it has an API key in it.

My exact PHP version is:

PHP 5.6.30 (cli) (built: Feb 2 2017 04:17:08)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
 with Zend Guard Loader v3.3, Copyright (c) 1998-2014, by Zend Technologies

I've tried disabling the firewall, I triple-checked in php.ini that "allow_url_fopen" was enabled, looked through every recently-updated file in the /etc/ directory (I'm running CentOS 6.8), and I'm coming up empty.

The only thing I'm finding online is related to HTTPS, though, so I modified it to this (based on the suggestion at php.net):

$streamContext = stream_context_create([
'ssl' => [
'verify_peer'  => false,
'verify_peer_name' => false,
'allow_self_signed' => true,
'verify_depth'  => 0
]
]);
$contents = file_get_contents($file, false, $streamContext);

But still, nothing.

And finally, I changed everything to cURL, but still no luck. If it were just one feed then I would think the problem was on their end, but this is several feeds on several different sites!

If any of you guys are running PHP 5.6.30, I would be much obliged if you would try running this script on your end and see if it works. I've literally spent 20 straight hours working on this, and no closer now than I was when I started! :'-(

[csdude55]

Last update... sorry, I would edit the last one, but this is really separate.

I downgraded back to 5.3.29, and everything started working again. I don't have much of a choice but to update to 5.6.30 in the near future since 5.3 is EOL, but for now the urgency is over. I still need to figure out the resolution to this, though.

[Dimitri]

To deal with URLs, I would recommend using the cURL functions of PHP. This will ensure a correct communication with the server.

[csdude55]

Thanks, Dimitri. I did try switching to cURL, but it didn't help with the problem. So the issue doesn't seem to be with the file_get_contents() command itself so much as the fopen wrapper.

[phranque]

HTTP request failed! HTTP/1.1 400 Bad Request

either you actually sent a Bad Request or the server responded as if you did.

is there a way to examine the request headers in php?
i think not in a file_get_contents scenario.
perhaps you could direct a request to a server under your control so you can see what is sent to the server...

[csdude55]

The error message I posted just came from:

if (!$contents = file_get_contents("http://w1.weather.gov/xml/current_obs/KUKF.xml")) {
 $error = error_get_last();
 echo $error['message'];
}

else echo "It worked!";

Now I'm back down to 5.3.29 so I can't retrieve more error information in 5.6.30 right now. Something weird, though, is that using cURL in 5.3.29 still gives an error, but file_get_contents() doesn't:

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $file);

curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$contents = curl_exec($ch);
curl_close($ch);

echo $contents;

// Using the weather.gov link, cURL gives me:
HTTP/1.1 403 Forbidden
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 303
Expires: Fri, 03 Feb 2017 21:48:02 GMT
Date: Fri, 03 Feb 2017 21:48:02 GMT
Connection: close

But file_get_contents() loads the XML data as expected, no errors.

I have 4 XML feeds that I'm testing. On 5.6.30, two of them worked and two gave an error like the above. Now that I'm on 5.3.29, all 4 work using file_get_contents(), but the two that didn't work with 5.6.30 apparently do not work using cURL on 5.3.29, either.

If you want me to include any other curl_setopt() commands for testing, please let me know! I don't understand 90% of them:

[php.net...]

I really appreciate the help!

[lammert]

The function file_get_contents() is not part of the standard PHP distribution. You have probably an old version on your server which is binary compatible with PHP 5.3, but not with PHP 5.6. Finding the right package to install differs somewhat dependent on the OS and distro you use, but you should update/upgrade all PHP packages to 5.6 to get everything working.

[keyplyr]

If you can, I'd go with php 5.7. Then you'd be set for a while.

[Dimitri]

I guess you meant 7.x

[csdude55]

I use WHM and EasyApache to update, but the highest it goes is 5.6.30. It's possible, though, that this is because I'm upgrading from 5.3; maybe if I was using 5.6 it would show higher options? I dunno.

Either way, if file_get_contents() and cURL are both failing in higher versions but not in 5.3 then I can't really upgrade to anything. Unless you guys using a newer version would be so kind as to copy my earlier script and see if it works for you?

[lammert]

I tried your script on PHP version 5.6.30 and it failed also with a 403 error sent by the remote site. I then changed the URL in the file_get_contents() call to one of my own sites and checked the request in the Apache log file. It seems file_get_contents() is using HTTP version 1.0 and it is not sending a User Agent.

Because of the missing User Agent the script on the server which you try to contact may have decided that the request is from a bot rather than a human and send a 403 error instead of the page contents. I have no idea though why PHP 5.3 succeeds as I have no older PHP versions in use than 5.6.30. But you can run a file_get_contents() call on one of your own pages yourselves and check the log file if there is a user agent present.

[keyplyr]

I guess you meant 7.x

I did intend to say 7, thanks Dimitri

[Dimitri]

To add a user agent, with cURL, you can set the option "CURLOPT_USERAGENT"


@keyplyr: Even the best can make mistakes :-)

[csdude55]

Good news! Setting the CURLOPT_USERAGENT on cURL worked on 5.3.29, so I'm going to modify all of my scripts and then try updating PHP again in a few days :D :D

I wonder, do you guys know if there's a performance boost comparing file_get_contents() to cURL?

Also, am I going to need to worry about the specified USERAGENT getting outdated? Is this something I'm going to have to update every few months or something? This feels like a hack, so I'm concerned about the reliability.

[Dimitri]

I have no idea if there are differences of performances between file_get_contents and using cURL, but I doubt it can be significant! It won't make the distant server to send the file faster or slower.

However, with cURL , you have much more flexibility to built your requests. You can even handle cookies (header, not javascriptof course), and define any of the request header fields. So, in my opinion, cURL is more robust choice.

[phranque]

I wonder, do you guys know if there's a performance boost comparing file_get_contents() to cURL?

curl uses dns caching and sends HTTP Keep-Alive headers by default, which could certainly affect speed if not memory usage.
not sure about file_get_contents but i doubt it.

[csdude55]

Sorry to bump this oldie, but...

Since 5.6.x no longer recognizes file_get_contents(), is it acceptable to create my own function using that name? In an effort to make all of my old scripts work without having to rebuild them?

function file_get_contents($url) {
 $ch = curl_init($url);

 curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36');
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

 curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 60);
 curl_setopt($ch, CURLOPT_TIMEOUT, 60);

 $contents = curl_exec($ch);
 curl_close($ch);

 return $contents;
}

[csdude55]

Just for the sake of posterity, and for future readers...

the answer is no, no you cannot create a function named file_get_contents(). I guess that would be too easy! I updated to 5.6.30 and then loaded the above function, and it resulted in a fatal error. I changed the function name to getFile(), though, and everything worked fine.