Welcome to WebmasterWorld Guest from 54.146.169.191

Forum Moderators: coopster & jatar k

Updated to 5.6.x, now file get contents() isn't working

     
10:59 pm on Feb 2, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


I've been digging all day, but haven't found an answer to this.

I was running 5.3.x (I think, whatever EasyApache said was EOL), and updated to 5.6 last night. I had done some research and didn't foresee any major problems with the update.

But today I discovered that none of my pages using file_get_contents() are loading data anymore! This is how I generally grab XML content from remote sites (news, weather, etc), so I have several broken pages!

To my knowledge, updating PHP was the only change made. I do see, though, that they're all loading HTTP remote files, while my site is now HTTPS. I did a test run on my end using HTTP and it still didn't work, but still, it's worth mentioning.

My original code was basic:

$contents = file_get_contents("http://www.example.com/whatever.xml");


And today I tried this, with no notable impact:

$streamContext = stream_context_create([
'ssl' => [
'verify_peer' => false,
'verify_peer_name' => false
]
]);

$contents = file_get_contents("http://www.example.com/whatever.xml", false, $streamContext);


That's based on what I read here:

[php.net...]

But both are coming back as "false", so it's definitely the file_get_contents() that's failing.

Any suggestions?
3:24 am on Feb 3, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


Update to this...

For whatever reason, it turns out that some of the RSS feeds work, and some don't. I can't find any rhyme or reason to it; just SOME feeds that come up fine in the browser won't load through PHP. I took it down to the bare basics:

// This one worked yesterday, but doesn't now
// $file = "http://w1.weather.gov/xml/current_obs/KUKF.xml";

// This one does work, though
$file = "https://news.google.com/news/feeds?output=rss";

$contents = file_get_contents($file);
echo $contents;


The error message is pretty vague:

failed to open stream: HTTP request failed! HTTP/1.1 400 Bad Request

I thought that maybe the first one didn't work because it's not HTTPS while the Google feed is, but one of my other feeds is HTTP and it works, too. I would post that one, but it has an API key in it.

My exact PHP version is:

PHP 5.6.30 (cli) (built: Feb 2 2017 04:17:08)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with Zend Guard Loader v3.3, Copyright (c) 1998-2014, by Zend Technologies


I've tried disabling the firewall, I triple-checked in php.ini that "allow_url_fopen" was enabled, looked through every recently-updated file in the /etc/ directory (I'm running CentOS 6.8), and I'm coming up empty.

The only thing I'm finding online is related to HTTPS, though, so I modified it to this (based on the suggestion at php.net):

$streamContext = stream_context_create([
'ssl' => [
'verify_peer' => false,
'verify_peer_name' => false,
'allow_self_signed' => true,
'verify_depth' => 0
]
]);
$contents = file_get_contents($file, false, $streamContext);


But still, nothing.

And finally, I changed everything to cURL, but still no luck. If it were just one feed then I would think the problem was on their end, but this is several feeds on several different sites!

If any of you guys are running PHP 5.6.30, I would be much obliged if you would try running this script on your end and see if it works. I've literally spent 20 straight hours working on this, and no closer now than I was when I started! :'-(
4:38 am on Feb 3, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


Last update... sorry, I would edit the last one, but this is really separate.

I downgraded back to 5.3.29, and everything started working again. I don't have much of a choice but to update to 5.6.30 in the near future since 5.3 is EOL, but for now the urgency is over. I still need to figure out the resolution to this, though.
9:21 am on Feb 3, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Nov 13, 2016
posts:260
votes: 33


To deal with URLs, I would recommend using the cURL functions of PHP. This will ensure a correct communication with the server.
9:47 am on Feb 3, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


Thanks, Dimitri. I did try switching to cURL, but it didn't help with the problem. So the issue doesn't seem to be with the file_get_contents() command itself so much as the fopen wrapper.
10:10 am on Feb 3, 2017 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10683
votes: 33


HTTP request failed! HTTP/1.1 400 Bad Request

either you actually sent a Bad Request or the server responded as if you did.

is there a way to examine the request headers in php?
i think not in a file_get_contents scenario.
perhaps you could direct a request to a server under your control so you can see what is sent to the server...
9:53 pm on Feb 3, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


The error message I posted just came from:

if (!$contents = file_get_contents("http://w1.weather.gov/xml/current_obs/KUKF.xml")) {
$error = error_get_last();
echo $error['message'];
}

else echo "It worked!";


Now I'm back down to 5.3.29 so I can't retrieve more error information in 5.6.30 right now. Something weird, though, is that using cURL in 5.3.29 still gives an error, but file_get_contents() doesn't:

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $file);

curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$contents = curl_exec($ch);
curl_close($ch);

echo $contents;

// Using the weather.gov link, cURL gives me:
HTTP/1.1 403 Forbidden
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 303
Expires: Fri, 03 Feb 2017 21:48:02 GMT
Date: Fri, 03 Feb 2017 21:48:02 GMT
Connection: close


But file_get_contents() loads the XML data as expected, no errors.

I have 4 XML feeds that I'm testing. On 5.6.30, two of them worked and two gave an error like the above. Now that I'm on 5.3.29, all 4 work using file_get_contents(), but the two that didn't work with 5.6.30 apparently do not work using cURL on 5.3.29, either.

If you want me to include any other curl_setopt() commands for testing, please let me know! I don't understand 90% of them:

[php.net...]

I really appreciate the help!
6:36 pm on Feb 4, 2017 (gmt 0)

Senior Member from KZ 

WebmasterWorld Senior Member lammert is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 10, 2005
posts: 2929
votes: 20


The function file_get_contents() is not part of the standard PHP distribution. You have probably an old version on your server which is binary compatible with PHP 5.3, but not with PHP 5.6. Finding the right package to install differs somewhat dependent on the OS and distro you use, but you should update/upgrade all PHP packages to 5.6 to get everything working.
7:06 pm on Feb 4, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7550
votes: 243


If you can, I'd go with php 5.7. Then you'd be set for a while.
9:25 pm on Feb 4, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Nov 13, 2016
posts:260
votes: 33


I guess you meant 7.x
9:41 pm on Feb 4, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


I use WHM and EasyApache to update, but the highest it goes is 5.6.30. It's possible, though, that this is because I'm upgrading from 5.3; maybe if I was using 5.6 it would show higher options? I dunno.

Either way, if file_get_contents() and cURL are both failing in higher versions but not in 5.3 then I can't really upgrade to anything. Unless you guys using a newer version would be so kind as to copy my earlier script and see if it works for you?
10:13 pm on Feb 4, 2017 (gmt 0)

Senior Member from KZ 

WebmasterWorld Senior Member lammert is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 10, 2005
posts: 2929
votes: 20


I tried your script on PHP version 5.6.30 and it failed also with a 403 error sent by the remote site. I then changed the URL in the file_get_contents() call to one of my own sites and checked the request in the Apache log file. It seems file_get_contents() is using HTTP version 1.0 and it is not sending a User Agent.

Because of the missing User Agent the script on the server which you try to contact may have decided that the request is from a bot rather than a human and send a 403 error instead of the page contents. I have no idea though why PHP 5.3 succeeds as I have no older PHP versions in use than 5.6.30. But you can run a file_get_contents() call on one of your own pages yourselves and check the log file if there is a user agent present.
12:10 am on Feb 5, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7550
votes: 243


I guess you meant 7.x
I did intend to say 7, thanks Dimitri
11:56 am on Feb 5, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Nov 13, 2016
posts:260
votes: 33


To add a user agent, with cURL, you can set the option "CURLOPT_USERAGENT"


@keyplyr: Even the best can make mistakes :-)
3:59 am on Feb 6, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Mar 15, 2013
posts: 222
votes: 23


Good news! Setting the CURLOPT_USERAGENT on cURL worked on 5.3.29, so I'm going to modify all of my scripts and then try updating PHP again in a few days :D :D

I wonder, do you guys know if there's a performance boost comparing file_get_contents() to cURL?

Also, am I going to need to worry about the specified USERAGENT getting outdated? Is this something I'm going to have to update every few months or something? This feels like a hack, so I'm concerned about the reliability.
9:53 am on Feb 6, 2017 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Nov 13, 2016
posts:260
votes: 33


I have no idea if there are differences of performances between file_get_contents and using cURL, but I doubt it can be significant! It won't make the distant server to send the file faster or slower.

However, with cURL , you have much more flexibility to built your requests. You can even handle cookies (header, not javascriptof course), and define any of the request header fields. So, in my opinion, cURL is more robust choice.
10:48 am on Feb 8, 2017 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10683
votes: 33


I wonder, do you guys know if there's a performance boost comparing file_get_contents() to cURL?


curl uses dns caching and sends HTTP Keep-Alive headers by default, which could certainly affect speed if not memory usage.
not sure about file_get_contents but i doubt it.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members