Welcome to WebmasterWorld Guest from 54.167.112.42

Forum Moderators: Ocean10000 & phranque

Message Too Old, No Replies

W3C ScreamingFrog no longer work on a site after https

W3C ScreamingFrog no longer work on a site after https

     
5:30 pm on Feb 23, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


Ever since https was set (by the host) up on a site I manage Screaming Frog doesn't work (connection refused error) which as far as I can tell means I need to update Java on my computer (Mac Sierra), which I did with no change.

Also when I try to validate the site in W3C it gives me an IO error (I can't find definitive info on what is causing this). I used another validator and it said the site has no html errors and I checked the CSS elsewhere with the same result.

Can someone tell me what might be wrong with the site or what to look for?
8:56 am on Feb 24, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:1982
votes: 330


Run your site through the SSL Server Test [ssllabs.com]. Does anything stand out? An incomplete certificate chain, perhaps?
2:41 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


@Robzilla

I ran the site through SSL Server test and the overall rating is A-

Do you need more info?
2:57 pm on Feb 24, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:1982
votes: 330


That rules out any chain issues, because your grade would be capped to a B.

It's probably an issue with Java, as you said, so I wouldn't worry too much about it. This would only affect tools written in Java, not any of the common web browsers. Could be something related to SNI, but it's hard for me to tell.
2:57 pm on Feb 24, 2018 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11451
votes: 172


Screaming Frog doesn't work (connection refused error)

did you check the log file for clues?

I need to update Java on my computer (Mac Sierra), which I did with no change

did you install this security fix?
Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8 Download [oracle.com]

in W3C it gives me an IO error

https://validator.w3.org/feed/docs/error/IOError.html [validator.w3.org]:
One specific case that causes this is a server that claims the file is compressed, but then serves an uncompressed file.
3:24 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


@phranque

I had compression set up in htaccess but took it back out when this problem came up just to make sure that wasn't the problem.

I'll check the security fix.
3:55 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


@pranque

I know very little about access logs, but I think I understand what they say.

I checked the access logs but can't find any of my recent visits to the site. I searched for W3.org and my IP via the "find" feature on Firefox but it doesn't seem to be working. Also the most recent entries for today are not recorded.
3:58 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


@phranque

I downloadr the Java Cryptography extension (local one) but it wouldn't install - hung up.
4:13 pm on Feb 24, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:1982
votes: 330


If the connection is refused, I don't think it will show up in the access or error logs.

Are you on shared hosting? Running Apache? If so, what version? (See response headers, or search "check apache version online"). Does handshake simulation work with Java clients, as reported by the SSL Labs tool? Java 6u45 will probably show "No SNI", but the newer versions should be OK.
4:40 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


shared hosting - yes
Apache/2
Re handshake -- can you tell me how to check this? I'm not a programmer.

I'm on Mac Sierra. I have Fireworks CS5 which requires Java in order to work on the new OS, and it is working.
4:52 pm on Feb 24, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:1982
votes: 330


Do you know what Apache version exactly?

Regarding the handshake simulation, it's in the SSL Labs report.
6:15 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


This site is hosted on ipower and I couldn't find the exact version of apache. I tried a few redirect checkers but they only list it as apache/2

Can you tell me where the SSL labs report is? I checked all the SSL data in the control panel.
6:17 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


BTW, as I mentioned above this is the only site I work on with this problem so why would i need to install the Java Cryptography extension on my computer?
6:54 pm on Feb 24, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:1982
votes: 330


I ran the site through SSL Server test and the overall rating is A-

Can you tell me where the SSL labs report is? I checked all the SSL data in the control panel.

It's this one :-)

BTW, as I mentioned above this is the only site I work on with this problem so why would i need to install the Java Cryptography extension on my computer?

It might help solve the issue of Screaming Frog not being able to crawl your site, because that's a local client (i.e. on your computer). But it doesn't get to the cause of the issue, which is likely server-side.
8:38 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


>Does handshake simulation work with Java clients, as reported by the SSL Labs tool? Java 6u45 will probably show "No SNI", but the newer versions should be OK.

Ok, I see what you mean.

Yes Java6u45 shows it's an incorrect certificate - this client doesn't support SNI. Java 7 and 8 are ok.
Also IE8 /XP fatal alert but not many still using it. Also android 2.23.7 (very old).

That's all the handshake errors.
9:03 pm on Feb 24, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


I checked Java on my computer after upgrading and it's version 8.

I restarted the computer in case that was needed, then ran screaming frog again and got the same error: connection refused.
1:28 am on Feb 25, 2018 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11451
votes: 172


Screaming Frog doesn't work (connection refused error)

did you check the log file for clues?

I checked the access logs but can't find any of my recent visits to the site.

not the server access log file - you can view seo screaming frog's log file by going to Help->Debug.
8:23 am on Feb 25, 2018 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator ianturner is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 19, 2001
posts: 3648
votes: 50


Is that the only site on that particular host?

They might have blocked Screaming Frog if it has any telltales.
4:59 pm on Feb 25, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


@Ian Yes, this is the only site I manage on that host.

However, W3.org and other validators produce an IO errror and will not validate this site so it's not just screaming frog.
5:13 pm on Feb 25, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts: 1895
votes: 3


@phranque

Ok, I figured out how to view the Screaming Frog log file.

Fatal Log File: /Users/lorrie/.ScreamingFrogSEOSpider/crash.txt

This problem may be that it does not recognize the Java 8 version that is installed.

Java Info: Vendor 'Oracle Corporation' URL 'http://java.oracle.com/' Version '1.8.0_131' Home '/Applications/Screaming Frog SEO Spider.app/Contents/jre/Home'

I downloaded the security fix but it starts installing then quits. I tried it twice.

BTW, I"m using the free version.
9:06 am on Feb 26, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:1982
votes: 330


According to the SSL Labs test results, the server sends certificates in the wrong order, and there are multiple certificate paths. Older clients may trip on that, as you can tell from the "IE 8 / XP Server sent fatal alert: handshake_failure" error under "Handshake Simulation". I once even had trouble getting Chrome to connect properly, though it seemed to work in most cases. You can take that up with your web host (show them the SSL Labs results with the "Incorrect order" warning), but since you're on shared hosting I kind of doubt they would do anything about it. Whereas many web hosts with cPanel or Plesk as their control panel offer free turn-key Let's Encrypt certificates, it looks like yours does not.

Do you have any control over the certificates in your host's control panel?
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members