Screaming Frog only crawls /en/ pages

[jediviper]

Hello to everyone,

I have noticed that for a couple of javascript websites with many different languages available, as domain.com/en/ or domain.com/de/ etc, the SF is only crawling the /en/ pages.
I think that it's because the /en/ is something like a default redirection set by javascript for some geolocations.

Anyone knows which settings may be needed to change in the configuration menu of SF?

[Dimitri]

I think that it's because the /en/ is something like a default redirection set by javascript for some geolocations.

Do you mean, your js code is redirecting visitors to the /en/de/... pages, depending of their localisation? If so, this is normal that, the SF crawer get redirected to the /en/ page, since it's certainly being operated from a datacenter in North america.

[jediviper]

Yes this is how the redirection works, but still there must be a way to force SF to crawl all locales.
I believe I have managed to run such a crawl in the past for some other sites, but I can't recall what was the setting that helped about this.

[Dimitri]

So, if YOU redirect requests, this is unrelated to the SF crawler itself. It's up to you, do detect the crawler and redirect it where you want.

[jediviper]

I think we are talking about 2 different things.
Anyway, thanks for your help.

Someone else who might know of what configuration setting could fix the problem ?

[jay5r]

You you have links on the /en/ version to the other versions? If not, adding them might be an obvious fix.

[jediviper]

@jay5r
Do u mean, if we have href lang links for all other languages? Sure we do.
Also there is a drop down menu with all available languages for all pages.
But I don't see how is this helping with the crawling problem of SF.