403 Forbidden Error

[nobsseo]

When checking a client's site using a server response tool, i noticed it returns a "403 error". This forbidden 403 error seems to be stopping bots from crawling the site. However, if you type the URL into your browser's address bar, you can still access all of the pages of the site, no problems.

So it's returning a forbidden 403 error to bots, but the pages work fine for users.

Does anyone know what specifically could cause this? And how to fix it?

Cheers

[le_gber]

if you use firefox download the header checking extension and try browsing the page with this extension on.

the 403 might be because they are banning certain ip addresses or automated queries etc...

using the browser extension might show you the code 200.

If this is the case (ie 200 on browser and 403 on server header tools) you should contact the host and ask him/her to enable certain ips / countries.

If you get a 403 for both it's a server config problem so the host should be able to solve this.

[scintex]

If you're behind a proxy you might not have set the tool up properly to work with your proxy (for example my corp proxy requires my network credentials so I have to enter them into automated tools as well as the proxy address). The 403 could be coming from the proxy and not the site.

Sounds silly but I've done that a few times!