Welcome to WebmasterWorld Guest from 52.91.176.251

Forum Moderators: phranque

403 Forbidden WP-CONTENT

     
10:27 am on Aug 29, 2018 (gmt 0)

New User

joined:Apr 20, 2016
posts:22
votes: 1


My /wp-content/ folder is forbidden as i have disabled directory indexing. But this has started showing up in webmaster tools as Access Denied - Why is Google crawling /wp-content/uploads and how do i stop them from crawling these folders?

Images are still accessible on the full URL
1:10 pm on Aug 29, 2018 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts:523
votes: 46


All images, attachments are stored in your /wp-content/uploads directories. Therefore Google and most bots just go directly to the source and bypass your page. For indexing just images, Google does not want your web page content, just the image. This is very common. I think you should enable these directories.

Can you disallow viewing the directory index but allow direct access to the contents, if they know they are there? It is easy for a bot to scrape your page and extract only the images/download files. This is a common script request. They then have the direct URLs. Google and others do this.
4:00 pm on Aug 29, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:9066
votes: 752


Google does not want your web page content, just the image


Rather amazing statement if you think about it. The images are available via the normal content, so it does beg the question why the need to bypass?
4:17 pm on Aug 29, 2018 (gmt 0)

Senior Member from FR 

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Feb 15, 2004
posts:7139
votes: 410


The images are available via the normal content, so it does beg the question why the need to bypass?

So they can cache a copy and wrap ads around it* rather than send the searcher to the site.
If they don't do it now, they will, this is "just in case we need them"..
6:06 pm on Aug 29, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4168
votes: 262


You can stop the crawling with Disallow: in robots.txt for the folders you don't want crawled. They may still list these as "blocked resources" but they won't crawl if disallowed.

If you want them to have access to the images, you can use "Disallow:" for the folders and "Allow:" for the */*.jpg, */*.png or the filenames you use within those disallowed folders. Use Allow after Disallow.

Google will follow your instructions, but other search engines may follow or not. It doesn't block the robots, just tells them your preferences.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members