Welcome to WebmasterWorld Guest from 54.205.130.92

Rendering pages with Fetch as Google

   
11:03 pm on May 27, 2014 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



google announced today on the webmaster central blog that the Fetch as Google feature of Webmaster Tools will now show how Googlebot would render the page.

http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html [googlewebmastercentral.blogspot.com]:
In order to render the page, Googlebot will try to find all the external files involved, and fetch them as well. Those files frequently include images, CSS and JavaScript files, as well as other files that might be indirectly embedded through the CSS or JavaScript. These are then used to render a preview image that shows Googlebot's view of the page.


since this feature respects robots.txt, you will see the effects of any images, javascript and css that are excluded from crawling, such as broken navigation.
3:39 pm on May 28, 2014 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



About time Google made it easier for me to test my spammy cloaked pages :)
8:28 pm on May 28, 2014 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



When it comes to Google's so called *rendering* tools, my dilemma has always been them not including the images I disallow in robots.txt. But if I allow these image directories, Google will include my image files in their Image-Search, thus inviting every hobby page and summer-vacation forum on the www to hot-link.

Browsers (whether desk-top or mobile) have no issue, only Google's rendering tools... so what's the point?
10:14 pm on May 28, 2014 (gmt 0)

WebmasterWorld Senior Member Top Contributors Of The Month



What's the point? To let you see what Googlebot "sees." Some people will find the tool useful or reassuring; others will ignore it.
10:39 pm on May 28, 2014 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



A free tool is better than no tool. I can see this being helpful for educating clients that don't understand best practices or how crawling works.

As much as I joke about Google, I don't forget that Google needs to address a wide range of webmasters from ignorant and/or stupid to web geniuses. This tool might not be for everyone but I do think it can be helpful for many.
10:41 pm on May 28, 2014 (gmt 0)

10+ Year Member



You can allow Google a fetch of images yet still keep it out of the index. Allow it in robots.txt, then send this header with the image:

Header set X-Robots-Tag "noindex"
12:25 am on May 29, 2014 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month




You can allow Google a fetch of images yet still keep it out of the index. Allow it in robots.txt, then send this header with the image:

Header set X-Robots-Tag "noindex"


Thanks for the suggestion, but no... I've used the X-Robots header tag for years and still my images were scraped by both Bing and Google because at one point I didn't have the image directories disallowed in robots.txt. I have spent endless hours writing emails to them both, finally getting my images (10s of thousands) out of their image-search.

I'm not about to experiment again with this just so Google's little tool looks better.
9:15 pm on May 30, 2014 (gmt 0)



big news for SEOs ... its now really easy to show customers how bad the website renders for google crawler, you can even check mobile device renderings.

@keyplyr why not prevent hotlinking with referer-check?
12:57 am on May 31, 2014 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month




@derfmann - I do that and more.
10:31 am on May 31, 2014 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Update on my above statement concerning image file indexing: I've just been in a discussion at Google Groups concerning X-Robots tags. I've been assured by a Google employee they now fully support this tag (not so years ago.)
11:10 am on May 31, 2014 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



thanks for following up on that, keyplyr.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month