Welcome to WebmasterWorld Guest from 54.162.205.151

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Rendering pages with Fetch as Google

     
11:03 pm on May 27, 2014 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10551
votes: 10


google announced today on the webmaster central blog that the Fetch as Google feature of Webmaster Tools will now show how Googlebot would render the page.

http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html [googlewebmastercentral.blogspot.com]:
In order to render the page, Googlebot will try to find all the external files involved, and fetch them as well. Those files frequently include images, CSS and JavaScript files, as well as other files that might be indirectly embedded through the CSS or JavaScript. These are then used to render a preview image that shows Googlebot's view of the page.


since this feature respects robots.txt, you will see the effects of any images, javascript and css that are excluded from crawling, such as broken navigation.
3:39 pm on May 28, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3100
votes: 86


About time Google made it easier for me to test my spammy cloaked pages :)
8:28 pm on May 28, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6157
votes: 75


When it comes to Google's so called *rendering* tools, my dilemma has always been them not including the images I disallow in robots.txt. But if I allow these image directories, Google will include my image files in their Image-Search, thus inviting every hobby page and summer-vacation forum on the www to hot-link.

Browsers (whether desk-top or mobile) have no issue, only Google's rendering tools... so what's the point?
10:14 pm on May 28, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:June 28, 2013
posts:2544
votes: 328


What's the point? To let you see what Googlebot "sees." Some people will find the tool useful or reassuring; others will ignore it.
10:39 pm on May 28, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3100
votes: 86


A free tool is better than no tool. I can see this being helpful for educating clients that don't understand best practices or how crawling works.

As much as I joke about Google, I don't forget that Google needs to address a wide range of webmasters from ignorant and/or stupid to web geniuses. This tool might not be for everyone but I do think it can be helpful for many.
10:41 pm on May 28, 2014 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 10, 2004
posts: 388
votes: 12


You can allow Google a fetch of images yet still keep it out of the index. Allow it in robots.txt, then send this header with the image:

Header set X-Robots-Tag "noindex"
12:25 am on May 29, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6157
votes: 75



You can allow Google a fetch of images yet still keep it out of the index. Allow it in robots.txt, then send this header with the image:

Header set X-Robots-Tag "noindex"


Thanks for the suggestion, but no... I've used the X-Robots header tag for years and still my images were scraped by both Bing and Google because at one point I didn't have the image directories disallowed in robots.txt. I have spent endless hours writing emails to them both, finally getting my images (10s of thousands) out of their image-search.

I'm not about to experiment again with this just so Google's little tool looks better.
9:15 pm on May 30, 2014 (gmt 0)

New User

joined:May 30, 2014
posts:8
votes: 0


big news for SEOs ... its now really easy to show customers how bad the website renders for google crawler, you can even check mobile device renderings.

@keyplyr why not prevent hotlinking with referer-check?
12:57 am on May 31, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6157
votes: 75



@derfmann - I do that and more.
10:31 am on May 31, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6157
votes: 75


Update on my above statement concerning image file indexing: I've just been in a discussion at Google Groups concerning X-Robots tags. I've been assured by a Google employee they now fully support this tag (not so years ago.)
11:10 am on May 31, 2014 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10551
votes: 10


thanks for following up on that, keyplyr.