Welcome to WebmasterWorld Guest from 54.167.46.29

Message Too Old, No Replies

Couldn't Google have a Prefetch Command in the Robots.txt

proposal...

     
8:50 am on Apr 7, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 4, 2003
posts:94
votes: 0


Just a quick thought I'd like to run past you:

How about some lines in the robots.txt file

Disallow
xmoz-prefetch

When Googlebot checks the robots.txt file it would be nice IF they could note they we would rather they didn't put link tags in to prefetch the any pages.

Having Mozilla check the robots.txt before requesting the page would still result in a hit, but at least its only a single hit.

Any thoughts or suggestions that google could implement?

2:10 pm on Apr 7, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Because the prefetch is on a per-query basis and not on a per page basis. It would be possible, but I think the processing overhead wouldn't be warrented.

Again, lets not worry about the side effect of Google using prefetch and instead, worry about the disease that the Mozilla group would put the feature in there to start with.

[webmasterworld.com...]

4:14 pm on Apr 7, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 24, 2005
posts:965
votes: 0


Why would you want to stop preople prefetching your web site?

I go to extreme lengths to make sure my pages are fast loading, so visitors get a good first impression. Prefetch would be an added boost.

Is google using prefetch on any of your pages, or is it your question a hypothetical one?

8:26 pm on Apr 7, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2002
posts:741
votes: 0


Why would you want to stop preople prefetching your web site?

Bandwidth costs. The page might be pre-fetched without the searcher actually clicking on your result = wasted bandwidth.
8:32 pm on Apr 7, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 3, 2003
posts:785
votes: 0


Isn't prefetch a browser side technology...
effectively the bots are server side technology...

why would Google want to include "prefetch" in a bot crawl?
they are already storing enough data about each page they index...

I would think that the bot would always want to find the latest version of a page .. not anticipate that page..but actually see it and calculate the latest version...

8:36 pm on Apr 7, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2002
posts:741
votes: 0


Taken from the Google Helpcenter, Results Prefetching section:-

On some searches, Google automatically instructs your browser to start downloading the top search result before you click on it.
9:11 pm on Apr 7, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 24, 2005
posts:965
votes: 0


The page might be pre-fetched without the searcher actually clicking on your result = wasted bandwidth

It'll be a tiny fraction of your total bandwidth. By far the majority of people will click on the link.

Have you seen the kind of search terms the prefetch is used for? It's things like "webmaster world". How many people are likely to type in webmaster world but not click on the link?

9:14 pm on Apr 7, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 28, 2005
posts:614
votes: 0


How about Google prefetches only graphic files and only from the Google cache saving us the bandwidth?

Freq---

9:28 pm on Apr 7, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Mozilla prefetch command:

- uses bandwidth. With little indication that a user would ever visit.
- if the site is dynamic and using cache busting headers, the vistor will have to redownload the page anywhere - thus, you get speed penalized twice.
- falsifies referral information in your referral log.
- flasifies agent information in your agent log.
- causes system load.
- is basically an unstopable spider.

Lastly, from those whacky fun loving affiliate guys, comes this little tidbit:

Mozies prefetch offers the perfect opportunity to blindly cloak away for moz users with little risk of ever being caught - think about it.

9:22 pm on Apr 8, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 4, 2003
posts:94
votes: 0


hmm...

I just disliked their suggestion of returning a 404 error to the x-moz agent...

Mainly 'cos if someone looks at your page after its been prefetched how would you know they've read it.

Basically my customers could claim that I haven't provided as much traffic due to pages being prefetched and not actually read and I'd have know way of proving them wrong.

I thought if Google was aware that I didn't want any of my site prefetched then they could be kind enough not to tell mozilla to prefetch any of it.

End of my rant...

8:10 am on Apr 10, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


My gripe is as a user, with the browser fetching something that I would never have clicked on.

I saw that the other day when a site asked to set a cookie, and I was still looking at the SERP - not clicked on anything yet.

How many rogue sites are setting a prefetch to rogue content scam sites? I disabled it immediately.

There is an option to disable this in Mozilla 1.3+ which I had never noticed. It is in:
Preferences --> Advanced --> Cache

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members