Forum Moderators: open
The web site in question has a "robots.txt"-file which dissallows all spidering (I donīt want it in the SEs at the moment). The emails gives instructions how to change my robots.txt, so that Googlebot is allowed to spider my web site (using Googleīs very special "allow" tag).
Ok, I admit, it feels good to have this game the other way round - Google asking *me* to have my web site included. I feel flattered. :)
Back to reality, I donīt think they love my site so much that they selected it by hand. So Google probably sends out these emails automatically and in large numbers. The bad "s"-word comes to my mind. This would be 100% spam, in my opinion.
My first thought was, that the email was a fake, not from Google at all. But it looks pretty real, and I donīt really see the benefit a third party would have from faking these emails. The email comes from "crawl-coverage@google.com" and is signed by a real person from Googleīs "Business Development".
Second thought - I am running AdWords for the web site in question. Maybe they only send these emails to advertisers with robots.txt that keep Googlebot from spidering. Google could justify this, IMO, and it makes some sense - in most cases, if you run Adwords for a certain web site, you also want it to be spidered.
But if this would be true, this would be the first proven connection between advertising on Google and having your web site spidered by Googlebot. Anybody else got these emails?
Anybody else got these emails?
Did Google Ask Me to Rewrite My robots.txt? - A strange email. [webmasterworld.com]
He never mentioned AdSense.
Hmmm..
In the other thread mentioned above, GoogleGuy says an email like this should be regarded as a single event, some sort of a personal decision from someone at Google. Surely we are the smart guys and donīt believe everything the google guys want us to. :)
GGīs version of how and why such an email might have been sended is a little hard to follow - there is a fixed text for these emails, so sending them out was obviously planned, and planned in larger numbers. You wouldnīt sit down and write a template for these emails (and even translate them) if you just wanted to send one or two emails, right?
The evil "s"-word is back in town.
You thought it was an honor to receive such an email, and now I spoiled your party. Sorry.
I understand your site is in German. Well, if you want to boycott Google, I have also some good news for you. In the last 13 months the percentage of German pages in the index of FAST/AllTheWeb grew from 6.6% to 8.6% (see this [webmasterworld.com] thread).
First, just because two webmasters have received similar emails (actually I think there was a third example discussed some time back), does not mean that emails are being sent out en masse.
Second, this is not a commercial email except in the remotest sense. In fact, while there are lots of valid reasons to exclude Googlebot from a site, I'm sure most people here will agree that Google is potentially doing this webmaster a huge favor.
Just because it is a form letter does not mean that they are not sent out on a case by case basis. It just means that they want to be careful about the wording.
If so, check the IP of the mailserver that mail came from. It is very possible that someone just spoofed in the google.com domain, but why?
If a bot wanted to crawl your site so bad, what does it need permission in the robots.txt for? Any bot that would want to crawl a site (look for email, etc) would just do it, wouldn't it? If it was a spam situation, and the sender for some reason wants your site listed in google (?) he probubly already used some sort of bot to get your email address off your page.
If a spammer was responsible for writting these letters to people, for a reason that i can't really understand, why would they take the time to hand pick out email addresses from websites? All of this leads me to think it was someone legit.
Hey GoogleGuy, can you "ask around the plex" to make sure everyone is finding my site in the SERPs easily? :)
Google is just being nice (in this case), paying attention to the robots.txt file, and sending targeted email.
Even if Google were sending these mails automatically to everyone who blocks them in robots.txt I do not see how it is spam. It a specifc email telling you something is wrong and how to fix it, it's not trying to sell you anything or waste your time with something you don't want. OK it might take some time to read it and you might already know but do we really want companies to feel like they can't do things like this?
Yes spams annoying, but don't tarnish everything with the same brush in my opinion.
Ok, I admit I am a little sensible when it comes to spam (in fact I am drowning in it, the ultimate test for any anti spam software), but I donīt mind personally about this email from Google. If my web site was hand picked, itīs charming, isnīt it..?
I started this thread to hear if others got these emails and to hear opinions. As there are not hundreds of other webmasters raising there hands, saying they got these emails as well, I agree with the previous posts and wouldnīt call this spam.
.. talking about boycotting Google is just ludicrous.
It a specifc email telling you something is wrong and how to fix it ..
This is the reason why Google could never institute a system (even if they wanted, which i'm sure they don't), where they auto sent a webmaster an email notifying them that their site is banned and how to fix the problem.
So many people want to scream "spam!"
Even if that letter was a template and sent to 100's of sites I don't see where the problem is.
It doesn't appear to be an auto-message sent all across the 'net to any site blocking Google because than it would be pretty common. So if you received that email than most likely someone liked your site and is wondering why you're blocking it from getting into the index.
The only way I can see someone not from Google sending that email is if the person was a link partner and wants your site in Google for pagerank purposes. Most of the other conspiracy theories about this are pretty silly.
It's just a harmless email.
Just because it is a form letter does not mean that they are not sent out on a case by case basis. It just means that they want to be careful about the wording.
Exactly. We have several prewritten emails and other documents at my company which are used as templates, some of which may be sent out only every couple of months. Usually it's because I wrote them and have someone else send them out, and want to be sure the message is communicated uniformly and accurately.
I'm sure it's a very common practice.
Perhaps AdWords "context" ads could be the answer.
In reviewing our Adwords results using their new reporting format, I see "total content targeting" is used in place of keywords for several destination URLs.
Possibly AdWords is using the whole page to match against AdSense publisher pages - so it needs to crawl your pages?
My 2 clams.
Persons Name
Business Development
Google Inc.
2400 Bayshore Parkway
Mountain View, CA 94043
personsname@google.com
I think its great, I'm flattered.
It seems an odd combo, and I am curious.
But you're right, very odd combo.
Just a guess for one situation... someone may have sites designed for the same keywords, one designed to close the sale which they advertise though Adwords. The other one is more of a soft-sell info site that is better suited for the main listings, and they dont really want to compete with their own site for the info searchers.
This way they can have one listing in the buying section (adwords), one in the info section (on the left) and as a bonus they dont have to worry about duplicate content or cross-linking issues for the info site(s) caused by their commercial site.
Sorry, no genius marketing or SEO ideas involved. :)
Sometimes parts of my web pages had examples that I didn't want indexed, yet GoogleBot went in there anyway. Some of these pages were eventually banned by human editors, because of Google's own mistakes.
So, I found that I could exclude GoogleBot from this kind of non-core content through an included Javascript file. Just document.write("CONTENT") in the .js file.
<script language=javascript src=yourfile.js></script>
This trick keeps overly-testosteronous GoogleBot from indexing parts of your web page that are distractors from your main message. Then, you don't get banned by "human editors": minimum wage slackers at Google that can barely read. You are at their mercy.
They do index images, but there are several ways to prevent his and I've never seen evidence they apply OCR to them.
#
# Hello Dear Googlebot, nice to see you again.
# Please get in and take it as your own home
# You can crawl as deep as you want, no problem.
# I will be in the kitchen , just whistle if you need something
User-agent: Mr Googlebot
allow: /Absolutely ALL
User-agent: Mr Googlebot
allow: /Wife and bedroom if you like, no prob.
by the way, it's not so much bootlicker, isn't?
