Forum Moderators: Robert Charlton & goodroi
Figured I'd make a post here since I haven't in a while. I recieved an email from someone at google telling me the benifits of allowing the google spider to index my site. The site in question has a robots.txt file disallowing all search engines.
The email includes the URL in question and states that atleast 50% of internet searches come from google and its important to allow google to spider the site.
First reaction was, automated responce? Did I send an inquiree to them? Nopes certainly not for this domain so I sent an email back asking why the email and I expected it to bounce. To my surprise this morning I got a reply back saying they want to include the site because it was identified as being highly ranked.
New sort of team at google trying to get people uninformed and disallowing google to switch back and allow google to spider the data?
I've heard of these emails before but they are not common, so I guess kudos is in order. :-)
DaveN
very interesting, i hadn't heard of these emails until now.
ciml - i would have put money on you knowing about them ;o)
must be some sort of site ;o)
dazz
DaveN - missed your post, you're too quick for me
Odd it would trigger some sort of email. Im wondering if other people are going to start getting these. I know it will be odd here since most people love the spiders and dont disallow em but post if it does happen to you ;)
If there's an automated mechanism, I'd guess it would be triggered by some level of external linkage to a site Google couldn't spider.
Couldn't that email from Google be considered unsolicited? After all, they're a commercial business and the email is unsolicited.
Spam and Spyware (toolbar)
My first thoughts exactly...
I know many people freak out at receiving a link request and liken it to spam.
What is the difference here?
Geees.I feel unloved.I'm taking this personally.
None of my banned domains ever got one.
hahahahahaha. Block it again and see what happens.
wanted it back after all that time needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again
Perhaps they want YOUR SITES under the radar, but they don't want YOU under the radar. ;-)
why would you not want to get indexed?
Couldn't that email from Google be considered unsolicited? After all, they're a commercial business and the email is unsolicited.
Is it not true that you want to be contacted by people who actually have interesting things to say to you? If a friend e-mails you asking you for something, is this unsolicited as well? I'd say while it may be "unsolicited", as most e-mails are (how often do you ask people to e-mail you as opposed to how many just e-mail you without asking?), it's certainly not unwelcome.
So when is unsolicited really unsolicited? When we want it? :)
I realize there are some flaws in what I've said. This post is meant to be a tounge in cheek sort of post. :-P
Oh, and I am by no means saying I agree with spam in anyway. I hate spam with about 4 passions.
Wouldn't that be like me telling you not to contact me and you show up on my doorstep anyway?
If I add my phone number to the "Do Not Call" list and a telemarketer calls me there's severe penalties involved.
why would you not want to get indexed?
What am I getting out of this relationship? Sure, I'm just a small pebble on the beach to Google, but I do have word of mouth and other SEs to promote my site. And at some point, someone is going to search Google for something and not find exactly what they wanted. They'll go to another SE, find my site, and wonder why it isn't in Google.
Google loses credibility, because that person has discovered that all the best sites aren't necessarily listed in Google, even if it is the biggest, it's not the best.
Just knowing my pages exist isn't good enough, Google has to be able to also list them when they apply to a search query. And right now, I've got quite a few unique pages with content that no one else has. And you can't find them in Google for any keyword that could possibly apply. Oh, they're in the Google index, Google has them, they just can't be found by searchers. But go to MSN, Yahoo, or any of the other "little" search engines, and there they are - right where they should be - in the #1 spot! They aren't in a terribly competitive field, so few people care to deal with this topic, which makes me believe they should be pretty relevant, since there isn't much else.
Andy
popthestack, in this case the website owner purposely blocked the bot from hitting their site. They told Googlebot to go away.Wouldn't that be like me telling you not to contact me and you show up on my doorstep anyway?
If I add my phone number to the "Do Not Call" list and a telemarketer calls me there's severe penalties involved.
Ah... I somehow overlooked that about the site actually blogging Google. Yes, that would be considered unsolicted e-mail. Hmmmm... still, if my reason for blocking was that they'd blocked me first I wouldn't mind. If I blocked them just to keep it away from them, I would consider it spam.
They spend the time on invites to be INCLUDED. What about the sites that are EXCLUDED for the actions of others? Google... what about us?
C
Search engines should PAY webmasters for the privilege of including their content in the index, yah? :-)
We have exactly the same experience currently our site doesnt feature anywhere in Google. Yet we have rich content. We feature well in MSN and Yahoo.
Googlebot hits our site every day and they cash the pages just dont index them. Im past caring now. May as well block the bot frankly its just taking up our bandwith.
In regards to this..
Is it not true that you want to be contacted by people who actually have interesting things to say to you?
Most people have some interest in pr0n, does that make it ok for those spammers to email you?
Would it be ok for 1000 smaller engines to email you? Assuming you said no, then isn't Google abusing their power by emailing you and getting away with it?
They perform 50% of searches because webmasters promoted them.. that doesn't give them some sort of right to email me when it's not right for others to do so!
May as well block the bot frankly its just taking up our bandwith.
Just banned my 6k+ pages of unique content from Googlebot - going to still allow others spiders.
I disallowed Google from a big chunk of my site, and noticed the MSN bot came by later that day, hit my robots.txt, left, came back and hit robots.txt again, then immediately went and gobbled up every page in that section! It didn't miss a single one! It's almost like MSN is going to make sure they have it listed, since Google can't get it now! Kinda funny in a way...SEs competing with each other. Maybe webmasters can start striking deals for exclusivity with a particular SE - they ban all the others, so that SE has EXCLUSIVE, ORIGINAL CONTENT!
It does seem like people are getting fed up - based on the posts here and on other boards, it seems a lot of people are having the same issue. It can't possibly all be bad sites being penalized, it has to be a problem at Google.
It does seem like Google would be heading off a potential PR nightmare if they'd just say they were still working on things, refining their filters, or whatever. And if they're done and happy with things, they should say so! Stand behind their work. At least then we'd know whether or not to rebuild our sites from scratch because of something we've done.
Andy
[edited by: AndyA at 3:23 am (utc) on Feb. 19, 2005]
I wonder how many other sites will begin to block Google in light of the most recent update
Not many - most people are too chicken-sheat.
At least then we'd know whether or not to rebuild our sites from scratch because of something we've done.
DO NOT build your sites from scratch. You've worked on your site, if google doesn't like it, then they can go a stick a whatever into their own G-Spot... Your work is your work.. If you look at the stats, Google traffic doesn't actually convert to $$$ as good as MSN or Yahoo, or even dogpile for that matter once you consider the traffic/income ratio per engine. So in a sense it's the highest proportion of wasted bandwidth, especially once you consider the spidering.
There's lots of other ways - Google looses out because I won't be using AdWords, closed a multi-hundred/day dollar account the other week.
