Welcome to WebmasterWorld Guest from 54.197.94.141

Message Too Old, No Replies

Inclusion in the Google Index

E-Mail sent to me from Google.

   
6:46 pm on Feb 16, 2005 (gmt 0)

WebmasterWorld Senior Member eliteweb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Greetings WebmasterWorld :)

Figured I'd make a post here since I haven't in a while. I recieved an email from someone at google telling me the benifits of allowing the google spider to index my site. The site in question has a robots.txt file disallowing all search engines.

The email includes the URL in question and states that atleast 50% of internet searches come from google and its important to allow google to spider the site.

First reaction was, automated responce? Did I send an inquiree to them? Nopes certainly not for this domain so I sent an email back asking why the email and I expected it to bounce. To my surprise this morning I got a reply back saying they want to include the site because it was identified as being highly ranked.

New sort of team at google trying to get people uninformed and disallowing google to switch back and allow google to spider the data?

7:07 pm on Feb 16, 2005 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It's good to see you here, EliteWeb.

I've heard of these emails before but they are not common, so I guess kudos is in order. :-)

7:11 pm on Feb 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



eliteweb not sure what triggers it but I think it's automated, I got one for a site they banned two years ago and as i always do i ban the bot when it gets kicked, funny in away.. that they wanted it back after all that time needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again

DaveN

7:14 pm on Feb 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



congrats EliteWeb!

very interesting, i hadn't heard of these emails until now.

ciml - i would have put money on you knowing about them ;o)

must be some sort of site ;o)

dazz

DaveN - missed your post, you're too quick for me

7:58 pm on Feb 16, 2005 (gmt 0)

WebmasterWorld Senior Member eliteweb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



The site in question wouldn't be an important site for content I wouldn't think. It has a PR6. Most all the links to it are from our main site and nothing much else. It has one or two pages of indexable data...

Odd it would trigger some sort of email. Im wondering if other people are going to start getting these. I know it will be odd here since most people love the spiders and dont disallow em but post if it does happen to you ;)

1:33 pm on Feb 17, 2005 (gmt 0)

10+ Year Member



This is new to me. But this certainly tells how concerned the engineers at Google are about making important pages rank high. Good to know!
8:02 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Great story, DaveN. I guess it confirms that Google has a sense of humor. :)

If there's an automated mechanism, I'd guess it would be triggered by some level of external linkage to a site Google couldn't spider.

8:11 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That's an interesting email. I guess they're trying to get more pages in their index to increase the overall count.

Couldn't that email from Google be considered unsolicited? After all, they're a commercial business and the email is unsolicited.

8:28 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



Spam and Spyware (toolbar), Google is becoming a respected company.
8:42 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



Spam and Spyware (toolbar)

My first thoughts exactly...

I know many people freak out at receiving a link request and liken it to spam.

What is the difference here?

9:05 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I can understand it from Google's point of view. They want to continue being the dominant SE, so they need to cover as much of the net as possible. A Robots.txt file with a disallow prevents them from adding some sites to their database.
9:06 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>...I got one for a site they banned two years ago and as i always do i ban the bot when it gets kicked..<<<

Geees.I feel unloved.I'm taking this personally.
None of my banned domains ever got one.

9:16 pm on Feb 18, 2005 (gmt 0)



"..needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again"

hahahahahaha. Block it again and see what happens.

9:18 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member caveman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



wanted it back after all that time needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again

LOL.

Perhaps they want YOUR SITES under the radar, but they don't want YOU under the radar. ;-)

9:36 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



I'm just curious and excuse my ignorance, but why would you not want to get indexed?
10:00 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



why would you not want to get indexed?

There are many many reasons:
- closed message boards only open to certain members
- internal websites like intranets
- company extranets
- sites that have been banned in Google (if they ban me I'll ban them)
- sites that rank really really well in MSN and Yahoo! because of the optimization techniques being used (think black hat) but won't rank in Google
10:04 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



Couldn't that email from Google be considered unsolicited? After all, they're a commercial business and the email is unsolicited.

Is it not true that you want to be contacted by people who actually have interesting things to say to you? If a friend e-mails you asking you for something, is this unsolicited as well? I'd say while it may be "unsolicited", as most e-mails are (how often do you ask people to e-mail you as opposed to how many just e-mail you without asking?), it's certainly not unwelcome.

So when is unsolicited really unsolicited? When we want it? :)

I realize there are some flaws in what I've said. This post is meant to be a tounge in cheek sort of post. :-P

Oh, and I am by no means saying I agree with spam in anyway. I hate spam with about 4 passions.

10:11 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



popthestack, in this case the website owner purposely blocked the bot from hitting their site. They told Googlebot to go away.

Wouldn't that be like me telling you not to contact me and you show up on my doorstep anyway?

If I add my phone number to the "Do Not Call" list and a telemarketer calls me there's severe penalties involved.

10:14 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



why would you not want to get indexed?

Why should I allow Google to count all of my pages as part of their index, and include them in their total pages indexed on THEIR site, when doing relevant searches won't list any of my pages?

What am I getting out of this relationship? Sure, I'm just a small pebble on the beach to Google, but I do have word of mouth and other SEs to promote my site. And at some point, someone is going to search Google for something and not find exactly what they wanted. They'll go to another SE, find my site, and wonder why it isn't in Google.

Google loses credibility, because that person has discovered that all the best sites aren't necessarily listed in Google, even if it is the biggest, it's not the best.

Just knowing my pages exist isn't good enough, Google has to be able to also list them when they apply to a search query. And right now, I've got quite a few unique pages with content that no one else has. And you can't find them in Google for any keyword that could possibly apply. Oh, they're in the Google index, Google has them, they just can't be found by searchers. But go to MSN, Yahoo, or any of the other "little" search engines, and there they are - right where they should be - in the #1 spot! They aren't in a terribly competitive field, so few people care to deal with this topic, which makes me believe they should be pretty relevant, since there isn't much else.

Andy

10:29 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



popthestack, in this case the website owner purposely blocked the bot from hitting their site. They told Googlebot to go away.

Wouldn't that be like me telling you not to contact me and you show up on my doorstep anyway?

If I add my phone number to the "Do Not Call" list and a telemarketer calls me there's severe penalties involved.

Ah... I somehow overlooked that about the site actually blogging Google. Yes, that would be considered unsolicted e-mail. Hmmmm... still, if my reason for blocking was that they'd blocked me first I wouldn't mind. If I blocked them just to keep it away from them, I would consider it spam.

10:51 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



I have received it recently (for a site that I have no interest in allowing Google to spider), it's automated spam (using whois info at times) that includes removal instructions (from their mailings) to comply with can-spam laws.
11:08 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member crobb305 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



They are sending invites to be included, yet they have many many great sites that are EXCLUDED because of duplication penalties, despite being the ORIGINAL author. Sites also EXCLUDED because of hijacking, 302s, etc. Ridiculous.

They spend the time on invites to be INCLUDED. What about the sites that are EXCLUDED for the actions of others? Google... what about us?

C

11:17 pm on Feb 18, 2005 (gmt 0)

10+ Year Member



Opposite thinking...

Search engines should PAY webmasters for the privilege of including their content in the index, yah? :-)

11:27 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I think mother google should learn to take care of her current shoe-full of sites before adopting new ones.
11:36 pm on Feb 18, 2005 (gmt 0)

WebmasterWorld Senior Member eliteweb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It did indeed seem automated because of the removal request but it did also come from a persons email at google rather than support or some generic address.
12:08 am on Feb 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



AndyA

We have exactly the same experience currently our site doesnt feature anywhere in Google. Yet we have rich content. We feature well in MSN and Yahoo.

Googlebot hits our site every day and they cash the pages just dont index them. Im past caring now. May as well block the bot frankly its just taking up our bandwith.

3:00 am on Feb 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just banned my 6k+ pages of unique content from Googlebot - going to still allow others spiders.

In regards to this..

Is it not true that you want to be contacted by people who actually have interesting things to say to you?

Most people have some interest in pr0n, does that make it ok for those spammers to email you?

Would it be ok for 1000 smaller engines to email you? Assuming you said no, then isn't Google abusing their power by emailing you and getting away with it?

They perform 50% of searches because webmasters promoted them.. that doesn't give them some sort of right to email me when it's not right for others to do so!

May as well block the bot frankly its just taking up our bandwith.

Join the Gang! Whatever little part I can play in taking away Googles power, I'm in!
3:18 am on Feb 19, 2005 (gmt 0)

10+ Year Member



I wonder how many other sites will begin to block Google in light of the most recent update. If anything, large quantities of quality sites refusing Google would eventually skew the results and make users reconsider which search engine they use. Hopefully those how are up in arms will actually put their money where there mouth is and not bother with Google if they are consistantly being penalized for questionable reasons. It's one of the few ways we can effect change in the land of search.
3:20 am on Feb 19, 2005 (gmt 0)

10+ Year Member



Just banned my 6k+ pages of unique content from Googlebot - going to still allow others spiders.

Hmmm...I just got two E-mails from friends with websites telling me they had also just banned Googlebot! One told me he didn't use robots.txt, he went straight to htaccess and banned Google by IPs! He really sounded like he was fed up!

I disallowed Google from a big chunk of my site, and noticed the MSN bot came by later that day, hit my robots.txt, left, came back and hit robots.txt again, then immediately went and gobbled up every page in that section! It didn't miss a single one! It's almost like MSN is going to make sure they have it listed, since Google can't get it now! Kinda funny in a way...SEs competing with each other. Maybe webmasters can start striking deals for exclusivity with a particular SE - they ban all the others, so that SE has EXCLUSIVE, ORIGINAL CONTENT!

It does seem like people are getting fed up - based on the posts here and on other boards, it seems a lot of people are having the same issue. It can't possibly all be bad sites being penalized, it has to be a problem at Google.

It does seem like Google would be heading off a potential PR nightmare if they'd just say they were still working on things, refining their filters, or whatever. And if they're done and happy with things, they should say so! Stand behind their work. At least then we'd know whether or not to rebuild our sites from scratch because of something we've done.

Andy

[edited by: AndyA at 3:23 am (utc) on Feb. 19, 2005]

3:22 am on Feb 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I wonder how many other sites will begin to block Google in light of the most recent update

Not many - most people are too chicken-sheat.

At least then we'd know whether or not to rebuild our sites from scratch because of something we've done.

DO NOT build your sites from scratch. You've worked on your site, if google doesn't like it, then they can go a stick a whatever into their own G-Spot... Your work is your work.. If you look at the stats, Google traffic doesn't actually convert to $$$ as good as MSN or Yahoo, or even dogpile for that matter once you consider the traffic/income ratio per engine. So in a sense it's the highest proportion of wasted bandwidth, especially once you consider the spidering.

There's lots of other ways - Google looses out because I won't be using AdWords, closed a multi-hundred/day dollar account the other week.

This 85 message thread spans 3 pages: 85
 

Featured Threads

Hot Threads This Week

Hot Threads This Month