homepage Welcome to WebmasterWorld Guest from 54.226.10.234
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 85 message thread spans 3 pages: 85 ( [1] 2 3 > >     
Inclusion in the Google Index
E-Mail sent to me from Google.
EliteWeb

WebmasterWorld Senior Member eliteweb us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 6:46 pm on Feb 16, 2005 (gmt 0)

Greetings WebmasterWorld :)

Figured I'd make a post here since I haven't in a while. I recieved an email from someone at google telling me the benifits of allowing the google spider to index my site. The site in question has a robots.txt file disallowing all search engines.

The email includes the URL in question and states that atleast 50% of internet searches come from google and its important to allow google to spider the site.

First reaction was, automated responce? Did I send an inquiree to them? Nopes certainly not for this domain so I sent an email back asking why the email and I expected it to bounce. To my surprise this morning I got a reply back saying they want to include the site because it was identified as being highly ranked.

New sort of team at google trying to get people uninformed and disallowing google to switch back and allow google to spider the data?

 

ciml

WebmasterWorld Senior Member ciml us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 7:07 pm on Feb 16, 2005 (gmt 0)

It's good to see you here, EliteWeb.

I've heard of these emails before but they are not common, so I guess kudos is in order. :-)

DaveN

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 7:11 pm on Feb 16, 2005 (gmt 0)

eliteweb not sure what triggers it but I think it's automated, I got one for a site they banned two years ago and as i always do i ban the bot when it gets kicked, funny in away.. that they wanted it back after all that time needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again

DaveN

diddlydazz

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 7:14 pm on Feb 16, 2005 (gmt 0)

congrats EliteWeb!

very interesting, i hadn't heard of these emails until now.

ciml - i would have put money on you knowing about them ;o)

must be some sort of site ;o)

dazz

DaveN - missed your post, you're too quick for me

EliteWeb

WebmasterWorld Senior Member eliteweb us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 7:58 pm on Feb 16, 2005 (gmt 0)

The site in question wouldn't be an important site for content I wouldn't think. It has a PR6. Most all the links to it are from our main site and nothing much else. It has one or two pages of indexable data...

Odd it would trigger some sort of email. Im wondering if other people are going to start getting these. I know it will be odd here since most people love the spiders and dont disallow em but post if it does happen to you ;)

webnewton

10+ Year Member



 
Msg#: 28142 posted 1:33 pm on Feb 17, 2005 (gmt 0)

This is new to me. But this certainly tells how concerned the engineers at Google are about making important pages rank high. Good to know!

rogerd

WebmasterWorld Administrator rogerd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 8:02 pm on Feb 18, 2005 (gmt 0)

Great story, DaveN. I guess it confirms that Google has a sense of humor. :)

If there's an automated mechanism, I'd guess it would be triggered by some level of external linkage to a site Google couldn't spider.

bhartzer

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 8:11 pm on Feb 18, 2005 (gmt 0)

That's an interesting email. I guess they're trying to get more pages in their index to increase the overall count.

Couldn't that email from Google be considered unsolicited? After all, they're a commercial business and the email is unsolicited.

bears5122

10+ Year Member



 
Msg#: 28142 posted 8:28 pm on Feb 18, 2005 (gmt 0)

Spam and Spyware (toolbar), Google is becoming a respected company.

chicagohh

10+ Year Member



 
Msg#: 28142 posted 8:42 pm on Feb 18, 2005 (gmt 0)

Spam and Spyware (toolbar)

My first thoughts exactly...

I know many people freak out at receiving a link request and liken it to spam.

What is the difference here?

Rugles

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 9:05 pm on Feb 18, 2005 (gmt 0)

I can understand it from Google's point of view. They want to continue being the dominant SE, so they need to cover as much of the net as possible. A Robots.txt file with a disallow prevents them from adding some sites to their database.

cabbie

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 9:06 pm on Feb 18, 2005 (gmt 0)

>>...I got one for a site they banned two years ago and as i always do i ban the bot when it gets kicked..<<<

Geees.I feel unloved.I'm taking this personally.
None of my banned domains ever got one.

walkman



 
Msg#: 28142 posted 9:16 pm on Feb 18, 2005 (gmt 0)

"..needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again"

hahahahahaha. Block it again and see what happens.

caveman

WebmasterWorld Senior Member caveman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 9:18 pm on Feb 18, 2005 (gmt 0)

wanted it back after all that time needless to say i removed to line from the robots.txt and it came back to life and then was banned 3 months later again

LOL.

Perhaps they want YOUR SITES under the radar, but they don't want YOU under the radar. ;-)

mikko123

5+ Year Member



 
Msg#: 28142 posted 9:36 pm on Feb 18, 2005 (gmt 0)

I'm just curious and excuse my ignorance, but why would you not want to get indexed?

bhartzer

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 10:00 pm on Feb 18, 2005 (gmt 0)

why would you not want to get indexed?

There are many many reasons:
- closed message boards only open to certain members
- internal websites like intranets
- company extranets
- sites that have been banned in Google (if they ban me I'll ban them)
- sites that rank really really well in MSN and Yahoo! because of the optimization techniques being used (think black hat) but won't rank in Google

popthestack

5+ Year Member



 
Msg#: 28142 posted 10:04 pm on Feb 18, 2005 (gmt 0)

Couldn't that email from Google be considered unsolicited? After all, they're a commercial business and the email is unsolicited.

Is it not true that you want to be contacted by people who actually have interesting things to say to you? If a friend e-mails you asking you for something, is this unsolicited as well? I'd say while it may be "unsolicited", as most e-mails are (how often do you ask people to e-mail you as opposed to how many just e-mail you without asking?), it's certainly not unwelcome.

So when is unsolicited really unsolicited? When we want it? :)

I realize there are some flaws in what I've said. This post is meant to be a tounge in cheek sort of post. :-P

Oh, and I am by no means saying I agree with spam in anyway. I hate spam with about 4 passions.

bhartzer

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 10:11 pm on Feb 18, 2005 (gmt 0)

popthestack, in this case the website owner purposely blocked the bot from hitting their site. They told Googlebot to go away.

Wouldn't that be like me telling you not to contact me and you show up on my doorstep anyway?

If I add my phone number to the "Do Not Call" list and a telemarketer calls me there's severe penalties involved.

AndyA

5+ Year Member



 
Msg#: 28142 posted 10:14 pm on Feb 18, 2005 (gmt 0)

why would you not want to get indexed?

Why should I allow Google to count all of my pages as part of their index, and include them in their total pages indexed on THEIR site, when doing relevant searches won't list any of my pages?

What am I getting out of this relationship? Sure, I'm just a small pebble on the beach to Google, but I do have word of mouth and other SEs to promote my site. And at some point, someone is going to search Google for something and not find exactly what they wanted. They'll go to another SE, find my site, and wonder why it isn't in Google.

Google loses credibility, because that person has discovered that all the best sites aren't necessarily listed in Google, even if it is the biggest, it's not the best.

Just knowing my pages exist isn't good enough, Google has to be able to also list them when they apply to a search query. And right now, I've got quite a few unique pages with content that no one else has. And you can't find them in Google for any keyword that could possibly apply. Oh, they're in the Google index, Google has them, they just can't be found by searchers. But go to MSN, Yahoo, or any of the other "little" search engines, and there they are - right where they should be - in the #1 spot! They aren't in a terribly competitive field, so few people care to deal with this topic, which makes me believe they should be pretty relevant, since there isn't much else.

Andy

popthestack

5+ Year Member



 
Msg#: 28142 posted 10:29 pm on Feb 18, 2005 (gmt 0)

popthestack, in this case the website owner purposely blocked the bot from hitting their site. They told Googlebot to go away.

Wouldn't that be like me telling you not to contact me and you show up on my doorstep anyway?

If I add my phone number to the "Do Not Call" list and a telemarketer calls me there's severe penalties involved.

Ah... I somehow overlooked that about the site actually blogging Google. Yes, that would be considered unsolicted e-mail. Hmmmm... still, if my reason for blocking was that they'd blocked me first I wouldn't mind. If I blocked them just to keep it away from them, I would consider it spam.

Craven de Kere

10+ Year Member



 
Msg#: 28142 posted 10:51 pm on Feb 18, 2005 (gmt 0)

I have received it recently (for a site that I have no interest in allowing Google to spider), it's automated spam (using whois info at times) that includes removal instructions (from their mailings) to comply with can-spam laws.

crobb305

WebmasterWorld Senior Member crobb305 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 11:08 pm on Feb 18, 2005 (gmt 0)

They are sending invites to be included, yet they have many many great sites that are EXCLUDED because of duplication penalties, despite being the ORIGINAL author. Sites also EXCLUDED because of hijacking, 302s, etc. Ridiculous.

They spend the time on invites to be INCLUDED. What about the sites that are EXCLUDED for the actions of others? Google... what about us?

C

siteseo

10+ Year Member



 
Msg#: 28142 posted 11:17 pm on Feb 18, 2005 (gmt 0)

Opposite thinking...

Search engines should PAY webmasters for the privilege of including their content in the index, yah? :-)

sailorjwd

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 11:27 pm on Feb 18, 2005 (gmt 0)

I think mother google should learn to take care of her current shoe-full of sites before adopting new ones.

EliteWeb

WebmasterWorld Senior Member eliteweb us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 28142 posted 11:36 pm on Feb 18, 2005 (gmt 0)

It did indeed seem automated because of the removal request but it did also come from a persons email at google rather than support or some generic address.

RichTC

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 12:08 am on Feb 19, 2005 (gmt 0)

AndyA

We have exactly the same experience currently our site doesnt feature anywhere in Google. Yet we have rich content. We feature well in MSN and Yahoo.

Googlebot hits our site every day and they cash the pages just dont index them. Im past caring now. May as well block the bot frankly its just taking up our bandwith.

Chico_Loco

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 3:00 am on Feb 19, 2005 (gmt 0)

Just banned my 6k+ pages of unique content from Googlebot - going to still allow others spiders.

In regards to this..
Is it not true that you want to be contacted by people who actually have interesting things to say to you?

Most people have some interest in pr0n, does that make it ok for those spammers to email you?

Would it be ok for 1000 smaller engines to email you? Assuming you said no, then isn't Google abusing their power by emailing you and getting away with it?

They perform 50% of searches because webmasters promoted them.. that doesn't give them some sort of right to email me when it's not right for others to do so!

May as well block the bot frankly its just taking up our bandwith.

Join the Gang! Whatever little part I can play in taking away Googles power, I'm in!

Mr_N

10+ Year Member



 
Msg#: 28142 posted 3:18 am on Feb 19, 2005 (gmt 0)

I wonder how many other sites will begin to block Google in light of the most recent update. If anything, large quantities of quality sites refusing Google would eventually skew the results and make users reconsider which search engine they use. Hopefully those how are up in arms will actually put their money where there mouth is and not bother with Google if they are consistantly being penalized for questionable reasons. It's one of the few ways we can effect change in the land of search.

AndyA

5+ Year Member



 
Msg#: 28142 posted 3:20 am on Feb 19, 2005 (gmt 0)

Just banned my 6k+ pages of unique content from Googlebot - going to still allow others spiders.

Hmmm...I just got two E-mails from friends with websites telling me they had also just banned Googlebot! One told me he didn't use robots.txt, he went straight to htaccess and banned Google by IPs! He really sounded like he was fed up!

I disallowed Google from a big chunk of my site, and noticed the MSN bot came by later that day, hit my robots.txt, left, came back and hit robots.txt again, then immediately went and gobbled up every page in that section! It didn't miss a single one! It's almost like MSN is going to make sure they have it listed, since Google can't get it now! Kinda funny in a way...SEs competing with each other. Maybe webmasters can start striking deals for exclusivity with a particular SE - they ban all the others, so that SE has EXCLUSIVE, ORIGINAL CONTENT!

It does seem like people are getting fed up - based on the posts here and on other boards, it seems a lot of people are having the same issue. It can't possibly all be bad sites being penalized, it has to be a problem at Google.

It does seem like Google would be heading off a potential PR nightmare if they'd just say they were still working on things, refining their filters, or whatever. And if they're done and happy with things, they should say so! Stand behind their work. At least then we'd know whether or not to rebuild our sites from scratch because of something we've done.

Andy

[edited by: AndyA at 3:23 am (utc) on Feb. 19, 2005]

Chico_Loco

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 28142 posted 3:22 am on Feb 19, 2005 (gmt 0)

I wonder how many other sites will begin to block Google in light of the most recent update

Not many - most people are too chicken-sheat.

At least then we'd know whether or not to rebuild our sites from scratch because of something we've done.

DO NOT build your sites from scratch. You've worked on your site, if google doesn't like it, then they can go a stick a whatever into their own G-Spot... Your work is your work.. If you look at the stats, Google traffic doesn't actually convert to $$$ as good as MSN or Yahoo, or even dogpile for that matter once you consider the traffic/income ratio per engine. So in a sense it's the highest proportion of wasted bandwidth, especially once you consider the spidering.

There's lots of other ways - Google looses out because I won't be using AdWords, closed a multi-hundred/day dollar account the other week.

This 85 message thread spans 3 pages: 85 ( [1] 2 3 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved