Forum Moderators: open

Message Too Old, No Replies

Robots.txt and Google Ban

         

markh8624

8:11 pm on Dec 27, 2002 (gmt 0)

10+ Year Member



Since I cannot find the old thread on my site's Google ban, I thought I would post this new thread (and see how long it stays in place?).

Mardi Gras - tried to find the robot.txt thread you mentioned in the last post before the thread I started was closed (and removed?). The link is 404.

I am still not sure why this would be the most important reason for a ban - if there is a file in our directory that is problematic it might be one file out of hundreds. And other search engines have no problem indexing our site.

Why would Google ban an educational site for an old page on a huge directory?

Thanks,

Mark

BigDave

7:16 pm on Dec 30, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That sure looks like it was the DNS problem. Google was going to your old IP address for 3 months. You might have even showed up a couple of days after the 2 visits as fresh content.

It could still conceivably be something else, but is sure seems like the DNS issue. Hopefully you will make it back in permanently at the end of January, and start picking up fresh pages through January.

NFFC

7:17 pm on Dec 30, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looks like BigDave laid it all out in message 24.

>the guy who follows our logging data

You mean the guy who doesn't, I find it almost unbelievable that today is the first time you have known about the prolonged absense of googlebot.

I would also urgently sort out the misconfigured 404/robots.txt issue.

markh8624

9:48 pm on Dec 30, 2002 (gmt 0)

10+ Year Member



NFFC - Thanks (?) for your message.

The 404/robots.txt issue has been resolved.

I still need information about the reason why the Googlebot would not have visited the site after the new site version was released - this was a revision and not a new site launch. I gather that whatever DNS problem exists lies within Google's operations?

That information *would* be helpful...

Mark

BigDave

9:58 pm on Dec 30, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It did visit your site. Your old site.

If you had left up the site on your old ISP it would have kept crawling that one till their DNS cache was updated.

Old ISP had yoursite.com at ip address 1.2.3.4
New ISP has yoursite.com address at 4.3.2.1

as soon as your site was up on 4.3.2.1, you removed everything from 1.2.3.4. But google kept looking for yoursite.com at 1.2.3.4 and there was nothing there.

If you had put up a new copy of your site at the old ISP along with your new ISP for a couple of months you would have been all set.

I am not positive about this, but I believe that putting a Redirect Permanent on the old site to the new site would have helped you out too.

NFFC

9:58 pm on Dec 30, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> I gather that whatever DNS problem exists lies within Google's operations?

Yes, they update their DNS stuff in their own time.

>I still need information about the reason why the Googlebot would not have visited the site

Imho you needed that information in September, I would put in place procedures to ensure that it will not happen again.

markh8624

10:07 pm on Dec 30, 2002 (gmt 0)

10+ Year Member



"If you had put up a new copy of your site at the old ISP along with your new ISP for a couple of months you would have been all set."

BigDave: This would not have been considered duplicate content that would have lead to S.E. problems?

Sounds like Google could do all of us a favor and let us know how long it takes them to update their DNS cache.

Thanks,

Mark

BigDave

10:28 pm on Dec 30, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No, it would not be duplicate content, because there is only one copy avalable to google at any given time. Before they update their DNS, they will only see the old copy. After they update their DNS, they will only see the new copy.

That is assuming that people are linking to you using your domain name, and not your ip addr.

And google will not penalize you for duplicate content, they will just throw out the least important copy. Since you own both copies, one of them will still be in the index.

Too many people sit around worrying about getting banned and getting penalized. Most of the posts on the subject are neither. Do things right, and don't play any tricks, and you should be fine.

markh8624

7:30 am on Dec 31, 2002 (gmt 0)

10+ Year Member



"Too many people sit around worrying about getting banned and getting penalized. Most of the posts on the subject are neither. Do things right, and don't play any tricks, and you should be fine."

BigD:

That *sounds* good but does not seem to always be true. I believe in our case we followed Google's webmaster guidelines, didn't play any tricks, and we are still trying to determine why our site is not appearing at *all* in Google some 4 months after the launch of our revised site.

Slow DNS updating within Google's system and/or some unknown 'content quality' issue seem to be our problem - but, still, there is no one who can tell us for sure what we did to not be included in Google's search results (I'm trying to avoid the word 'banned'). I believe it is this state of ambiguity that leads to the worry you see in this forum.

Perhaps you can be 'innocent' under the 'rules' and still end up penalized/banned?....

Just my .02

Mark

WebWalla

9:13 am on Dec 31, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Mark,

If Google is so important to your business - and it certainly seems to be, since your original thread mentioned laying people off - then why didn't you get somebody to at least read the Google guidlines for webmasters.
They give good infomation about robots.txt and 301 redirects. It seems to me that this simple information would have solved all your problems.

markh8624

4:44 pm on Jan 1, 2003 (gmt 0)

10+ Year Member



The guidelines *have* been read several times and have not resolved the problem.

Not only that, a half dozen SEO 'experts' have looked over the code, directories, etc., with no conclusive finding as to why we are excluded.

Any other *helpful* advice?

NFFC

4:58 pm on Jan 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Any other *helpful* advice?

When running an Internet based business try and have at least one person on staff that understands the basics of how the Internet functions.

If search engine traffic is crucial to the sucess of the venture employ an SEO.

WebWalla

5:04 pm on Jan 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You read the guidelines several times and yet didn't use a 301 redirect?

markh8624

5:26 pm on Jan 1, 2003 (gmt 0)

10+ Year Member



NFFC -

We *did* hire an SEO company over a year ago. Charged much $$$, no results. They went out of business without any notification. So much for SEO firms...

What I have found in the months I've dealt with this problem from a managerial perspective is that no one "knows the Internet" as you imply - not our webmaster, not SEO 'experts,' not even *you.*

Most of the posts here are guesses only, implying no certain base of knowledge. I know your ego may not deal well with this. I have received no conclusive answer to my questions here (by you or others). In spite of your posts. They are guesses that are negated elsewhere (like the Google DNS updating issue).

Another *helpful* response from a Kleinburgher on WebmasterWorld.

[edited by: markh8624 at 5:40 pm (utc) on Jan. 1, 2003]

markh8624

5:31 pm on Jan 1, 2003 (gmt 0)

10+ Year Member



WebWalla -

Tell me where you found *any* specific mention of 301 redirects on [google.com...] Redirects *are* mentioned as being problematic on guidelines, however (not really very helpful information). Maybe you should read this page again....

Yidaki

5:40 pm on Jan 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



markh8624, do a google search for google+webmasters+301 [google.com] and you'll find the answers to your 301 question. (No 3 of the webmasters guidelines.)

BTW: and try to be a bit more polite to the contributors here ... even if they are admins ;)

WebWalla

5:44 pm on Jan 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Mark - that isn't the only page to guide webmasters. The information I was referring to is here [google.com]. This page is referenced from the "Incorrect Listing" section of the help for webmasters. If you are relying so much on Google for your business, somebody in your company should have read all sections thoroughly.

BigDave

5:44 pm on Jan 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Mark,

I put a great deal of thought into your problem *for free*, as did many others. And lately, your messages seem to be getting rather insulting to a lot of us. No one here had anything to do with the cause of your problem, but we are (were) trying to help you out.

You are discounting my solution as being probably wrong, because apparently some unknown person, who is unable to come up with anything on their own, has said it was wrong. What is their argument against it?

There was also some helpful advice to fix up your 404 problem, that admittedly was not the original cause of getting dropped, but might have kept you from getting reindexed in January now that Googlebot has started crawling you again.

I suspect that the reason you are not getting any more *helpful* advice is that most of the people here consider your problem solved.

I hate being wrong, but if I am reading the tone of the last few of your messages correctly, then I actually hope that it was not the DNS update problem, and I regret helping you. If I am misreading your attitude, then I appologize for this statement.

This 47 message thread spans 2 pages: 47