Welcome to WebmasterWorld Guest from 54.163.25.166

Message Too Old, No Replies

Why googlebot keep visiting banned sites?

I couldn't find an answer myself maybe you can

     
7:57 am on Aug 14, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 2003
posts:427
votes: 0


I have a few sites that got nuked from google recently.

Tracking the sites logs shows repetetive visits from some or one of googlebots, real googlebots not just faked user-agent ones. Of course when these sites were unbanned they were getting much more visits from all or most googlebots. Questions is, since these sites were banned, why do googlebots still visiting them?

8:00 pm on Aug 15, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


One reason could be they are trying to figure out if you are smart enough to know what upset them and if you have corrected the problem. Remember not all penalties are the same.
8:04 pm on Aug 15, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


"why do googlebots still visiting them"

To see if they still suck (in googlebot's opinion)?

7:24 pm on Aug 17, 2005 (gmt 0)

New User

10+ Year Member

joined:Dec 11, 2003
posts:29
votes: 0


Googlebot visits regularly my banned domain and tries to open pages I have removed 6 months ago. Googlebot is like a dumb machine and it is not able to act like a human and review sites to see if some tricks have been removed. (Unless it is programmed to do so)
8:23 pm on Aug 17, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 9, 2003
posts:735
votes: 0


Why exactly do they say they've forwarded your problem to the engineers if they know your site is banned and why it's banned?
10:40 pm on Aug 21, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 2003
posts:427
votes: 0


Googlebot is like a dumb machine and it is not able to act like a human and review sites to see if some tricks have been removed.

Actually it's more than a dumb machine.

It has been ten days so far since I modified my robots.txt to "force" googlebot not to spider what I "thought" google might consider as a duplicated content, but googlebots insists on spidering the disallowed contents.

My logs show that every particular bot downloads robots.txt then ironically spiders the pages, as if I asked him to spider them not the opposite!

To see if they still suck (in googlebot's opinion)?

Correct a mondo, and explains why does it insist on spidering pages that I told it not to spider.

Why exactly do they say they've forwarded your problem to the engineers if they know your site is banned and why it's banned?

I dont seem to understand the question.

9:01 am on Aug 22, 2005 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts:297
votes: 12


I donít think the bot would have time to check to see if its banned or not, spiders have to be quick and cant afford to get bogged down. Even if I am wrong its got something to do with a technical issue.
1:36 pm on Aug 22, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 13, 2003
posts:335
votes: 0


Obviously, the algorithm that determines what gets spidered is different from the algorith that determines what gets displayed in the SERPs.

Furthemore, many of the bans are based on algorithms which Google is always tweaking, so Google doesn't really want to totally lose the banned pages, it just doesn't want to display them at the moment.

8:52 pm on Aug 23, 2005 (gmt 0)

New User

10+ Year Member

joined:June 25, 2002
posts:12
votes: 0


I have been nuked too. Since 18th July my site has been out of the google serps. Strange things I have noticed:

I have many pages cached in google including the index page.
Googlebot is coming to the site every day (did 980 hits yesterday)

I had problem with code which has cleaned up mistaken dup content and emailed G, no reply so far..

So in anyones opinion am I banned or is it a question of waiting to get reindexed?

I am so confused, many people have different theories, just dont know who to believe!

Any ideas would be appreciated!

11:03 am on Aug 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 9, 2003
posts:735
votes: 0


I finally got an email back from Google yesterday that strongly suggested I check out the webmaster guidelines. I did, made changes to the only things I can figure were wrong, and replied to the email. I told them what I changed and how I interpreted their guidelines. I also kissed their hind quarters a bit. Within an hour I had a reply that said once again, everything was being forwarded to the engineers for review. The whole thing has the feeling of talking to a human ( a cut and paster) but it does seem they have people reviewing the sites.