homepage Welcome to WebmasterWorld Guest from 23.22.194.120
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Googlebot doesn't spider 1 particular site of mine...
designaweb




msg:90471
 12:10 pm on Sep 22, 2004 (gmt 0)

Hi peepz,

I run multiple websites, and all of them are listed in my "companies" portfolio. This portfolio get's spidered almost daily, so each new link I add will be visited by googlebot almost instantly. Then, after a few days, Googlebot will actually start indexing the subpages on this new domain as well.

However, I have one domain where this does not work for some reason. I have added it to my portfolio, waited for a while, but googlebot didn't even visit the new domain, eventhough it did visit my portfolio. Then I added a few links from other sites to this domain, but still a no-go.

Finally, I manually submitted the domain to google. And yes! Googlebot visited! However, it only hit 4 pages (where there are a lot more), and doesn't seem to be coming back... It has been quite a while now, so I am getting frustrated.

The only difference between this site/domain and the others I have launched, is the fact that this domain used to belong to another owner. It has been offline for quite a while as well. Then I took over and built a new site, which is not indexed like I wrote.

Any ideas on how this is caused? Any ideas on what I can do?

 

rfgdxm1




msg:90472
 11:09 pm on Sep 22, 2004 (gmt 0)

>The only difference between this site/domain and the others I have launched, is the fact that this domain used to belong to another owner. It has been offline for quite a while as well. Then I took over and built a new site, which is not indexed like I wrote.

Have you checked archive.org to see if the old site was somehow dodgy? It may still have a penalty.

dmz17




msg:90473
 11:39 pm on Sep 22, 2004 (gmt 0)

designaweb, we have a similar experience. The most recent site we launched more than 10 weeks ago still hasn't been spidered by gooblebot. All the other search engines have already spidered it and revisit it daily. It's a subdomain of our main site, and other subdomains we launched before were spidered by Google within 2 to 4 weeks.

cmendla




msg:90474
 1:43 am on Sep 23, 2004 (gmt 0)

I have a couple of sites on a particular host (not sure if I can name names). One site was doing well and is in the indexes. However, the other site went from about 100+ pages to 30 pages and traffic went to nothing.

I looked at a couple of things for the site that isn't doing anything. It is about 6 months old and as far as I know, I'm not doing anything ungooglelike.

I took a closer look at the august and sept logs and it appears that their is only one googlebot entry. (when I look at the logs from a site hosted elsewhere, I see hundreds of googlebot hits)

If I do a site:www.mysite.com -aasdasdas I see that only some of the listings are cached. The cache date is Aug 24. If I look at the log file, I would expect to have seen the googlebot in the august logs but it isn't there. If I look at July, I don't see googlebot there either (i do see the msnbot and a few others.)

I ran some robots.txt validators and robots.txt seems ok.

I tried the poodle predictor and that looks OK also.

I have a robots all,follow, index tag in all of the pages.

I did some google searches with my hostname to see if anyone else was complaining but didn't find anything.

Anyway, I'm a bit of a google novice. Here are a couple of questions

1. I'm assuming that not seeing the googlebot is really bad. Is this correct?

2. I tried a couple of sites where you could fake out the agent to see how the site reacts to the googlebot and I don't see any problems. Any suggests for sites to check my site out?

3. I'm not sure if it's within the TOS of this board to post a question about a specific host. I'd appreciate any thoughts on this.

I'd appreciate any ideas

Thanks

chris

jnmconsulting




msg:90475
 1:53 am on Sep 23, 2004 (gmt 0)

Chris, All,

My sites are usually visited by Google daily, however the last 4-5 weeks nothing. About an hour ago all 10 of my sites are in the process of a deep crawl. So it appears whatever slow down there was has been put into high gear.

designaweb




msg:90476
 11:44 am on Sep 23, 2004 (gmt 0)

Right, thanks for the reply's so far. I guess I will simply sit back and wait to see what happens over the next few weeks...

And yes, I did check archive.org - it seems to have been a mirror site for a site which currently is still online - could this cause anything to trigger google(bot)?

dirkz




msg:90477
 4:53 pm on Sep 23, 2004 (gmt 0)

> it seems to have been a mirror site for a site which currently is still online

This could indeed explain it. If it was dupe content it got neglected in favor for the higher PR site. If it was a 301, does Googlebot stop visiting them at all? Or just look every while?

How long have you been waiting?

designaweb




msg:90478
 6:54 pm on Sep 23, 2004 (gmt 0)

> How long have you been waiting?

Over a month now...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved