Welcome to WebmasterWorld Guest from 54.147.20.131

Forum Moderators: open

Message Too Old, No Replies

Googlebot doesn't spider 1 particular site of mine...

     

designaweb

12:10 pm on Sep 22, 2004 (gmt 0)

10+ Year Member



Hi peepz,

I run multiple websites, and all of them are listed in my "companies" portfolio. This portfolio get's spidered almost daily, so each new link I add will be visited by googlebot almost instantly. Then, after a few days, Googlebot will actually start indexing the subpages on this new domain as well.

However, I have one domain where this does not work for some reason. I have added it to my portfolio, waited for a while, but googlebot didn't even visit the new domain, eventhough it did visit my portfolio. Then I added a few links from other sites to this domain, but still a no-go.

Finally, I manually submitted the domain to google. And yes! Googlebot visited! However, it only hit 4 pages (where there are a lot more), and doesn't seem to be coming back... It has been quite a while now, so I am getting frustrated.

The only difference between this site/domain and the others I have launched, is the fact that this domain used to belong to another owner. It has been offline for quite a while as well. Then I took over and built a new site, which is not indexed like I wrote.

Any ideas on how this is caused? Any ideas on what I can do?

rfgdxm1

11:09 pm on Sep 22, 2004 (gmt 0)

WebmasterWorld Senior Member rfgdxm1 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>The only difference between this site/domain and the others I have launched, is the fact that this domain used to belong to another owner. It has been offline for quite a while as well. Then I took over and built a new site, which is not indexed like I wrote.

Have you checked archive.org to see if the old site was somehow dodgy? It may still have a penalty.

dmz17

11:39 pm on Sep 22, 2004 (gmt 0)

10+ Year Member



designaweb, we have a similar experience. The most recent site we launched more than 10 weeks ago still hasn't been spidered by gooblebot. All the other search engines have already spidered it and revisit it daily. It's a subdomain of our main site, and other subdomains we launched before were spidered by Google within 2 to 4 weeks.

cmendla

1:43 am on Sep 23, 2004 (gmt 0)

10+ Year Member



I have a couple of sites on a particular host (not sure if I can name names). One site was doing well and is in the indexes. However, the other site went from about 100+ pages to 30 pages and traffic went to nothing.

I looked at a couple of things for the site that isn't doing anything. It is about 6 months old and as far as I know, I'm not doing anything ungooglelike.

I took a closer look at the august and sept logs and it appears that their is only one googlebot entry. (when I look at the logs from a site hosted elsewhere, I see hundreds of googlebot hits)

If I do a site:www.mysite.com -aasdasdas I see that only some of the listings are cached. The cache date is Aug 24. If I look at the log file, I would expect to have seen the googlebot in the august logs but it isn't there. If I look at July, I don't see googlebot there either (i do see the msnbot and a few others.)

I ran some robots.txt validators and robots.txt seems ok.

I tried the poodle predictor and that looks OK also.

I have a robots all,follow, index tag in all of the pages.

I did some google searches with my hostname to see if anyone else was complaining but didn't find anything.

Anyway, I'm a bit of a google novice. Here are a couple of questions

1. I'm assuming that not seeing the googlebot is really bad. Is this correct?

2. I tried a couple of sites where you could fake out the agent to see how the site reacts to the googlebot and I don't see any problems. Any suggests for sites to check my site out?

3. I'm not sure if it's within the TOS of this board to post a question about a specific host. I'd appreciate any thoughts on this.

I'd appreciate any ideas

Thanks

chris

jnmconsulting

1:53 am on Sep 23, 2004 (gmt 0)

10+ Year Member



Chris, All,

My sites are usually visited by Google daily, however the last 4-5 weeks nothing. About an hour ago all 10 of my sites are in the process of a deep crawl. So it appears whatever slow down there was has been put into high gear.

designaweb

11:44 am on Sep 23, 2004 (gmt 0)

10+ Year Member



Right, thanks for the reply's so far. I guess I will simply sit back and wait to see what happens over the next few weeks...

And yes, I did check archive.org - it seems to have been a mirror site for a site which currently is still online - could this cause anything to trigger google(bot)?

dirkz

4:53 pm on Sep 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> it seems to have been a mirror site for a site which currently is still online

This could indeed explain it. If it was dupe content it got neglected in favor for the higher PR site. If it was a 301, does Googlebot stop visiting them at all? Or just look every while?

How long have you been waiting?

designaweb

6:54 pm on Sep 23, 2004 (gmt 0)

10+ Year Member



> How long have you been waiting?

Over a month now...

 

Featured Threads

Hot Threads This Week

Hot Threads This Month