Forum Moderators: open

Message Too Old, No Replies

Why won't google do a deep crawl?

Just grabs the index page

         

4string

1:17 pm on Aug 27, 2004 (gmt 0)

10+ Year Member



I have textlinks (no javascript or anything funny) for navagation. Other bots go through all my pages fine. My site finally started showing up again after getting a new URL and using a 301. I was sandboxed for a while. I went from #2 or 3 to 30 or 40 now. Nothing like doing loads of work and getting such a great reward. Anyway, Googlebot comes around everyday and grabs the index page, but that's it. How do I get it to do a deep crawl? The index page never changes. Is it looking for changes to it? Is it part 2 of the sandboxing effect?

isitreal

4:21 pm on Aug 28, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I recently did a similar thing, 301 to new domain name brand, same drop in serps, but googlebot spiders the site fully even if my serps are pretty much gone. Make sure your 301 is working correctly, triple check it.

Double check your logs to make sure google is only grabbing your index page, I haven't seen google do a full site spidering of a large site for a while, unlike yahoo and msn alpha, but grabbing only the index page is not a good sign from my experience.

If google is only grabbing your index page by next week I'd worry I think, if you have plenty of inbound links.

Also double check robots.txt etc to make sure you didn't make some syntax error or typo.

Check the programming of the site if there is any, try surfing your site with firefox using the 'user agent switcher' with the googlebot navigator user agent added to the switcher navuseragent list and see if the site works correctly.

jimbeetle

4:48 pm on Aug 28, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is it looking for changes to it?

Yes. Add new links, change some links, mix it up a bit and give the bot a reason to dig deeper. G-bot loves to roam, just have to put some spider bait out there for it to go after.

4string

12:53 am on Aug 29, 2004 (gmt 0)

10+ Year Member



Thanks. I will check out my logs more thoroughly. Seems funny. I don't know why Gbot would follow differently than other bots. It's not the robots file from what I can tell. I have very little disallowed right now.

dirkz

10:29 am on Aug 30, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Googlebot takes time to digest a 301. I wouldn't be surprised if some of them are still trying to crawl the former location for some time to come (have seen that before).

I would consider 1-4 weeks Google lag time for a 301 as normal.