Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Current Google spidering pattern: how long is it taking?



2:40 pm on Jul 29, 2003 (gmt 0)

10+ Year Member

Hey y'all

Google has listed my index.html but I wanted to know how long it will be before the rest of the site is index. The date in the description of my site in google shows as July 6th, does it normally take this long for google to come back or is it possible it just doesnt loke my site?

Thanking you


1:18 am on Jul 30, 2003 (gmt 0)

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

It can take a couple of months. It's not unusual for the homepage to be grabbed first and then other pages picked up later on.

Get good inbound links with decent PR from quality sites and have patience. It's not a problem, it's just the way it works.

What's bothering me is that I'm working on a new site - slowly and gradually - that is NOT linked to from anyplace whatsoever and Google got the whole thing right off it seems. I have no idea how or when since there are no stats available with that hosting - but there those pages sit, right in the index and two are ranking not bad at all considering it's all under construction.

There's no set time or rule that we can know for certain - but it often does take a couple of months to get more than just the index page in.


1:26 am on Jul 30, 2003 (gmt 0)

10+ Year Member

There are those who think that visiting your new pages with the toolbar can get them in the index without any external links to the site. I haven't a clue whether this is true. Brett implied yes in a previous post but GG thinks not.


12:28 am on Jul 30, 2003 (gmt 0)

10+ Year Member

I started a new site in very early June and it had visits from quite a few bots about 2 weeks later. Google came to my site and only hit the index page. Same thing with Slurp. To this day, I haven't had the bots back to my site.

AltaVista Scooter came and indexed my whole site and I am listed there but that's it.

I don't have any text hidden, the site is very straight forward but they haven't been back. Maybe my HTML stinks or google is just afraid of me.

My other website gets hit almost everyday by google and I have a very good ranking but still nothing.

Can anyone tell me what I am doing wrong?

Thanks for anyone who can help.


2:21 am on Jul 30, 2003 (gmt 0)

10+ Year Member

Please forgive my naivety on the subject… I have created a few small personal websites; my current project is the first real big go at a large audience website.

I put up the first 20 pages of the site in late may, and have added 100 since… 60 in late June… 40 this past week. I will continue to add pages in blocks of 20-60 at a time as I am working on a regional website, and it will take some time to cover the area.

My question is… Is it normal to get 4 weeks of nearly daily googlebot visits followed by, what is now a week of none? I should not say any necessarily… I am still getting a robots and index.htm request every other day, but nothing more.

I suppose this is normal for a new site with relatively few back links as of yet… but I have been wondering, and wanted to get some more input on what to expect (even though it sounds like I should expect the unexpected with google).

I recently also added a basic sitemap to the new site from an old site, which I will be closing soon. The old site is currently linking to the new site, and to each subsection (towns within the area I am creating the site for). Is this not a wise idea? Will googlebot penalize me for doing this?


2:33 am on Jul 30, 2003 (gmt 0)

10+ Year Member

Google has listed my index.html ...

So, get external links to your other pages and speed up it.


2:37 am on Jul 30, 2003 (gmt 0)

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Radiolabs, it seems like this is the normal pattern, it's just seems to be taking time in most cases - some just more than others and there can be a few reasons for that.

nwctwx, it's hard to tell with some things with any degree of certainty, but if you're not linking back to the old site you're most likely OK. How is the PR for the old site, and are there a lot of decent links pointing to it?

How about the new site? Any good links to it besides from your old sites? Do your older sites link to each other?


3:12 am on Jul 30, 2003 (gmt 0)

10+ Year Member


The old site has a PR of 3, it never was a huge draw, with a very select local market. There are 10 links pointing to the old site (mostly PR 2-4).

The new site has a few good links, 2 from local libraries, and 3 from local directories. The site is nowhere near complete, so I am being very selective on who I ask for links from at this point. The new site does not link to the old site, but links to a page which has links to the 5 sites I have created in the past year (including 2 from this past month I made for customers, excluding the old site which will be dead in September).

My "market" is fairly untouched by the internet and I already was seeing very good SERPS when the index (county) and each "sub index" (towns) were in the results.

Last Wednesday freshbot did the last major visit (grabbing the index and each main town), since then it has been sparse. I added a site map this past weekend, and am now just waiting for googlebot to go back to it's previous avtivities on the site...

I guess I just need to wait a bit longer and not be impatient ;)


5:27 pm on Jul 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Today Googlebot visited my new website for the first time. I am wondering:

a) How long is it going to take for Google to get at least few pages off my site and have it in the index?

b) How long is it going to take to spider the whole website (over 20,000 dynamic pages)?

c) How to make Google spider the most relevant content first? (I've got several articles related to the content, and want Google to get them first).



12:40 pm on Jul 30, 2003 (gmt 0)

10+ Year Member

Did it just visit robots.txt or has it started to grab pages?
If pages were grabbed, you might get lucky and have them in the index within a few days.
If you want googlebot to visit certain pages first, there is no guarantee but you might want to get external links to those pages and also make sure they are not buried deep down from the root directory.


2:19 pm on Jul 30, 2003 (gmt 0)

10+ Year Member

Guess I was being overanxious...

Googlebot hit the site last evening and gobbled up the new site map :)


2:12 pm on Aug 4, 2003 (gmt 0)

10+ Year Member

Hey guys thanks for all the replies!

Google actually came by and grabbed the whole site yesterday. Unfortunately my site went down, but I think google had finished what it was doing & hasnt been back since. Does google normally forget everything its spidered if it finds a site is down?



3:55 pm on Aug 4, 2003 (gmt 0)

10+ Year Member

For my sites, I have noticed it takes about a week...give or take...for new / modified pages to show in the SERPS. This is great! What used to take a month now only takes a week. I have begun noticing this for the past 2-3 weeks. My sites are PR4?..if the tool bar is correct. Judging from what other have said, the strength of your site determines how regularly the GBOT visits. My sites are visited every 2-3 days.


4:22 pm on Aug 4, 2003 (gmt 0)

10+ Year Member

Hi there...

My 2 cents... usually takes us 48 hours... often, we get fully crawled on first try, though the past 2 weeks it's taken a half week or so...

If you're linking to a site from another site, use multiple links, e.g. home page + 3 internal pages... that MAY help speed it up and keep from just index + robots. Make sure your link text is GOOD.


8:00 pm on Aug 4, 2003 (gmt 0)

10+ Year Member

oh dear bad news for me then :(

our site was crawled and listed just over a month and a half ago and then dropped.

we were then crawled (the whole site) about two weeks ago but nothing since :(

i was hoping for an end of month update but have lost hope now, been left rather annoyed with google recently!


11:50 pm on Aug 4, 2003 (gmt 0)

10+ Year Member

Kanama... not necessarily bad news...

How was the site initially found by Google?

If you submitted... then there's not necessarily much you can do.

If it was linked from another site... IMPORTANT... make sure that link (or links) stays up for a while... at least 1 month, but preferably 3.

That makes a big difference in whether a site stays or drops...

We haven't had a single site (with tons to choose from) dropped on us.


Featured Threads

Hot Threads This Week

Hot Threads This Month