homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Current Google spidering pattern: how long is it taking?

10+ Year Member

Msg#: 15638 posted 2:40 pm on Jul 29, 2003 (gmt 0)

Hey y'all

Google has listed my index.html but I wanted to know how long it will be before the rest of the site is index. The date in the description of my site in google shows as July 6th, does it normally take this long for google to come back or is it possible it just doesnt loke my site?

Thanking you



WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 15638 posted 1:18 am on Jul 30, 2003 (gmt 0)

It can take a couple of months. It's not unusual for the homepage to be grabbed first and then other pages picked up later on.

Get good inbound links with decent PR from quality sites and have patience. It's not a problem, it's just the way it works.

What's bothering me is that I'm working on a new site - slowly and gradually - that is NOT linked to from anyplace whatsoever and Google got the whole thing right off it seems. I have no idea how or when since there are no stats available with that hosting - but there those pages sit, right in the index and two are ranking not bad at all considering it's all under construction.

There's no set time or rule that we can know for certain - but it often does take a couple of months to get more than just the index page in.


10+ Year Member

Msg#: 15638 posted 1:26 am on Jul 30, 2003 (gmt 0)

There are those who think that visiting your new pages with the toolbar can get them in the index without any external links to the site. I haven't a clue whether this is true. Brett implied yes in a previous post but GG thinks not.


10+ Year Member

Msg#: 15638 posted 12:28 am on Jul 30, 2003 (gmt 0)

I started a new site in very early June and it had visits from quite a few bots about 2 weeks later. Google came to my site and only hit the index page. Same thing with Slurp. To this day, I haven't had the bots back to my site.

AltaVista Scooter came and indexed my whole site and I am listed there but that's it.

I don't have any text hidden, the site is very straight forward but they haven't been back. Maybe my HTML stinks or google is just afraid of me.

My other website gets hit almost everyday by google and I have a very good ranking but still nothing.

Can anyone tell me what I am doing wrong?

Thanks for anyone who can help.


10+ Year Member

Msg#: 15638 posted 2:21 am on Jul 30, 2003 (gmt 0)

Please forgive my naivety on the subject… I have created a few small personal websites; my current project is the first real big go at a large audience website.

I put up the first 20 pages of the site in late may, and have added 100 since… 60 in late June… 40 this past week. I will continue to add pages in blocks of 20-60 at a time as I am working on a regional website, and it will take some time to cover the area.

My question is… Is it normal to get 4 weeks of nearly daily googlebot visits followed by, what is now a week of none? I should not say any necessarily… I am still getting a robots and index.htm request every other day, but nothing more.

I suppose this is normal for a new site with relatively few back links as of yet… but I have been wondering, and wanted to get some more input on what to expect (even though it sounds like I should expect the unexpected with google).

I recently also added a basic sitemap to the new site from an old site, which I will be closing soon. The old site is currently linking to the new site, and to each subsection (towns within the area I am creating the site for). Is this not a wise idea? Will googlebot penalize me for doing this?


10+ Year Member

Msg#: 15638 posted 2:33 am on Jul 30, 2003 (gmt 0)

Google has listed my index.html ...

So, get external links to your other pages and speed up it.


WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 15638 posted 2:37 am on Jul 30, 2003 (gmt 0)

Radiolabs, it seems like this is the normal pattern, it's just seems to be taking time in most cases - some just more than others and there can be a few reasons for that.

nwctwx, it's hard to tell with some things with any degree of certainty, but if you're not linking back to the old site you're most likely OK. How is the PR for the old site, and are there a lot of decent links pointing to it?

How about the new site? Any good links to it besides from your old sites? Do your older sites link to each other?


10+ Year Member

Msg#: 15638 posted 3:12 am on Jul 30, 2003 (gmt 0)


The old site has a PR of 3, it never was a huge draw, with a very select local market. There are 10 links pointing to the old site (mostly PR 2-4).

The new site has a few good links, 2 from local libraries, and 3 from local directories. The site is nowhere near complete, so I am being very selective on who I ask for links from at this point. The new site does not link to the old site, but links to a page which has links to the 5 sites I have created in the past year (including 2 from this past month I made for customers, excluding the old site which will be dead in September).

My "market" is fairly untouched by the internet and I already was seeing very good SERPS when the index (county) and each "sub index" (towns) were in the results.

Last Wednesday freshbot did the last major visit (grabbing the index and each main town), since then it has been sparse. I added a site map this past weekend, and am now just waiting for googlebot to go back to it's previous avtivities on the site...

I guess I just need to wait a bit longer and not be impatient ;)


WebmasterWorld Senior Member 10+ Year Member

Msg#: 15638 posted 5:27 pm on Jul 29, 2003 (gmt 0)

Today Googlebot visited my new website for the first time. I am wondering:

a) How long is it going to take for Google to get at least few pages off my site and have it in the index?

b) How long is it going to take to spider the whole website (over 20,000 dynamic pages)?

c) How to make Google spider the most relevant content first? (I've got several articles related to the content, and want Google to get them first).



10+ Year Member

Msg#: 15638 posted 12:40 pm on Jul 30, 2003 (gmt 0)

Did it just visit robots.txt or has it started to grab pages?
If pages were grabbed, you might get lucky and have them in the index within a few days.
If you want googlebot to visit certain pages first, there is no guarantee but you might want to get external links to those pages and also make sure they are not buried deep down from the root directory.


10+ Year Member

Msg#: 15638 posted 2:19 pm on Jul 30, 2003 (gmt 0)

Guess I was being overanxious...

Googlebot hit the site last evening and gobbled up the new site map :)


10+ Year Member

Msg#: 15638 posted 2:12 pm on Aug 4, 2003 (gmt 0)

Hey guys thanks for all the replies!

Google actually came by and grabbed the whole site yesterday. Unfortunately my site went down, but I think google had finished what it was doing & hasnt been back since. Does google normally forget everything its spidered if it finds a site is down?



10+ Year Member

Msg#: 15638 posted 3:55 pm on Aug 4, 2003 (gmt 0)

For my sites, I have noticed it takes about a week...give or take...for new / modified pages to show in the SERPS. This is great! What used to take a month now only takes a week. I have begun noticing this for the past 2-3 weeks. My sites are PR4?..if the tool bar is correct. Judging from what other have said, the strength of your site determines how regularly the GBOT visits. My sites are visited every 2-3 days.


10+ Year Member

Msg#: 15638 posted 4:22 pm on Aug 4, 2003 (gmt 0)

Hi there...

My 2 cents... usually takes us 48 hours... often, we get fully crawled on first try, though the past 2 weeks it's taken a half week or so...

If you're linking to a site from another site, use multiple links, e.g. home page + 3 internal pages... that MAY help speed it up and keep from just index + robots. Make sure your link text is GOOD.


10+ Year Member

Msg#: 15638 posted 8:00 pm on Aug 4, 2003 (gmt 0)

oh dear bad news for me then :(

our site was crawled and listed just over a month and a half ago and then dropped.

we were then crawled (the whole site) about two weeks ago but nothing since :(

i was hoping for an end of month update but have lost hope now, been left rather annoyed with google recently!


10+ Year Member

Msg#: 15638 posted 11:50 pm on Aug 4, 2003 (gmt 0)

Kanama... not necessarily bad news...

How was the site initially found by Google?

If you submitted... then there's not necessarily much you can do.

If it was linked from another site... IMPORTANT... make sure that link (or links) stays up for a while... at least 1 month, but preferably 3.

That makes a big difference in whether a site stays or drops...

We haven't had a single site (with tons to choose from) dropped on us.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved