Forum Moderators: phranque

Message Too Old, No Replies

How often do SE spiders crawl?

         

Rightz

4:25 pm on Jun 13, 2006 (gmt 0)

10+ Year Member



I've got a new site which isn't live yet. I've simply put a 'coming soon' note on my index.html.

I've noticed I've had both Yahoo and Ask Jeeves crawl.

I'm worried that I've lost out because I haven't got my site linked up and ready for viewing.

When will the SE robots come back?

Terabytes

4:43 pm on Jun 13, 2006 (gmt 0)

10+ Year Member



that's the problem with submitting your site before it's ready the first time...

they come...see nothing...and take a while to return...

Once you get the site going, they will return...eventually....

I'd get going on the site now....you don't know when they will return to index the site again...

return visits depend on other factors...is the content worth indexing....?

how often do you update the site..?
(if you put up a site, and do nothing else with it for like 9 months, the SE's notice...and they don't index the site very often....could actually be months before they come back....)

The spiders learn....(IMO) during a redesgn of my site, the site got indexed...the spider came back the next day...found content had changed and indexed again...eventually the spider saw the site getting updated very frequently during this period and actually indexed every 30 minutes for about a week thinking that the site had changing content on a regular basis...after the site was done...it eventually stopped comming so often, and now comes about once every 2-3 days to check for changes)

NOTE: I saw question here in the forum that was placed at like 10:30am....and was in G at 11:15am (amazing!)

bottom line...don't submit the site till you're ready to have it crawled, then keep making updates, no matter how small you think the update is, I believe it keys the return visits frequency....

hope that helps...
Tera

Rightz

4:47 pm on Jun 13, 2006 (gmt 0)

10+ Year Member



I haven't submitted my site with any search engines. I have tried to link to anyone.

The domain name is about a month old. Should I put the proper index.html up there in case they come back soon? I was hoping to have it 'live' in a couple of weeks anyay and then start submitting it.

Thanks

Terabytes

4:54 pm on Jun 13, 2006 (gmt 0)

10+ Year Member



I'd get it out there as soon as you can...especially the index page...

a couple more visits to "coming soon" may develop into a problem for you that may force you to "sit and wait"...

Somewhere, the SE picked up a link to your domain..
(they don't just pull a domain name out of thin air and try to index them)

just my $0.02
Tera

Rightz

4:59 pm on Jun 13, 2006 (gmt 0)

10+ Year Member



Thanks for the advice :) I'll get my site all linked up to the index.html then.