Welcome to WebmasterWorld Guest from

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Google should spider within a few days

8:57 am on Sep 16, 2005 (gmt 0)

New User

10+ Year Member

joined:July 20, 2005
votes: 0

I've always wondered how do you know if a search engine (Google)has spidered your site. Does typing your site in give you any idea?
9:08 am on Sept 16, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
votes: 0

I think most people know by looking at their RAW server log files. It's pretty easy to figure out. If you don't have access to these files, usually you can see search engine user agents in stats packages most hosting providers offer for free.
6:02 pm on Sept 16, 2005 (gmt 0)

New User

10+ Year Member

joined:Sept 13, 2005
votes: 0

Speaking of spidering... I see that G spiders one of my sites more frequently than the other. The one that is less spidered is older and less active. When I look at the page views, neither of them is really popular altho both have a PR of 4/10. I just added AdSense and have been really busy building up content pages. I add at least a page a day. I have a third site that is just not even listed by Google although I have had it for a couple of months. I finally requested Google to spider it a couple of weeks ago. Why would the older site be spidered less than the newer site -- G has not had time to know that I am adding content so frequently and both have about 20 pages. Well the second one has about 40 pages now.

As for the newest one...it is a .biz site rather than .com....hmm, just realized the older but less spidered page is .ca. Could this be a factor?

Why oh why does the spider not like my pages?