Google is not showing all spidered pages in spider log files
Nitin
5:33 am on Apr 28, 2003 (gmt 0)
Google is spidering my discussion pages having dynamic URL's with two parameters,but it is not showing in spider log files.
I am new to this field please guide.
Thanks for your time.
TheDave
6:26 am on Apr 28, 2003 (gmt 0)
I'm not quite sure what your question is, but I'll have a go. Are you looking in your website logs to see if googlebot has been, but not seeing it there? Are you wondering why there are pages in google which according to your logs havn't been spidered yet? The answer may be that google will index links as well as pages, and it is the links you are seeing in the google search results. You can tell if it is only a link by looking in the google SERP (SERP - Search Engine Results Page) and seeing if it has a description beneath the link, and also if there is a link to the cached copy. No description and no cache generally mean that google has indexed the links, but not spidered the pages. I hope this answers you question, but please explain further if it doesn't.
Nitin
6:48 am on Apr 28, 2003 (gmt 0)
Thanks Dave,
SERP is showing description as well as cached copy.It means Googlebot has spidered my pages,still not showing in log files.
TheDave
7:38 am on Apr 28, 2003 (gmt 0)
All I can suggest then, if you are determined to find googlebot in your logs, is to dig deeper/further back into your logs, maybe even get the raw log file and search that if you havent already (ie., load it in a text editor and go Edit->Find)