Forum Moderators: open
Over the weekend i was pleased to see ive been listed in google and yahoo (not dmoz yet). Only 3 days ago, i was told to use guestbooks and external links to get listed. I done this and a few days later im in the search listing?
Ive also looked in my logs and there is no sign of google, now that im listed through these external links, will i be spidered on the next run, or is there still a chance they wont send in there spiders?
thanx
If you have a snipped in the Google results (the description under your site title) then you will have been crawled by Googlebot. Otherwise, your URL was listed from a link but your page wasn't crawled in time. Hopefully you'll be in at the next update.
Im sure their is no googlebot in my logs too, does it always say google or could it come under an ip address?
216.239.46* is the most common block of Googlebot IP's.
strangly enough i got listed in both (yahoo and google) on the same day?
If you're talking about a listing on Yahoo under "web page listings" it's not strange at all, since those are Google results.
In any case, if you're listed in Google as of the latest update (within the past week) it's almost a certainty that the site was spidered in the first few days of June or the last days of May. Those would be the logs to check.
Are there any other IP adresses i should be looking out for in my logs for evidence of spiders etc
Anyway, it's really interesting to see how the pattern of spidering follows the link patterns in my site - through sitemap pages, etc. Last month every page of my site went into the index. This month I have 50% more pages. Again, they have all been spidered over the last three days, so with luck they will all end up in the index at the end of the month.
Depending on how many pages you have in your website, you may like to try a trick I have worked out. I extract the log entries for Googlebot into a text file, then export to an Excel spreadsheet. Then I can sort the entries in different ways, such as by alphabetical page name, time or spidering, response type (to pick out any 404's, etc). This makes it easy to spot any pages that have been spidered more than once, or maybe not at all.
Addictive? No question!
Hey, i found it addictive looking in logs and didnt even know what i was properly looking for, ha ha..
Now i know i'll be looking all the time for googles IP address, as ive been looking out for the txt 'googlebot' for the past 3 months.
