8. I have changed my DNS/Ip and Googlebot doesn't come anymore.
Recent thread about this: google not liking new IPs? [webmasterworld.com]
Also from the Google Knowledge base:
Googlebot How long does Google cache IP's? [webmasterworld.com]
From it:

Personally I prefer to keep a site on both IPs for a month. This required a helpful Web host if you don't run your own equipment. (-ciml)

I think this is a very good tip.

9. How do i get Freshbot to visit my site?
This is also stated in the excellent
Google Updates and Everflux, the Monthly Mid-Cycle Changes [webmasterworld.com]
The easiest way is to have good inbound links, and it will help if they have a higher PageRank.
Also if you change the content on your site it will help, I would say these are
the 2 most important factors for getting the freshbot to visit your site.

10. Freshbot has been to my pages what happens then?
Freshbot visits for a numerous of different reasons.
They best way to get pages spidered by the Freshbot is stated in the

Google Updates and Everflux, the Monthly Mid-Cycle Changes [webmasterworld.com]
Another tip was suggested by GoogleGuy:
Are you using If Modified Since? [webmasterworld.com]
If your site is completly new and gets spidered by the Freshbot before it's indexed in the mothly update,
it may fall out of the index, from my experience if you haven't changed anything on the pages
it normally drops out after around 5-7 days.
This is not a "general" rule though, it deppends on a lot of other factors too.

11. My site was down during some parts of the Deep crawl what happens now?
GoogleGuy reported last year that they increased support for re-spidering of sites which
had problems during the "normal" deep spidering. Vitaplease made a very interesting comment in:
Google page dropping question [webmasterworld.com]

I think basically I'm asking... does Google only drop a page if it can't reach it? I asked a Google rep what would happen if a site was down for a day during the deep crawl. Would Googlebot come back? He asked what Pagerank? I said 6, he said no problem, the site should be revisited. (that was last October during a pubconference)

12. What is a spider trap and how do i prevent it?
A spider trap is when a spider re-spiders the same page over
and over again, you can compare it to a maze (labyrinth).
The biggest problem with spider traps is often the amount of bandwidth and server load it puts on the site which is spiders.
It also creates a problem for Googlebot which spiders the same page over
and over again even though the content is almost always the same.
The most known spider trap is Session ID's, a Session ID is often used to keep track
of the visitors, and some sites puts a unique ID in the URL:
An example is www.webmasterworld.com/page.php?id=264684413484654
(Note this URL doesn't exist).
Each user gets a unique ID and it's often requested from each page.
The problem here is when Googlebot comes to the page, it spiders the page and
then leaves, it comes back to another page and it finds a link to the same page but since it has been given
a different session id now, the link shows up as another URL. This is one of the reasons
why Googlebot is very very carefully when it spiders pages which uses the querystring "ID=".
I've seen and heard many cases where the same page have been spidered over
1000 times, and sometimes it's been indexed the same amount as it's been spidered, most
search engines have very advanced duplicate filters which removes the duplicates and selets one url.

13. Should i include Googlebot in my traffic reports?
The general suggestion is NO, Googlebot is not a real human being visiting your site.
One quote which i often use is "Don't build pages for search engines, build them for users, it's
the user who will buy thing off your website, not search engines, and search engines wants to
generate the best results for the user, and therefore tries, to think as a user, when it ranks its results"

14. How do i get Googlebot to spider my dynamic (url) pages?
First thing is, do you need to have a Dynamic URL?
There is very many things you can do to get a dynamic site spidered, the support for spidering
dynamic URL's seem to get bigger and bigger each day.
Always try to stay out of using Session Id's in the URL, this is the ultimate killer when
it comes to prevent Googlebot from spidering your dynamic URL'S.
Also try to stay out of using the query string "ID=" Since this is the most common
used query string when it comes to presenting Session Id's googlebot seem to
put a "flag" each time it sees it in the URL. This question have been coveded quite a
few times, but it seem to change over time:
If I Use PHP Will Google Still Like Me? [webmasterworld.com]
Googlebot & Dynamic Pages [webmasterworld.com]
Does Google index dynamic content? [webmasterworld.com]

PageRank seem to play a major roll when it comes to dynamic url's and spidering, the more
PageRank the more chance there is that the url's will be spidered, and the more PageRank the deeper googlebot will go.

15. How do i prevent Googlebot from spidering my site/page/graphics?
Googlebot obey's the Robots.txt standard, to prevent Googlebot to spider your site, you
can put this in your Robots.txt file (Should be placed in the HTTP root category)

User-agent: Googlebot
Disallow: /

To prevent Google from Indexing your images, use:

User-agent: Googlebot-Image
Disallow: /

It's also described officialy at Google.com: No Index tags [google.com]
Also try using the Search Engine World Robots.txt Validator [searchengineworld.com]

16. I've been deep spidered what now?
If you have been deep spidered the biggest chance is that you will appear in the
upcomming "Google Update", This happens about once a month:
Google Update Chart [webmasterworld.com].
Be advised though that your site may not appear in the next update for a lot of reasons, most
of them which only Google knows, and will probably not tell you.
For the PageRank to be correctly calculated from the incomming links i advise to wait up to 2 updates.
And all inbound links may not show up with the LINK: commando
(Rule seem to be that only pages which have a PageRank of 4 and above will show up).

17. Which Ip does Froogle spider from?
From what i know Froogle uses the 64.* ip, and the same User Agent.

18. Googlebot spiders both my [domain.com...] and [domain.com,...] what should i do?
The best thing to do is to pick one of the url's and use a 301 redirect (Permananet redirect)
If you don't know what or how to use it try this

Google search for: 301 redirect [google.com]

19. Does Googlebot crawl Adwords?
From: Google crawls URLs in adwords? [webmasterworld.com]

I was talking with Google and they do not crawl the adwords links. ~allanp73

It doesn't seem to help buying adword listings, and getting spidered due to it.

Note: The information stated above may not be correct! and remember things change

Thanks to ciml and Mike_Mackin for editing
Edit: spelling

[edited by: lazerzubb at 6:49 pm (utc) on Feb. 10, 2003]