Page is a not externally linkable
the_nerd - 4:31 pm on Aug 12, 2004 (gmt 0)
But not every spider is allowed to spider as many pages a Google's. I'd probably block him because I see no commercial benefit to having somebody sucking in my pages. nerd
ogletree, There database is not what has valuable it is there algo and thee ability to dispense so many SERPS at once fast. Anybody can write a spider and collect the internet. I don't know what his problem is