Forum Moderators: open
In Danny Sullivan's searchenginewatch.com, it is stated that all engines are deep crawlers except Go.
[searchenginewatch.com...]
Brett Tabke says that only excite, google, inktomi, and Northern Light deep crawl.
[searchengineworld.com...]
Brett, how did you arrive at this conclusion?
We've had problems with our logs so I can't analyze these in order to figure out who deep crawls based on spider activity through our web pages. So I have to rely on outside information for now.
My link popularity model is based on the assumption that all spiders are deep crawlers.
In fact, I have a question about this as well.
Let's say I have site A. All of the pages in site A are linked to each other. In the index page of site A, I have a link to site B, and vice versa. Index of site C is also linked to index of site B. Would the engines consider site A and site C linked to each other? How many levels does this encompass? This is in fact a question about how engines measure link popularity, so if anyone can refer me to a few good, thorough articles about this I would appreciate it.
Thanks so much!!
2M
Crawler behavior is prety easy to determine. If it only hits the pages you submit, then it isn't a deep crawler. Most engines are a combo of the two forms. Like Alta: it adds the pages you submit, but will occasionally do a full crawl of your site if it likes it.
Glad you brought this up...feels good to know I've learned a few things in the past few months ;)
