Forum Moderators: Robert Charlton & goodroi
Is msnbot better too for that matter?
I'm curious because at first I thought something must be wrong with my site. Google isn't crawling anything other than the home page and I know there's links that work to go to other pages.
I noticed exabot ripping through my site. Actually, I didn't know what it was and upon researching it...I have to give them props. They have a real good thing going.
As I'm typing this, it's tearing through 90 pages of my site. My site is a portal. Which I hate. BUT it's necessary for the user contribution that it depends on. Much like search engines and the like, this site is a resource for my local area and the technology behind it allows it to grow.
MSN has hit 3, 4, 5 pages a day. Several times a day.
Slurp just one page, several times a day.
Google. One page a day. Just the main page each day, once.
Stinks. Why isn't google crawling my site properly? I don't have it blocked in the robots.txt.
Is Exabot better than Googlebot? Is MSN for that matter?
It's been a little while and once Googlebot hit my site I figured it would crawl the rest of it. So naturally I thought I had a problem with my site. Well if exabot and msn can see the rest of my site. Why can't google?
I noticed Exabot's search is very very nice. Nice thumbnails and a nice layout for related search topics.
I wonder if Google will take advantage of any of those features....
I'm just finding it a bit strange that the almighty G seems to give so many people trouble. Googlebot is missing a lot of sites that aren't junk.
I don't know if it's "missing" my site persay (I can find my site listed), but it certainly isn't doing as good of a job crawling it as some others....
what's up with that?
[edited by: tedster at 12:33 am (utc) on Mar. 1, 2006]
[edit reason] remove specifics [/edit]
