Forum Moderators: open

Message Too Old, No Replies

Pages on my site are not being followed

Google spider only looks at a couple pages then leaves

         

jmwebguy

1:05 am on Jan 17, 2004 (gmt 0)

10+ Year Member



When I look at the raw logs of my site, I see that googlebot hits my robots.txt file and then on more, alternate.htm. But then it leaves. Why does it not see my other pages? It was actually a blessing though that it I saw it spidering altername.htm so much. At first it was not a real webpage, just part of a drop down menu system. I then made it a real page that looks like the rest of my site, but added more keywords, so I'm showing up a lot in search engines.

BUT, still, it's not spidering any of my other pages. And here's the real catch...I don't see it spidering my other pages, but when I do a search for my website in google, I see lots of my pages. I'm just not sure how they got there.

But anyway, I have added a few more product specific pages to my site, but I am not convinced Google will find them when it spiders in the next day or two. Any ideas?

simonuk

1:44 am on Jan 17, 2004 (gmt 0)

10+ Year Member



1. Make sure the bot can find all your links. Site maps are good for large sites.

2. Make sure you don't have the same title tag on each page.

3. Google only deep crawls every 4 to 9 weeks. The normal bot which comes along often will only update the home page.

Simon.

Powdork

8:08 am on Jan 17, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You said drp down menu. Google has a hard time spidering the javascripty things like drop down menus, rollovers, etc. I recommend using a sim spider, such as the one on the search engine watch site(SE Tools) to see what links a spider finds. Freshbot will crawl much more than your index page, but its crawling is limited by your site's PR (or other factors). I would
1. Check your site with a sim spider
2. Get lots more links, especially to interior pages
3. Wait