Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: goodroi
My site map is well laid out, we use some (external) js and flash (embedded in the HTML). All pages also have text links to the other pages.
Why is googlebot not paying much attention to us anymore? Do I need a robots.txt file in my directory?
Thanks guys!!! :)
The reason for this is that all (most) robots ask for the robots.txt, and if yoy dosn't hav it the robot will recorde an error. Not good
Are your pages recently modified? Googlebot does a date check, and if your pages are listed as older than the last crawl then googlebot won't do a full spiderring. Google uses a "IF_MODIFIED_SINCE" header in it's requests. Look in your logs for a 304 error. That would be a tell tail singe.