Forum Moderators: DixonJones
Can anyone help me? Google and MSN bot refuse to spider my website.
I dont have a robot.txt file in the root of my site, but that shoudnt be a problem or is it? All my other websites dont have robot.txt either and are indexed/spidered.
Google has already responded in person to my question, and told me that it was only partially spidered, only the domain name is indexed by Google. If I look at my log, I can see that Google tries to find robot.txt and then leaves the site. Can anyone help e if it is a desing problem (the flash player? or if it is caused by another problem, robot or something else?)
Thanks!
[edited by: jatar_k at 2:47 am (utc) on Mar. 18, 2006]
[edit reason] no urls thanks [/edit]
When you attempt to access robots.txt does the server return a proper 404 Not Found error? What happens when a page does not exist? Also, are the page links standard href links or is the site mostly or all in Flash? Does each page have a unique URL?
When attempting the robots.txt (I assume it must be in the root so, www.domain.com/robots.txt?) I dont get a proper 404 not found error, but just the standard error page, which you get when attempting to visit an online page when you're offline. I get this in all cases. I don't have a webserver of my own, just a site hosted by a hosting service, so I dont know how to change this. All pages are standard href links (no dynamic php or flash links), and I only have one single Flash opject in the center of each page. All pages have a unique URL and pages have the extention .php (alltough there are no server side scripts on the pages as of yet).
Hope this is enough info! I really appreciate your effort of helping me out! Thanks again..
