Forum Moderators: phranque
I have been trying to get my site into google for so long now and am I linked to a few people in google.
I was thinking, maybe I am not allowing the bot to spider my pages?
How do I go about finding this robot.txt or wherever I need to go to be able to allow the google bot.
I appreciate any help I can be given. Thank you.
-modega
Robots.txt is used to EXCLUDE robots from specific areas or your entire site. If you don't have a robots.txt file at all, you don't have to worry that you are excluding Googlebot.
To get spidered, your best bet is to get inbound links to your site from other sites. You can also use Google's Add URL form to submit site pages, but inbound linkage will help both with spidering and ranking.
Good luck!
open a browser and type your domain name and then /robots.txt
This will show you what is in your robots.txt file if you have one
The file is located in the root of your site that is to say in the same directory as your index page so if you can ftp on to your sever and upload your index file then the robots.txt file should also be visible in this directory
Like rogerd said you don't have to have a robots.txt file but if you do it will stop the search engine spiders from putting a 404 error file not found in the server error logs
Hope this helps
ncw164x
