Forum Moderators: open
I am new to Webmaster World this is my first post here :)
I just started building websites for people. I started building sites when I was 17 years old I am 22 now, but I am not sure about the whole optimization of sites. I am not to sure how it all works, because I found out in the forum that most search engines do not allow use of SEO programs to check rank or submit websites. So what is the best way to submit websites and to check rank? See the sites I use to make were my own and I never bothered submitting them to search engines, but now I've started making websites for other people, so I need to know what the best ways to check rank and submit are. I just got my own website up not to long ago can someone here check out the meta tags and all that on my site tell me if they are set right please?
Also is it best to have Robot.txt files, or just putting the meta tag ROBOTS=ALL would work the same?
Thanks for the help,
Cheryl Vines
As far as submitting, most will tell you here they have not submitted a site in years. Work on your link popularity with vital sites, a listing in Yahoo! and Business.com is still worthwhile. Since spiders will crawl the web, getting a link from a site that is indexed often, will in turn get your site indexed far faster than submitting the URL to the engine itself.
As far as checking links Search Engine Commando seems to be the tool of choice.
I would suggest the robots.txt file as there are engines that will ignore the meta robots tag. However, as you will see from your reading here on WebmasterWorld, Google and Yahoo! often ignore it too!
CaboWabo
I found it helpful to submit sites when I'm done with the initial coding. It doesn't take a lot of time and the submission process makes you think about the site purpose and allows you a chance to reevaluate your keywords, anchor text and such.
Also be aware that having a great robots.txt file is no guarantee the spiders will obey it. Check out the robot.txt forum for more info.
jb
Cheryl
I’ve had a website for ages, but never really had time to promote it. All of this SE info is great!
I’m wondering if it’s better to not have a robots.txt file at all if you want all the robots to spider your site. I’ve always been someone cautions of breaking the robot file and perhaps telling the SE’s not to spider. Is it actually better to have a robots file in this case?
Paul
Read these posts about meta tags, msgs 49-61, still basically applicable. Meta tags are not very important these days.
[webmasterworld.com...]
The title tag is important in conjunction with relevant content on your page, but it's not technically a meta tag.
Robots.txt is a file to exclude robots, not to get them to spider your site. You seem to be talking about the robots meta tag but calling it "robots.txt". Most engines will spider a page without the robots meta tag. The tag is also more useful for excluding robots than for getting your site indexed.
The major engines these days find you by following links to your site from sites already in their index. If you don't have such inbound links, you won't rank anyway.
What we can’t tell you is the effect of not having a robots.txt file will have on how the SE’s look at your site. We cannot tell you that when a spider looks for the robots.txt and doesn’t find one, what affect that has on the spiders’ actions. Since it is such a small task to create one, why not?
You will find in this forum, many opinions about how a properly formatted site performs over an improperly formatted site. Should you have a robots.txt file? Should you validate your code? Should you use meta tags? Should you test on different browsers?
My opinion is that you want to eliminate as many potential problems areas as you can. Take the time to validate your code, test on different browsers and add a robots.txt file. You can then run spider simulations on the site to see if it’s working. You cannot have too much information.
jb
Google and having *no* robots.txt file
[webmasterworld.com...]
If you want all spiders to access all of your pages, then the only advantage to having a robots.txt file is that you won't get hundreds of 404-Not Found errors on your site each day caused by robots trying to fetch your non-existent robots.txt.
