homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

What is robots.txt?

 5:23 am on Mar 14, 2006 (gmt 0)

When I check "latest visitors" for my websites there are many robots.txt. What are they?



 5:25 am on Mar 14, 2006 (gmt 0)


That should get you started. We'll look forward to your follow up questions once you've had a chance to review the above link.


 5:49 am on Mar 14, 2006 (gmt 0)

Big Thanks for the information.

So if "latest visitors" for my website has robots.txt that means google spiders have crawled my websites & it appears in google search results


 6:01 am on Mar 14, 2006 (gmt 0)

If you see in your logs that GoogleBot has picked up your robots.txt then there is a good chance Google has at least hit your home page as well.

It doesn't mean your home page will appear in Google's index (most likely it will), but, it certainly doesn't mean the entire site will.

You have to play the same game we all play!

We all look at our logs, we look at how many pages Google, (or any other SE visits) and then we look at how they indexed them (if at all).

Don't expect immediate change, it can take anywhere from 2 days to several weeks.

There are commands you can use on each SE to determine which pages have been indexed.

For Google "site:mydomain.com" will help you find those pages indexed.

The more links to your site (& quality links), the more times GoogleBot will visit you, therefore the better the chance you have of getting the whole site indexed quickly :)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved