Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

What is robots.txt?




5:23 am on Mar 14, 2006 (gmt 0)

5+ Year Member

When I check "latest visitors" for my websites there are many robots.txt. What are they?


5:25 am on Mar 14, 2006 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month


That should get you started. We'll look forward to your follow up questions once you've had a chance to review the above link.


5:49 am on Mar 14, 2006 (gmt 0)

5+ Year Member

Big Thanks for the information.

So if "latest visitors" for my website has robots.txt that means google spiders have crawled my websites & it appears in google search results


6:01 am on Mar 14, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

If you see in your logs that GoogleBot has picked up your robots.txt then there is a good chance Google has at least hit your home page as well.

It doesn't mean your home page will appear in Google's index (most likely it will), but, it certainly doesn't mean the entire site will.

You have to play the same game we all play!

We all look at our logs, we look at how many pages Google, (or any other SE visits) and then we look at how they indexed them (if at all).

Don't expect immediate change, it can take anywhere from 2 days to several weeks.

There are commands you can use on each SE to determine which pages have been indexed.

For Google "site:mydomain.com" will help you find those pages indexed.

The more links to your site (& quality links), the more times GoogleBot will visit you, therefore the better the chance you have of getting the whole site indexed quickly :)


Featured Threads

Hot Threads This Week

Hot Threads This Month