Welcome to WebmasterWorld Guest from 50.19.53.104

Forum Moderators: goodroi

Message Too Old, No Replies

What is robots.txt?

Mystery

     

forex

5:23 am on Mar 14, 2006 (gmt 0)

5+ Year Member



When I check "latest visitors" for my websites there are many robots.txt. What are they?

pageoneresults

5:25 am on Mar 14, 2006 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member



[searchengineworld.com...]

That should get you started. We'll look forward to your follow up questions once you've had a chance to review the above link.

forex

5:49 am on Mar 14, 2006 (gmt 0)

5+ Year Member



Big Thanks for the information.

So if "latest visitors" for my website has robots.txt that means google spiders have crawled my websites & it appears in google search results

percentages

6:01 am on Mar 14, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you see in your logs that GoogleBot has picked up your robots.txt then there is a good chance Google has at least hit your home page as well.

It doesn't mean your home page will appear in Google's index (most likely it will), but, it certainly doesn't mean the entire site will.

You have to play the same game we all play!

We all look at our logs, we look at how many pages Google, (or any other SE visits) and then we look at how they indexed them (if at all).

Don't expect immediate change, it can take anywhere from 2 days to several weeks.

There are commands you can use on each SE to determine which pages have been indexed.

For Google "site:mydomain.com" will help you find those pages indexed.

The more links to your site (& quality links), the more times GoogleBot will visit you, therefore the better the chance you have of getting the whole site indexed quickly :)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month