homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
What is robots.txt?
Mystery
forex




msg:1529408
 5:23 am on Mar 14, 2006 (gmt 0)

When I check "latest visitors" for my websites there are many robots.txt. What are they?

 

pageoneresults




msg:1529409
 5:25 am on Mar 14, 2006 (gmt 0)

[searchengineworld.com...]

That should get you started. We'll look forward to your follow up questions once you've had a chance to review the above link.

forex




msg:1529410
 5:49 am on Mar 14, 2006 (gmt 0)

Big Thanks for the information.

So if "latest visitors" for my website has robots.txt that means google spiders have crawled my websites & it appears in google search results

percentages




msg:1529411
 6:01 am on Mar 14, 2006 (gmt 0)

If you see in your logs that GoogleBot has picked up your robots.txt then there is a good chance Google has at least hit your home page as well.

It doesn't mean your home page will appear in Google's index (most likely it will), but, it certainly doesn't mean the entire site will.

You have to play the same game we all play!

We all look at our logs, we look at how many pages Google, (or any other SE visits) and then we look at how they indexed them (if at all).

Don't expect immediate change, it can take anywhere from 2 days to several weeks.

There are commands you can use on each SE to determine which pages have been indexed.

For Google "site:mydomain.com" will help you find those pages indexed.

The more links to your site (& quality links), the more times GoogleBot will visit you, therefore the better the chance you have of getting the whole site indexed quickly :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved