Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies


Include Robots.txt in my site

2:38 pm on Sep 10, 2009 (gmt 0)

New User

5+ Year Member

joined:Sept 1, 2009
votes: 0


Guys i just include robots.txt & sitemap.xml in my website is there any benefit !

or these are useless

12:00 pm on Sept 11, 2009 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 67

Welcome to WebmasterWorld ravikant!

Robots.txt and sitemaps are tools. They may or may not be useful depending on what you need and how you use them.

Robots.txt can help you better control what files the search engines crawl on your website.

Sitemaps can help you expose your content to search engine crawlers. It does not guarantee the search engines will index the content but a sitemap can help the search engines discover your content.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members