Welcome to WebmasterWorld Guest from 54.226.246.160

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
multiple disallows
excluding multiple root directories.
5 Michel_Samuel 10:05 pm Oct 10, 2007
Multiple robots.txt files in different directories -will it work?
placing a robots.txt file in sub directories.
3 Michel_Samuel 7:44 pm Oct 9, 2007
Keep physical hosting domain from being indexed?
5 php4U 2:21 am Oct 9, 2007
How do I not allow google
2 pizzaguy 10:31 pm Oct 3, 2007
Should this file be "Public"
Is it good practice to have the robots.txt file.......
4 Propools 11:58 am Sep 26, 2007
Proper URL & File Structure
Fighting with Robots. Txt
3 srkmodi 12:34 pm Sep 24, 2007
Use & or & amp; in robots.txt
Which to use?
3 g1smd 1:53 pm Sep 21, 2007
Using allow to index search-only pages
3 timmbbo 12:31 pm Sep 18, 2007
Disallow - wait a day - add pages
1 brevetoxin 1:36 am Sep 12, 2007
Robots.txt to improve ranks
Will blocking of internal pages through robots.txt improve SE rankings
3 pro_seo 5:18 am Sep 10, 2007
Robots INclusions standard?
13 mikomido 12:28 am Sep 9, 2007
blocking the hostsite but not the hosted sites
3 messemaeker 12:12 pm Sep 7, 2007
Blocking Windows Live robot/spider.
2 Tidal2 10:56 am Sep 7, 2007
Do we have to use robots.txt on every website?
for robots.txt issues
5 masood82 8:42 am Sep 6, 2007
How to block tags?
hundreds of undesired links !
3 rashe18 4:33 pm Sep 4, 2007
Blocked through robots.txt . will the page be crawled ever
5 getxb 6:02 am Aug 31, 2007
URI Only Listing
Need a refresher course on this...
9 pageoneresults 12:20 pm Aug 24, 2007
robot.txt Security
Can be read publically
4 kkonline 10:08 am Aug 24, 2007
robots.txt="noindex,follow"?
2 davidcamus 9:55 am Aug 24, 2007
how to block a page
on a particular location
7 experienced 9:47 am Aug 24, 2007
Multiple "Sitemap: " lines in robots.txt?
Several TLDs use same site code / robots.txt.
2 geoffyp 1:33 pm Aug 10, 2007
Baiduspider and multiple user-agent text file.
4 Vimes 12:04 pm Aug 7, 2007
How can you Disallow a Sub-Domain?
2 zammo 10:11 am Jul 31, 2007
Google Exclude .swf Files
How do i exclude swf files from indexing by google
5 nxfx 5:26 pm Jul 30, 2007
How to disallow entire content, except a particular location
2 brakkar 12:07 pm Jul 30, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month