Welcome to WebmasterWorld Guest from 54.163.52.98

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
after google indexs is it too late to use Disallow
2 watercrazed 8:51 pm Oct 30, 2005
Exclude by filename, not directory?
19 joergnw10 7:36 pm Oct 30, 2005
Robots.txt File to Block from Dynamic URLs
Trying to eliminate duplicate content...
3 BillyS 2:07 am Oct 28, 2005
disallowing a folder with its own domain name
subdomain being spidered although it has its own domain name
2 chandrika 8:52 pm Oct 26, 2005
robots.txt in a folder not root
will this work?
3 fatpeter 6:27 am Oct 26, 2005
Robots.txt and Query Strings
Can they block non directory paths?
3 soquinn 4:45 am Oct 26, 2005
Spider and registred access to web
5 andreabg 8:07 am Oct 19, 2005
google accessing blocked image files
why does it appear that google is not abiding robots.txt?
3 latimer 8:01 am Oct 19, 2005
Disallow all Directories
2 Omega1Megs 8:48 pm Oct 14, 2005
Disallow: line in robots.txt
2 stefan009 5:33 pm Oct 14, 2005
Can a robots.txt be too long?
3 gingerspice 3:32 am Oct 13, 2005
Log file is robot.txt file?
Log file is robot.txt file?
5 i_am_dhaval 9:20 pm Oct 12, 2005
Excluding domain in robots.txt
10 triumph 7:25 pm Oct 11, 2005
Disallow dynamic content with mod_rewrite
Disallow dynamic content with mod_rewrite
2 crglmb 3:54 pm Oct 10, 2005
Robots.txt in Ascii
How-to?
2 funnycloud 6:20 pm Oct 7, 2005
is there a reason to use a robots.txt
5 thorthedestroyer 2:50 am Oct 7, 2005
Robot.txt and Sitemap
3 sierra11b 1:26 am Oct 5, 2005
Newbie Robots.txt file Q
5 CoffeeMan 12:34 pm Oct 3, 2005
SygolBot, from sygol.com: anyone seen this before?
just visited my site, and did NOT respect robots.txt
3 stapel 12:37 am Oct 3, 2005
Can I get a page crawled daily?
disallow commands all I can find
5 innocbystr 11:33 pm Oct 2, 2005
Need to double check my Entry
Q - regarding ( / ) (Forward Slash) in Entry
2 Tutti 3:59 pm Oct 2, 2005
Trying to add robots.txt to invisionfree forum
can't get it to work
5 Damaris 1:34 am Sep 29, 2005
What does robots.txt help to do?
3 flashcination 10:40 pm Sep 28, 2005
How to block URL's of other sites link on my site?
How to block URL's of other sites link on my site?
3 sweetu 9:09 pm Sep 25, 2005
How to make a robots.txt file?
What are the steps?
2 steelcutman1975 8:37 pm Sep 23, 2005

Featured Threads

Hot Threads This Week

Hot Threads This Month