Welcome to WebmasterWorld Guest from 54.163.35.238

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Duplicate Content in Robots.txt file
2 palmpal 12:17 am Jan 30, 2006
To allow some robots only
9 GoldenHammer 3:14 pm Jan 28, 2006
robots.txt vs htaccess for control
if the robots.txt is not dependable why not just use htaccess?
6 ccubed99 3:39 am Jan 28, 2006
robots.txt and htaccess
4 agradation 11:17 pm Jan 26, 2006
Typical robots.txt?
Anybody got a same robots.txt?
1 1Lit 6:58 pm Jan 26, 2006
Disallow for sub-domain directory?
7 palmpal 4:18 pm Jan 26, 2006
Blocking old pages
1 Nutter 3:21 pm Jan 26, 2006
Wildcards special syntax for Google
publish Google unswer about special Google syntax in robots.txt plus examle
2 spiral 3:44 am Jan 26, 2006
Spiders finding you against all odds
3 Lyndsay 6:49 pm Jan 23, 2006
yahoo/slurp spidering
We don't want these pages indexed by yahoo
9 latimer 8:28 am Jan 23, 2006
Correct way to ban a directory
5 palmpal 3:18 am Jan 23, 2006
WWW vs. Non WWW
Site indexed twice need robot.txt help
3 carjocky 4:10 pm Jan 20, 2006
Unknowingly restricting some directories
2 estaquieto 4:40 am Jan 20, 2006
MSNBOT and robots.txt
2 Impala 4:31 am Jan 20, 2006
How-to: robots.txt white-listing with PHP/Apache
2 encyclo 2:00 pm Jan 16, 2006
Prohibiting folders by name regardless of root relation
Is this possible or not?
6 Jeremy_H 2:38 pm Jan 15, 2006
question about robots.txt, am I banning a directory by accident
17 arbitrary 3:32 am Jan 15, 2006
Quick 'Sanity Check'
3 markd 10:20 am Jan 11, 2006
directory
4 sylvieg 4:38 am Jan 11, 2006
Removing pages from Google's cache
Is it best to allow a "robots" "noarchive" crawl first?
1 Phil_Payne 2:05 am Jan 10, 2006
Disallow ALL folders?
but allow top level html
6 Reno 9:14 pm Jan 9, 2006
How to disallow widgets from being indexed
Order of User-agents
2 albatros 8:10 pm Jan 9, 2006
Empty line after User-agent:
Could this do any harm?
1 taps 5:31 pm Jan 7, 2006
Robot Validator Problem
7 bunky6767 1:13 pm Jan 6, 2006
Blocking all URLS to a website using the robots.txt file?
2 scubajared 10:05 am Jan 6, 2006

Featured Threads

Hot Threads This Week

Hot Threads This Month