Welcome to WebmasterWorld Guest from 54.145.166.96

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
help.robots.txt/external link
disallow external link
7 dennis_oppelt 2:35 am Mar 14, 2006
more on robots and mirror sites
clarify please
3 dennis_oppelt 12:48 am Mar 14, 2006
robots.txt +ssi+.htaccess
3 OldWolf 5:40 pm Mar 13, 2006
How do I use a wild card for many pages?
wild card?
6 maha 1:38 am Mar 12, 2006
Elimination of pages from a website.
Do i have to notify the engines in some way?
9 Rani 12:26 am Mar 12, 2006
Robots.txt and htm - html
is disallow for htm killing pages that are html?
5 chewy 6:37 pm Mar 8, 2006
Newbie robots.txt question
Can a robot find content not referenced in pages on the site?
4 joykesilon 8:02 pm Mar 6, 2006
Disallowing double bot crawl due to addon domain
2 Alex_lon 8:04 pm Mar 2, 2006
Robots.txt Basic question
600 rules
3 omoutop 9:46 am Feb 27, 2006
robots.txt Help
disallow issue
3 malasorte 6:10 pm Feb 26, 2006
PHPBB Site
How do I get the bots to search my content instead of my intro page?
3 xmar 4:32 am Feb 25, 2006
Blocking Content On A New Site
New version of major website being launched.
2 caragh 3:04 pm Feb 23, 2006
htaccess Allow robots.txt and leech protection
3 war59312 3:30 am Feb 23, 2006
NimbleCrawler is Bad Bot, well worth banning.
NimbleCrawler is Bad Bot. Never reads robots.txt file.
10 d_mot 1:24 am Feb 23, 2006
In the era of increasing numbers of bad bots is robots.txt irrelevant?
Why bother? Why not go the other way and white-list with controls?[2] ( 1 2 )
34 Webwork 9:21 pm Feb 21, 2006
Disallow a directory, but allow one of its subdirectories
2 Nutter 9:48 am Feb 21, 2006
robots.txt disallow
How do I ensure re-direct pages are not duplicated?
1 robinantill 11:05 pm Feb 18, 2006
Complicated Robots.txt / htaccess Question
maybe
1 fraggle 9:23 pm Feb 18, 2006
robots.txt - subdomains and domain
where to place
3 Rose_1171 8:06 am Feb 18, 2006
Http: vs Https:
What are the best ways to stop crawlers?
3 thaedge 7:54 am Feb 18, 2006
Does robots.txt work per directory or per domain?
3 zver0 6:40 pm Feb 16, 2006
Do bots confuse .htm and .html files in robots.txt?
2 subgenius 6:08 pm Feb 16, 2006
robots.txt Advertising Methods
The new trend...
3 pageoneresults 4:49 pm Feb 16, 2006
Banning all subdirectories in a directory without the files?
5 zver0 1:28 pm Feb 16, 2006
renewed site not being indexed - old pages in serp cache
how to fix robots.txt mistake
4 Ring 10:47 am Feb 15, 2006

Featured Threads

Hot Threads This Week

Hot Threads This Month