Welcome to WebmasterWorld Guest from 107.20.122.81

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
libwww-perl/5.803
what is libwww-perl/5.803
4 evo81 8:12 pm Mar 28, 2006
robots -how to disallow search results
5 zuses 10:53 am Mar 23, 2006
Syntax check robots.txt
There should be atleast 1 disallow line in any Robots.txt.
3 mycutegoddess 10:32 am Mar 17, 2006
robots.txt in PHP Nuke
I need help with create robots.txt in PHP Nuke
2 spediter 2:28 am Mar 16, 2006
What is robots.txt?
Mystery
4 forex 6:01 am Mar 14, 2006
Validating robots.txt
Research on valid robots.txt and usage
4 smitha_ajay 5:18 am Mar 14, 2006
help.robots.txt/external link
disallow external link
7 dennis_oppelt 2:35 am Mar 14, 2006
more on robots and mirror sites
clarify please
3 dennis_oppelt 12:48 am Mar 14, 2006
robots.txt +ssi+.htaccess
3 OldWolf 5:40 pm Mar 13, 2006
How do I use a wild card for many pages?
wild card?
6 maha 1:38 am Mar 12, 2006
Elimination of pages from a website.
Do i have to notify the engines in some way?
9 Rani 12:26 am Mar 12, 2006
Robots.txt and htm - html
is disallow for htm killing pages that are html?
5 chewy 6:37 pm Mar 8, 2006
Newbie robots.txt question
Can a robot find content not referenced in pages on the site?
4 joykesilon 8:02 pm Mar 6, 2006
Disallowing double bot crawl due to addon domain
2 Alex_lon 8:04 pm Mar 2, 2006
Robots.txt Basic question
600 rules
3 omoutop 9:46 am Feb 27, 2006
robots.txt Help
disallow issue
3 malasorte 6:10 pm Feb 26, 2006
PHPBB Site
How do I get the bots to search my content instead of my intro page?
3 xmar 4:32 am Feb 25, 2006
Blocking Content On A New Site
New version of major website being launched.
2 caragh 3:04 pm Feb 23, 2006
htaccess Allow robots.txt and leech protection
3 war59312 3:30 am Feb 23, 2006
NimbleCrawler is Bad Bot, well worth banning.
NimbleCrawler is Bad Bot. Never reads robots.txt file.
10 d_mot 1:24 am Feb 23, 2006
In the era of increasing numbers of bad bots is robots.txt irrelevant?
Why bother? Why not go the other way and white-list with controls?[2] ( 1 2 )
34 Webwork 9:21 pm Feb 21, 2006
Disallow a directory, but allow one of its subdirectories
2 Nutter 9:48 am Feb 21, 2006
robots.txt disallow
How do I ensure re-direct pages are not duplicated?
1 robinantill 11:05 pm Feb 18, 2006
Complicated Robots.txt / htaccess Question
maybe
1 fraggle 9:23 pm Feb 18, 2006
robots.txt - subdomains and domain
where to place
3 Rose_1171 8:06 am Feb 18, 2006

Featured Threads

Hot Threads This Week

Hot Threads This Month