Welcome to WebmasterWorld Guest from 184.73.66.157

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Google gives 404 error on robots.txt
Googlebot not able to find robots.txt
7 Govind_Uppaluri 11:53 am Apr 10, 2006
Keeping bots out of one script
Is this valid?
4 trillianjedi 10:32 am Apr 10, 2006
Robots Validator?
Anyone know of one?
7 thaedge 11:01 pm Apr 5, 2006
how to disallow spider to index one of my subdomain' files?
4 peony 7:44 pm Apr 5, 2006
Sessions & robots - need help
3 zuses 6:54 am Mar 31, 2006
libwww-perl/5.803
what is libwww-perl/5.803
4 evo81 8:12 pm Mar 28, 2006
robots -how to disallow search results
5 zuses 10:53 am Mar 23, 2006
Syntax check robots.txt
There should be atleast 1 disallow line in any Robots.txt.
3 mycutegoddess 10:32 am Mar 17, 2006
robots.txt in PHP Nuke
I need help with create robots.txt in PHP Nuke
2 spediter 2:28 am Mar 16, 2006
What is robots.txt?
Mystery
4 forex 6:01 am Mar 14, 2006
Validating robots.txt
Research on valid robots.txt and usage
4 smitha_ajay 5:18 am Mar 14, 2006
help.robots.txt/external link
disallow external link
7 dennis_oppelt 2:35 am Mar 14, 2006
more on robots and mirror sites
clarify please
3 dennis_oppelt 12:48 am Mar 14, 2006
robots.txt +ssi+.htaccess
3 OldWolf 5:40 pm Mar 13, 2006
How do I use a wild card for many pages?
wild card?
6 maha 1:38 am Mar 12, 2006
Elimination of pages from a website.
Do i have to notify the engines in some way?
9 Rani 12:26 am Mar 12, 2006
Robots.txt and htm - html
is disallow for htm killing pages that are html?
5 chewy 6:37 pm Mar 8, 2006
Newbie robots.txt question
Can a robot find content not referenced in pages on the site?
4 joykesilon 8:02 pm Mar 6, 2006
Disallowing double bot crawl due to addon domain
2 Alex_lon 8:04 pm Mar 2, 2006
Robots.txt Basic question
600 rules
3 omoutop 9:46 am Feb 27, 2006
robots.txt Help
disallow issue
3 malasorte 6:10 pm Feb 26, 2006
PHPBB Site
How do I get the bots to search my content instead of my intro page?
3 xmar 4:32 am Feb 25, 2006
Blocking Content On A New Site
New version of major website being launched.
2 caragh 3:04 pm Feb 23, 2006
htaccess Allow robots.txt and leech protection
3 war59312 3:30 am Feb 23, 2006
NimbleCrawler is Bad Bot, well worth banning.
NimbleCrawler is Bad Bot. Never reads robots.txt file.
10 d_mot 1:24 am Feb 23, 2006

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members