Welcome to WebmasterWorld Guest from 54.196.116.152

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Need help blocking search results page
Lots of keyword search pages indexed, some with no content
4 ecmedia 2:28 pm Feb 26, 2008
disallow ALL bots.and then allow the good ones?
5 tokey666 8:40 pm Feb 25, 2008
Excluding a stub page with robots.txt
Dealing with trailing forward slash
4 bouncybunny 12:41 am Feb 24, 2008
What is wrong with my Robots.txt file?
I'd appreciate help with my Robots.txt file
7 Morthan 3:07 am Feb 22, 2008
Help with blocking URLS
5 jaffstar 7:17 am Feb 18, 2008
301 Redirect and Search Ranking
301 Redirect and Search Ranking
4 bilalseo 6:22 am Feb 18, 2008
No life without robots.txt file
This may be old hat to you but it knocked me for six
18 folkranger 7:17 pm Feb 15, 2008
Robots.txt and iframes
How to disallow iframes using robots.txt?
6 foxtunes 11:00 pm Feb 14, 2008
How to Handle Cname URL
Indexed in Google
7 experienced 7:47 am Feb 12, 2008
Google problem with robots.txt
Webmaster tools isn't helping.
21 wesg 8:16 pm Feb 5, 2008
*/? */index.*? is a must in robots.txt
I see /?widgetid=45 indexed
3 AjiNIMC 8:17 am Jan 30, 2008
Robots.txt (/index.jsp?pg name=ccpm/lob/sublob/pgname&site=a1)
Pattern on parameters
2 Surendra 12:20 am Jan 29, 2008
Denying access to everything but the home page?
Using robots.txt to only allow robot access to the home page
3 monkeylytics 8:00 pm Jan 28, 2008
Control Robots with Meta or robots.txt
3 driller41 12:21 pm Jan 15, 2008
Robots.txt disallowed.
12 g1smd 12:18 am Jan 15, 2008
robots.txt with wildcard?
3 tranceformer 12:39 am Jan 13, 2008
Disallowing with robots.txt
robots.txt, disallow certain urls
3 jim_knopf 12:36 am Jan 13, 2008
Getting correct user agent
getting from access log the correct user agent for robots.txt
2 hessodreamy 9:03 pm Jan 11, 2008
Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
4 timster 1:29 am Jan 4, 2008
404 Errors in Webmaster Console
What to do with these pages?
2 rowtc2 11:35 pm Jan 3, 2008
directory and subdirectory access
allowing access to subdirectory but not main directory
3 Mr_Cool_Dude 6:11 pm Jan 2, 2008
robots.txt question
robots.txt examples
2 jim_knopf 5:43 pm Jan 2, 2008
Reference robots.txt file?
Does anyone have one?
3 mmiller 11:30 pm Dec 25, 2007
how to manage cname in robot
don't was to be indexed
4 experienced 2:10 pm Dec 21, 2007
disallow vs ban in .htaccess
12 joergnw10 11:10 am Dec 16, 2007