Welcome to WebmasterWorld Guest from 54.159.11.9

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Unusual log entries
5 cyberdyne 6:17 pm Mar 8, 2008
Redirect robot.txt from http to https
how to redirect properly.
8 clyde4210 2:59 pm Mar 6, 2008
Uh oh, it's happening again
robots.txt blocked out
2 wesg 9:58 pm Feb 28, 2008
new badbot: EnaBot/1.0
7 Achernar 8:31 am Feb 28, 2008
Need help blocking search results page
Lots of keyword search pages indexed, some with no content
4 ecmedia 2:28 pm Feb 26, 2008
disallow ALL bots.and then allow the good ones?
5 tokey666 8:40 pm Feb 25, 2008
Excluding a stub page with robots.txt
Dealing with trailing forward slash
4 bouncybunny 12:41 am Feb 24, 2008
What is wrong with my Robots.txt file?
I'd appreciate help with my Robots.txt file
7 Morthan 3:07 am Feb 22, 2008
Help with blocking URLS
5 jaffstar 7:17 am Feb 18, 2008
301 Redirect and Search Ranking
301 Redirect and Search Ranking
4 bilalseo 6:22 am Feb 18, 2008
No life without robots.txt file
This may be old hat to you but it knocked me for six
18 folkranger 7:17 pm Feb 15, 2008
Robots.txt and iframes
How to disallow iframes using robots.txt?
6 foxtunes 11:00 pm Feb 14, 2008
How to Handle Cname URL
Indexed in Google
7 experienced 7:47 am Feb 12, 2008
Google problem with robots.txt
Webmaster tools isn't helping.
21 wesg 8:16 pm Feb 5, 2008
*/? */index.*? is a must in robots.txt
I see /?widgetid=45 indexed
3 AjiNIMC 8:17 am Jan 30, 2008
Robots.txt (/index.jsp?pg name=ccpm/lob/sublob/pgname&site=a1)
Pattern on parameters
2 Surendra 12:20 am Jan 29, 2008
Denying access to everything but the home page?
Using robots.txt to only allow robot access to the home page
3 monkeylytics 8:00 pm Jan 28, 2008
Control Robots with Meta or robots.txt
3 driller41 12:21 pm Jan 15, 2008
Robots.txt disallowed.
12 g1smd 12:18 am Jan 15, 2008
robots.txt with wildcard?
3 tranceformer 12:39 am Jan 13, 2008
Disallowing with robots.txt
robots.txt, disallow certain urls
3 jim_knopf 12:36 am Jan 13, 2008
Getting correct user agent
getting from access log the correct user agent for robots.txt
2 hessodreamy 9:03 pm Jan 11, 2008
Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
4 timster 1:29 am Jan 4, 2008
404 Errors in Webmaster Console
What to do with these pages?
2 rowtc2 11:35 pm Jan 3, 2008
directory and subdirectory access
allowing access to subdirectory but not main directory
3 Mr_Cool_Dude 6:11 pm Jan 2, 2008

Featured Threads

Hot Threads This Week

Hot Threads This Month