Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Redirect robot.txt from http to https
how to redirect properly.
|8||clyde4210||2:59 pm Mar 6, 2008|
|Uh oh, it's happening again
robots.txt blocked out
|2||wesg||9:58 pm Feb 28, 2008|
|new badbot: EnaBot/1.0
||7||Achernar||8:31 am Feb 28, 2008|
|Need help blocking search results page
Lots of keyword search pages indexed, some with no content
|4||ecmedia||2:28 pm Feb 26, 2008|
|disallow ALL bots.and then allow the good ones?
||5||tokey666||8:40 pm Feb 25, 2008|
|Excluding a stub page with robots.txt
Dealing with trailing forward slash
|4||bouncybunny||12:41 am Feb 24, 2008|
|What is wrong with my Robots.txt file?
I'd appreciate help with my Robots.txt file
|7||Morthan||3:07 am Feb 22, 2008|
|Help with blocking URLS
||5||jaffstar||7:17 am Feb 18, 2008|
|301 Redirect and Search Ranking
301 Redirect and Search Ranking
|4||bilalseo||6:22 am Feb 18, 2008|
|No life without robots.txt file
This may be old hat to you but it knocked me for six
|18||folkranger||7:17 pm Feb 15, 2008|
|Robots.txt and iframes
How to disallow iframes using robots.txt?
|6||foxtunes||11:00 pm Feb 14, 2008|
|How to Handle Cname URL
Indexed in Google
|7||experienced||7:47 am Feb 12, 2008|
|Google problem with robots.txt
Webmaster tools isn't helping.
|21||wesg||8:16 pm Feb 5, 2008|
|*/? */index.*? is a must in robots.txt
I see /?widgetid=45 indexed
|3||AjiNIMC||8:17 am Jan 30, 2008|
|Robots.txt (/index.jsp?pg name=ccpm/lob/sublob/pgname&site=a1)
Pattern on parameters
|2||Surendra||12:20 am Jan 29, 2008|
|Denying access to everything but the home page?
Using robots.txt to only allow robot access to the home page
|3||monkeylytics||8:00 pm Jan 28, 2008|
|Control Robots with Meta or robots.txt
||3||driller41||12:21 pm Jan 15, 2008|
||12||g1smd||12:18 am Jan 15, 2008|
|robots.txt with wildcard?
||3||tranceformer||12:39 am Jan 13, 2008|
|Disallowing with robots.txt
robots.txt, disallow certain urls
|3||jim_knopf||12:36 am Jan 13, 2008|
|Getting correct user agent
getting from access log the correct user agent for robots.txt
|2||hessodreamy||9:03 pm Jan 11, 2008|
|Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
|4||timster||1:29 am Jan 4, 2008|
|404 Errors in Webmaster Console
What to do with these pages?
|2||rowtc2||11:35 pm Jan 3, 2008|
|directory and subdirectory access
allowing access to subdirectory but not main directory
|3||Mr_Cool_Dude||6:11 pm Jan 2, 2008|
|2||jim_knopf||5:43 pm Jan 2, 2008|