Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Custom rule for google, url args and wordpress feeds
||6||Kurt||2:28 pm May 24, 2007|
|Wordpress - Robots.txt to fix Duplicate Content
I need a robots.txt file to prevent dup content from being crawled
|7||jman123||2:16 pm May 23, 2007|
issue with alexa bot & alexa toolbar users
|6||Michel_Samuel||10:45 am May 20, 2007|
Can you exclude a parameter in robots.txt?
|3||North_Shore||9:22 pm May 18, 2007|
||4||strawbleu||9:12 pm May 18, 2007|
||6||ajitaryan||11:35 am May 16, 2007|
|Is this valid?
||3||aha7||12:34 pm May 9, 2007|
|Disallowing content previously indexed
What happens to indexed content subsequently blocked by robots.txt
|3||daveozzz||11:22 am May 9, 2007|
|duplicate content indexing
||11||coburn||6:34 pm May 7, 2007|
|Denying one page with multiple variables
page.html?item=1, page.html?item=2, page.html?item=3
|7||wfernley||6:21 pm May 7, 2007|
|Can anyone post a complete robots.txt file here?
||4||PowerUp||2:29 pm May 7, 2007|
|How do I disallow bots from my Wordpress year Archive Folders
I guess this is a common question but I can't find an answer
|4||Drum||5:26 pm May 5, 2007|
|Add Sitemap To robots.txt For Autodiscovery
Ask.com, Google, Microsoft Live Search and Yahoo!  ( 1 2 )
|32||OldWolf||7:56 am May 4, 2007|
|How is alexa getting past my stoppers
stop alexa index
|1||Michel_Samuel||3:31 pm May 2, 2007|
|how to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
||8||d6rth7ader||5:48 am May 2, 2007|
||3||harish_s||2:17 pm May 1, 2007|
|Limiting the depth of Google's crawl?
Can robots.txt be used to limit depth
|6||daveozzz||7:21 pm Apr 27, 2007|
How did googlebot index my web server dir structure?
|3||thinkweb||5:24 am Apr 26, 2007|
|URLs restricted by robots.txt (41) AGH!
I'm a bit stumped about what's best to do re my robots.txt
|8||wrafter||8:38 pm Apr 25, 2007|
|Is a robots.txt file necessary?
if not trying to block bots?
|3||Jakes_Redding||5:52 am Apr 20, 2007|
|Googlebot ignoring robots.txt?
Or is my robots.txt file wrong...
|3||designaweb||9:28 am Apr 17, 2007|
|Blocking Google Only
From certain sub domains
|5||Moncao||7:25 pm Apr 13, 2007|
|4||kool002||1:59 pm Apr 11, 2007|
|Not being found by the search engines
|4||gtate||6:24 pm Apr 6, 2007|
|Efficient way to block multiple bots?
||11||JAB_Creations||2:33 pm Apr 3, 2007|