Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Add Sitemap To robots.txt For Autodiscovery
Ask.com, Google, Microsoft Live Search and Yahoo!  ( 1 2 )
|32||OldWolf||7:56 am May 4, 2007|
|How is alexa getting past my stoppers
stop alexa index
|1||Michel_Samuel||3:31 pm May 2, 2007|
|how to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
||8||d6rth7ader||5:48 am May 2, 2007|
||3||harish_s||2:17 pm May 1, 2007|
|Limiting the depth of Google's crawl?
Can robots.txt be used to limit depth
|6||daveozzz||7:21 pm Apr 27, 2007|
How did googlebot index my web server dir structure?
|3||thinkweb||5:24 am Apr 26, 2007|
|URLs restricted by robots.txt (41) AGH!
I'm a bit stumped about what's best to do re my robots.txt
|8||wrafter||8:38 pm Apr 25, 2007|
|Is a robots.txt file necessary?
if not trying to block bots?
|3||Jakes_Redding||5:52 am Apr 20, 2007|
|Googlebot ignoring robots.txt?
Or is my robots.txt file wrong...
|3||designaweb||9:28 am Apr 17, 2007|
|Blocking Google Only
From certain sub domains
|5||Moncao||7:25 pm Apr 13, 2007|
|4||kool002||1:59 pm Apr 11, 2007|
|Not being found by the search engines
|4||gtate||6:24 pm Apr 6, 2007|
|Efficient way to block multiple bots?
||11||JAB_Creations||2:33 pm Apr 3, 2007|
I need to block google from my index page
|4||loner||4:25 pm Apr 2, 2007|
||1||hugh||4:03 pm Apr 2, 2007|
|What's the Proper Way to Block Sub Folders
Looking for the best way to block 2nd level sub folders
|5||graywolf||7:43 pm Mar 28, 2007|
|ROBOT and PR rank?
how does the robot effect your pr rank?
|7||moppy56||1:48 pm Mar 28, 2007|
|Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
|5||SandySEO||8:49 pm Mar 27, 2007|
|Another bad bot to ban
this one is using 302 redirects .. bad bad
|1||Bewenched||3:43 am Mar 25, 2007|
|excluding a subdomain
Using the robots.txt file to exclude a subdomain
|10||joebray||9:47 am Mar 23, 2007|
|G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
|1||OnetSzukaj||2:58 pm Mar 22, 2007|
|How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
|1||incrediBILL||12:32 am Mar 22, 2007|
Anyone seen this?
|3||bobbarnes||10:28 pm Mar 16, 2007|
|problem with msnbot and wildcard matching
||2||Achernar||1:36 pm Mar 16, 2007|
|How Can I Make Bots Visit My Web Site?
||3||emadido||2:34 pm Mar 15, 2007|