Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Is a robots.txt file necessary?
if not trying to block bots?
|3||Jakes_Redding||5:52 am Apr 20, 2007|
|Googlebot ignoring robots.txt?
Or is my robots.txt file wrong...
|3||designaweb||9:28 am Apr 17, 2007|
|Blocking Google Only
From certain sub domains
|5||Moncao||7:25 pm Apr 13, 2007|
|4||kool002||1:59 pm Apr 11, 2007|
|Not being found by the search engines
|4||gtate||6:24 pm Apr 6, 2007|
|Efficient way to block multiple bots?
||11||JAB_Creations||2:33 pm Apr 3, 2007|
I need to block google from my index page
|4||loner||4:25 pm Apr 2, 2007|
||1||hugh||4:03 pm Apr 2, 2007|
|What's the Proper Way to Block Sub Folders
Looking for the best way to block 2nd level sub folders
|5||graywolf||7:43 pm Mar 28, 2007|
|ROBOT and PR rank?
how does the robot effect your pr rank?
|7||moppy56||1:48 pm Mar 28, 2007|
|Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
|5||SandySEO||8:49 pm Mar 27, 2007|
|Another bad bot to ban
this one is using 302 redirects .. bad bad
|1||Bewenched||3:43 am Mar 25, 2007|
|excluding a subdomain
Using the robots.txt file to exclude a subdomain
|10||joebray||9:47 am Mar 23, 2007|
|G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
|1||OnetSzukaj||2:58 pm Mar 22, 2007|
|How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
|1||incrediBILL||12:32 am Mar 22, 2007|
Anyone seen this?
|3||bobbarnes||10:28 pm Mar 16, 2007|
|problem with msnbot and wildcard matching
||2||Achernar||1:36 pm Mar 16, 2007|
|How Can I Make Bots Visit My Web Site?
||3||emadido||2:34 pm Mar 15, 2007|
|why i can't reaply the topic
why i can't reaply the topic
|3||unknown||8:43 am Mar 14, 2007|
|robotsxx.txt not found.
||2||benevolent001||1:16 pm Mar 13, 2007|
|Something a little different
||2||futureX||1:56 pm Mar 12, 2007|
|Why is WebmasterWorld robot validator giving back HTML
||3||Clair||7:11 pm Mar 10, 2007|
|When Robots.txt And Meta Robots Collide
|13||Easy_Coder||1:57 am Mar 10, 2007|
|OSC, dupe content, robots.txt and 301 redirects
Need to ensure bots 301 to new SEO URL
|5||spina45||8:23 pm Mar 9, 2007|
|Disallow / access to our directory
|2||drew68||3:47 pm Mar 6, 2007|