Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|ROBOT and PR rank?
how does the robot effect your pr rank?
|7||moppy56||1:48 pm Mar 28, 2007|
|Blocking Partial Contents of Folder Without Using Robots Tag
10 files (7 want indexed, 3 want blocked)
|5||SandySEO||8:49 pm Mar 27, 2007|
|Another bad bot to ban
this one is using 302 redirects .. bad bad
|1||Bewenched||3:43 am Mar 25, 2007|
|excluding a subdomain
Using the robots.txt file to exclude a subdomain
|10||joebray||9:47 am Mar 23, 2007|
|G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
|1||OnetSzukaj||2:58 pm Mar 22, 2007|
|How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
|1||incrediBILL||12:32 am Mar 22, 2007|
Anyone seen this?
|3||bobbarnes||10:28 pm Mar 16, 2007|
|problem with msnbot and wildcard matching
||2||Achernar||1:36 pm Mar 16, 2007|
|How Can I Make Bots Visit My Web Site?
||3||emadido||2:34 pm Mar 15, 2007|
|why i can't reaply the topic
why i can't reaply the topic
|3||unknown||8:43 am Mar 14, 2007|
|robotsxx.txt not found.
||2||benevolent001||1:16 pm Mar 13, 2007|
|Something a little different
||2||futureX||1:56 pm Mar 12, 2007|
|Why is WebmasterWorld robot validator giving back HTML
||3||Clair||7:11 pm Mar 10, 2007|
|When Robots.txt And Meta Robots Collide
|13||Easy_Coder||1:57 am Mar 10, 2007|
|OSC, dupe content, robots.txt and 301 redirects
Need to ensure bots 301 to new SEO URL
|5||spina45||8:23 pm Mar 9, 2007|
|Disallow / access to our directory
|2||drew68||3:47 pm Mar 6, 2007|
|Block the main page of a directory but not its contents?
||6||bresso||7:39 am Mar 6, 2007|
|Robots.txt and pointing domains
||3||Shanada||6:58 am Feb 28, 2007|
|Will Googlebot be able to crawl these pages of my website?
||4||saqitude||3:40 pm Feb 25, 2007|
||2||seomack||8:14 am Feb 19, 2007|
|Is correct this robots.txt?
||2||MalastiC||9:44 pm Feb 18, 2007|
|Sitemap and robot.txt
||3||spuddy||3:53 pm Feb 15, 2007|
|How to exclude from crawling subdomain
exclude the crawling subdomain
|4||fendy||1:00 pm Feb 15, 2007|
|Confused over Google spider and robots.txt
||4||flowermark||12:56 pm Feb 15, 2007|
|Which bots to block?
|2||cyclesolutions||9:51 pm Feb 12, 2007|