Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|G. ignores robots.txt at other ports, gives false page count
Google shows in its index pages that it never saw, even prohibited
|1||OnetSzukaj||2:58 pm Mar 22, 2007|
|How to use Robots.txt as an INCLUSION method
It's exclusion only if you use it that way
|1||incrediBILL||12:32 am Mar 22, 2007|
Anyone seen this?
|3||bobbarnes||10:28 pm Mar 16, 2007|
|problem with msnbot and wildcard matching
||2||Achernar||1:36 pm Mar 16, 2007|
|How Can I Make Bots Visit My Web Site?
||3||emadido||2:34 pm Mar 15, 2007|
|why i can't reaply the topic
why i can't reaply the topic
|3||unknown||8:43 am Mar 14, 2007|
|robotsxx.txt not found.
||2||benevolent001||1:16 pm Mar 13, 2007|
|Something a little different
||2||futureX||1:56 pm Mar 12, 2007|
|Why is WebmasterWorld robot validator giving back HTML
||3||Clair||7:11 pm Mar 10, 2007|
|When Robots.txt And Meta Robots Collide
|13||Easy_Coder||1:57 am Mar 10, 2007|
|OSC, dupe content, robots.txt and 301 redirects
Need to ensure bots 301 to new SEO URL
|5||spina45||8:23 pm Mar 9, 2007|
|Disallow / access to our directory
|2||drew68||3:47 pm Mar 6, 2007|
|Block the main page of a directory but not its contents?
||6||bresso||7:39 am Mar 6, 2007|
|Robots.txt and pointing domains
||3||Shanada||6:58 am Feb 28, 2007|
|Will Googlebot be able to crawl these pages of my website?
||4||saqitude||3:40 pm Feb 25, 2007|
||2||seomack||8:14 am Feb 19, 2007|
|Is correct this robots.txt?
||2||MalastiC||9:44 pm Feb 18, 2007|
|Sitemap and robot.txt
||3||spuddy||3:53 pm Feb 15, 2007|
|How to exclude from crawling subdomain
exclude the crawling subdomain
|4||fendy||1:00 pm Feb 15, 2007|
|Confused over Google spider and robots.txt
||4||flowermark||12:56 pm Feb 15, 2007|
|Which bots to block?
|2||cyclesolutions||9:51 pm Feb 12, 2007|
|Accidentally blocked root access, doh!
||3||ninjaNeen||6:51 pm Feb 8, 2007|
|robots.txt usage and examples
Using robots.txt to improve SEO
|3||cduke250||3:50 pm Feb 7, 2007|
|disallowing subdomain access rule
||1||coburn||1:37 pm Feb 7, 2007|
|can put two robots at one line at robot.txt to ban two spiders?
robots.txt, ban google, yahoo bots
|4||johnlim9988||3:54 am Feb 7, 2007|