Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap
|4||stapel||12:01 pm Feb 3, 2004|
|Is it illegal to ignore a robots.txt?
Could I sue google?
|2||transactiongeek||6:44 am Feb 3, 2004|
I have noindex,follow but the page is indexed
|2||nevetS||4:41 am Feb 3, 2004|
|save and upload robots.txt
||4||helenp||8:27 pm Feb 2, 2004|
|Is this robots.txt blocking anybody
||8||nalavanje||2:20 pm Feb 2, 2004|
|Is this enough
||5||dougie||11:20 am Feb 2, 2004|
|How Do I Code My Robots.txt File To Disallow Subdomians?
Is it Disallow: /apple.domain.com or Disallow: /apple/
|2||Christyl_Stevens||9:34 am Feb 2, 2004|
Looks for streaming media files...
|6||pendanticist||11:43 pm Feb 1, 2004|
|will all robots crawl the sites without robot.txt?
robot crawl, robot.txt
|2||johnlim||3:08 am Feb 1, 2004|
|What will Inktomi do in this scenario?
||18||flobaby||6:02 am Jan 31, 2004|
|Google spidering past secure server
Robots.txt file to boot the bot?
|2||xbase234||12:52 am Jan 31, 2004|
|Logs keeping showing 404 error
I have a robots file and it looks ok
|3||eesnard||12:03 am Jan 30, 2004|
|need robots.txt for pages that are not linked?
|5||rld3||5:09 pm Jan 29, 2004|
|SEO Robot.txt tag and robot meta tags... need advice
Help about robot.txt and robot meta tags
|3||jamieabbott||2:09 pm Jan 28, 2004|
|Robots.txt returning "test"
Host has really messed me up
|4||GodLikeLotus||10:37 am Jan 28, 2004|
|Need quick help
Prevent robots from clicking on PPC links
|2||hostlead||11:11 am Jan 27, 2004|
|how to let robot crawl the links at a page but not index this page?
|2||johnlim||11:39 pm Jan 26, 2004|
|Wildcard use in robots.txt
Need to disallow spidering of graphics files
|2||Probably||4:58 am Jan 26, 2004|
|robots.txt and htaccess
.htaccess circumvents to site root
|4||stefan009||5:35 pm Jan 24, 2004|
|Altavista Scooter eats up too much trafic
||2||maxidrom11||5:28 am Jan 24, 2004|
|Does every site need a robots.txt?
I have one, but I'm not sure I need it
|2||MatthewHSE||4:00 pm Jan 23, 2004|
||5||artdog||12:21 pm Jan 23, 2004|
|My page has never been indexed
||2||kknusa||6:45 am Jan 23, 2004|
||2||BBB_Guy||6:38 am Jan 23, 2004|
|Indexed pages that are disalowed by robots.txt
is this normal?
|10||WebJoe||7:03 pm Jan 20, 2004|