Welcome to WebmasterWorld Guest from 50.19.62.95

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap
4 stapel 12:01 pm Feb 3, 2004
Is it illegal to ignore a robots.txt?
Could I sue google?
2 transactiongeek 6:44 am Feb 3, 2004
robots metatag
I have noindex,follow but the page is indexed
2 nevetS 4:41 am Feb 3, 2004
save and upload robots.txt
4 helenp 8:27 pm Feb 2, 2004
Is this robots.txt blocking anybody
8 nalavanje 2:20 pm Feb 2, 2004
Is this enough
5 dougie 11:20 am Feb 2, 2004
How Do I Code My Robots.txt File To Disallow Subdomians?
Is it Disallow: /apple.domain.com or Disallow: /apple/
2 Christyl_Stevens 9:34 am Feb 2, 2004
asterias/2.0
Looks for streaming media files...
6 pendanticist 11:43 pm Feb 1, 2004
will all robots crawl the sites without robot.txt?
robot crawl, robot.txt
2 johnlim 3:08 am Feb 1, 2004
What will Inktomi do in this scenario?
18 flobaby 6:02 am Jan 31, 2004
Google spidering past secure server
Robots.txt file to boot the bot?
2 xbase234 12:52 am Jan 31, 2004
Logs keeping showing 404 error
I have a robots file and it looks ok
3 eesnard 12:03 am Jan 30, 2004
need robots.txt for pages that are not linked?
robots.txt
5 rld3 5:09 pm Jan 29, 2004
SEO Robot.txt tag and robot meta tags... need advice
Help about robot.txt and robot meta tags
3 jamieabbott 2:09 pm Jan 28, 2004
Robots.txt returning "test"
Host has really messed me up
4 GodLikeLotus 10:37 am Jan 28, 2004
Need quick help
Prevent robots from clicking on PPC links
2 hostlead 11:11 am Jan 27, 2004
how to let robot crawl the links at a page but not index this page?
index, robot
2 johnlim 11:39 pm Jan 26, 2004
Wildcard use in robots.txt
Need to disallow spidering of graphics files
2 Probably 4:58 am Jan 26, 2004
robots.txt and htaccess
.htaccess circumvents to site root
4 stefan009 5:35 pm Jan 24, 2004
Altavista Scooter eats up too much trafic
2 maxidrom11 5:28 am Jan 24, 2004
Does every site need a robots.txt?
I have one, but I'm not sure I need it
2 MatthewHSE 4:00 pm Jan 23, 2004
creating robots.txt
5 artdog 12:21 pm Jan 23, 2004
My page has never been indexed
2 kknusa 6:45 am Jan 23, 2004
Subdomains
2 BBB_Guy 6:38 am Jan 23, 2004
Indexed pages that are disalowed by robots.txt
is this normal?
10 WebJoe 7:03 pm Jan 20, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month