Welcome to WebmasterWorld Guest from 23.23.46.20

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
I don't have a robots.txt file..error 404
Can't get good Google or Yahoo placement
6 jagbitsllc 2:17 am Feb 11, 2004
WebmasterWorldForumBot?
what is this?
2 Trisha 10:33 pm Feb 10, 2004
How to stop robots on .php files
3 djtaverner 6:40 pm Feb 10, 2004
What in the world is a robot.txt?
I need this to be stated simple for me
5 Claudia_T 6:35 pm Feb 10, 2004
Help! I need to disallow a specific query string
1 Hissingsid 11:26 am Feb 10, 2004
Please verify
2 mikemcs 4:25 am Feb 10, 2004
Googlebot chasing .php tracking script
is robots.txt enough?
2 Edouard_H 12:51 am Feb 9, 2004
301 redirects & robots.txt
What is the implication of a 301 redirect on robots.txt
2 Jerry_Pannu 10:17 pm Feb 7, 2004
Testing robots.txt
Why won't my site get spidered
2 DavidAtWork 5:56 pm Feb 5, 2004
I've been looking at the WW robots.txt ...
...but why are these spiders disallowed?
7 elgumbo 1:32 pm Feb 4, 2004
GoogleBot
Will using robots exclusion and meta exclusion hurt me?
1 mercury 9:12 pm Feb 3, 2004
Spiders that ignore or skip robots.txt
Can bad spiders be identified?
4 DavidAtWork 2:05 pm Feb 3, 2004
Robots Meta and .txt
Where?
4 admin_amoroustoys 1:34 pm Feb 3, 2004
robots.txt used in court case
2 transactiongeek 1:27 pm Feb 3, 2004
what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap
4 stapel 12:01 pm Feb 3, 2004
Is it illegal to ignore a robots.txt?
Could I sue google?
2 transactiongeek 6:44 am Feb 3, 2004
robots metatag
I have noindex,follow but the page is indexed
2 nevetS 4:41 am Feb 3, 2004
save and upload robots.txt
4 helenp 8:27 pm Feb 2, 2004
Is this robots.txt blocking anybody
8 nalavanje 2:20 pm Feb 2, 2004
Is this enough
5 dougie 11:20 am Feb 2, 2004
How Do I Code My Robots.txt File To Disallow Subdomians?
Is it Disallow: /apple.domain.com or Disallow: /apple/
2 Christyl_Stevens 9:34 am Feb 2, 2004
asterias/2.0
Looks for streaming media files...
6 pendanticist 11:43 pm Feb 1, 2004
will all robots crawl the sites without robot.txt?
robot crawl, robot.txt
2 johnlim 3:08 am Feb 1, 2004
What will Inktomi do in this scenario?
18 flobaby 6:02 am Jan 31, 2004
Google spidering past secure server
Robots.txt file to boot the bot?
2 xbase234 12:52 am Jan 31, 2004

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month