Welcome to WebmasterWorld Guest from 54.198.222.129

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Inktomi Spider Behavior
what does this mean
5 graywolf 12:31 am Feb 17, 2004
Need help making robots to block all spiders
9 budbiss 7:47 pm Feb 16, 2004
Selective bot blocking
Scanning rules
2 fabfurs 11:43 am Feb 16, 2004
Do I need to
2 ronhollin 6:46 am Feb 16, 2004
How important is having a properly configured robots?
4 budbiss 3:45 pm Feb 14, 2004
DOS Line Enders
How to download sample robots.txt files
3 webman2 10:30 am Feb 14, 2004
robots.txt and template pages
Does blocking templates prevent spiders from following links?
2 butterfi 12:59 am Feb 12, 2004
disallow is allowing
Is this done correctly?
1 DavidAtWork 6:25 pm Feb 11, 2004
I don't have a robots.txt file..error 404
Can't get good Google or Yahoo placement
6 jagbitsllc 2:17 am Feb 11, 2004
WebmasterWorldForumBot?
what is this?
2 Trisha 10:33 pm Feb 10, 2004
How to stop robots on .php files
3 djtaverner 6:40 pm Feb 10, 2004
What in the world is a robot.txt?
I need this to be stated simple for me
5 Claudia_T 6:35 pm Feb 10, 2004
Help! I need to disallow a specific query string
1 Hissingsid 11:26 am Feb 10, 2004
Please verify
2 mikemcs 4:25 am Feb 10, 2004
Googlebot chasing .php tracking script
is robots.txt enough?
2 Edouard_H 12:51 am Feb 9, 2004
301 redirects & robots.txt
What is the implication of a 301 redirect on robots.txt
2 Jerry_Pannu 10:17 pm Feb 7, 2004
Testing robots.txt
Why won't my site get spidered
2 DavidAtWork 5:56 pm Feb 5, 2004
I've been looking at the WW robots.txt ...
...but why are these spiders disallowed?
7 elgumbo 1:32 pm Feb 4, 2004
GoogleBot
Will using robots exclusion and meta exclusion hurt me?
1 mercury 9:12 pm Feb 3, 2004
Spiders that ignore or skip robots.txt
Can bad spiders be identified?
4 DavidAtWork 2:05 pm Feb 3, 2004
Robots Meta and .txt
Where?
4 admin_amoroustoys 1:34 pm Feb 3, 2004
robots.txt used in court case
2 transactiongeek 1:27 pm Feb 3, 2004
what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap
4 stapel 12:01 pm Feb 3, 2004
Is it illegal to ignore a robots.txt?
Could I sue google?
2 transactiongeek 6:44 am Feb 3, 2004
robots metatag
I have noindex,follow but the page is indexed
2 nevetS 4:41 am Feb 3, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month