Welcome to WebmasterWorld Guest from 54.167.155.147

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
mod_rewrite and robots.txt
2 jewels 3:03 pm May 3, 2005
Blocking Yahoo from a specific folder
How do I do this?
2 Rollo 2:55 pm May 3, 2005
How to Validate robots.txt and do I even have one?
13 neophitemoneymaker 11:58 pm May 1, 2005
Page indexed in Google despite robots.txt entry
How did they find it in the first place?
23 pmkpmk 9:17 pm May 1, 2005
The Great SearchEngineWorld robots.txt Survey #2 is Under Way
9 Brett_Tabke 2:35 pm Apr 29, 2005
No-Follow, allow indexing of a file but not it's links?
2 JAB_Creations 8:32 am Apr 27, 2005
How to stop Yahoo Slurp?
Slurp ingnores my robots.txt
16 atlrus 9:47 pm Apr 26, 2005
How to remove ezboard forum from Wayback?
3 wmw99 12:04 pm Apr 26, 2005
Is there a definitive robots.txt to disallow bad bots
bad bots and bad bots only
6 Clark 11:40 am Apr 24, 2005
Disallow keyword Ignored?
Who guarantees me that disallow keyword isn't ignored
5 bkausbk 9:02 am Apr 24, 2005
Allowing and disallowing spiders
Easy one - can somebody confirm my understanding?
3 Zeberdy 9:19 am Apr 21, 2005
How to disallow getlastpost
2 annelisabeth 9:19 am Apr 21, 2005
Robots.txt Help request.
I need help with stopping spyder robots.
10 westmore 6:22 am Apr 21, 2005
Meta Descriptions
How many can I have?
8 Andromeda 8:07 pm Apr 20, 2005
Robots.txt Review
I need a confirmation for robots.txt from you experts
11 RaniaD 11:51 am Apr 20, 2005
I want to disallow extra for Slurp
is this syntax right?
2 baze22 12:55 pm Apr 19, 2005
Robots.txt vs Robots META tag
Does one over-rule the other?
3 obsos 7:14 am Apr 18, 2005
Validation of robots.txt
3 designei 10:50 am Apr 17, 2005
Help w/ robots.txt Google won't toch it
robots.txt google
2 JackieB 11:04 am Apr 16, 2005
What was wrong with this robots.txt if anything
4 montenegro 9:25 am Apr 16, 2005
block cgi bin on a virtual host
cgi bin is not in root directory
3 Reid 6:26 am Apr 16, 2005
Block specific words from being indexed
13 Jer1024 7:34 pm Apr 15, 2005
Allowing robots to index secure directories in root
13 naomi 2:46 pm Apr 14, 2005
Block certain links
2 liviu 1:13 pm Apr 14, 2005
Alexa bot
Do I NEED robots.txt?
3 TheRookie 11:47 pm Apr 12, 2005