Welcome to WebmasterWorld Guest from 54.162.165.158

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Is there any bots to block?
Else than the infamous "Web Content International"?
3 Daemon 11:56 pm Jan 16, 2004
Robots.txt help
5 mikemcs 1:04 pm Jan 16, 2004
Who are these spiders and what do they want?
7 obsos 4:51 pm Jan 14, 2004
Revisit instruction for robots
I want to specify different revisit instruction different sections
3 adityamodi 5:13 am Jan 10, 2004
Have I been blocking Altavista spider from spidering my site?
6 irock 6:10 pm Jan 9, 2004
robot.txt to remove an alias
Afraid of duplicated content
2 Ulkari 5:48 pm Jan 8, 2004
Spider Names
I need the names of some SEs usernames
3 The_Young_One 2:47 am Jan 8, 2004
Blocking Includes (SSI)
blocking ssi
3 direktor1 4:03 pm Jan 7, 2004
Subdomains
How to exclude
4 gosman 2:54 pm Jan 6, 2004
Does the leading / make any difference?
Sorry, I'm new to robots.txt
2 AsleepATheWheel 1:07 am Jan 6, 2004
faulty robots.txt?
5 tenaka 2:13 pm Jan 3, 2004
Only those on the list are allowed entrance
All others OUT
7 grnidone 5:29 pm Jan 2, 2004
txt text for website
txt
4 krikri 5:11 pm Jan 2, 2004
Check my robots.txt please
Don't want to block adsense bot
3 conroy 2:10 pm Jan 2, 2004
How to ban YahooSeeker?
Used robots file, but it doesn't obey.
2 zf007 3:59 am Jan 1, 2004
deny a spider from visiting a single page
3 Michael_Weir 3:08 pm Dec 29, 2003
Disallow indexing on dynamic files
Disallow indexing on dynamic files
6 premzero 2:58 am Dec 26, 2003
banning "vscooter" but allowing "scooter"
4 keyplyr 9:15 pm Dec 25, 2003
robots.txt not showing...
4 mubarak_zeb 3:37 pm Dec 19, 2003
Getting 403 Forbidden for robots.txt
Can anyone help
11 GodLikeLotus 4:59 am Dec 18, 2003
robots.txt use and hosting provider problem
robots.txt use and hosting provider problem
2 david_cool 6:55 pm Dec 16, 2003
robots.txt on phpbb forum
2 RammsteinNicCage 4:06 pm Dec 12, 2003
Spiders get / and /robots.txt but no more!
Is my robots.txt killing me?
6 dwilson 8:55 pm Dec 11, 2003
[404] HTTP Page robots.txt not found 404 error
4 Strike 12:44 pm Dec 10, 2003
changing robots.txt
replacing robots.txt
2 darex 8:24 pm Dec 9, 2003

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members