Welcome to WebmasterWorld Guest from 54.145.208.64

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robot.txt to remove an alias
Afraid of duplicated content
2 Ulkari 5:48 pm Jan 8, 2004
Spider Names
I need the names of some SEs usernames
3 The_Young_One 2:47 am Jan 8, 2004
Blocking Includes (SSI)
blocking ssi
3 direktor1 4:03 pm Jan 7, 2004
Subdomains
How to exclude
4 gosman 2:54 pm Jan 6, 2004
Does the leading / make any difference?
Sorry, I'm new to robots.txt
2 AsleepATheWheel 1:07 am Jan 6, 2004
faulty robots.txt?
5 tenaka 2:13 pm Jan 3, 2004
Only those on the list are allowed entrance
All others OUT
7 grnidone 5:29 pm Jan 2, 2004
txt text for website
txt
4 krikri 5:11 pm Jan 2, 2004
Check my robots.txt please
Don't want to block adsense bot
3 conroy 2:10 pm Jan 2, 2004
How to ban YahooSeeker?
Used robots file, but it doesn't obey.
2 zf007 3:59 am Jan 1, 2004
deny a spider from visiting a single page
3 Michael_Weir 3:08 pm Dec 29, 2003
Disallow indexing on dynamic files
Disallow indexing on dynamic files
6 premzero 2:58 am Dec 26, 2003
banning "vscooter" but allowing "scooter"
4 keyplyr 9:15 pm Dec 25, 2003
robots.txt not showing...
4 mubarak_zeb 3:37 pm Dec 19, 2003
Getting 403 Forbidden for robots.txt
Can anyone help
11 GodLikeLotus 4:59 am Dec 18, 2003
robots.txt use and hosting provider problem
robots.txt use and hosting provider problem
2 david_cool 6:55 pm Dec 16, 2003
robots.txt on phpbb forum
2 RammsteinNicCage 4:06 pm Dec 12, 2003
Spiders get / and /robots.txt but no more!
Is my robots.txt killing me?
6 dwilson 8:55 pm Dec 11, 2003
[404] HTTP Page robots.txt not found 404 error
4 Strike 12:44 pm Dec 10, 2003
changing robots.txt
replacing robots.txt
2 darex 8:24 pm Dec 9, 2003
Can we prevent a form link from being spidered?
form href prevent spider indexing
3 Frannie 2:32 pm Dec 9, 2003
Robots and directory hierachies
4 finlay 5:57 pm Dec 8, 2003
wildcard disallow for pages with varibles
would this work for pages ending with?keep= : Disallow: *keep=*?
2 The_Rabbit 5:25 pm Dec 8, 2003
disallowing based on part of a query string
3 jibbajabba 4:39 pm Dec 2, 2003
Whois.sc Probe/Bot
Prevent Snapshot Cache
1 CustomFit 3:41 am Dec 2, 2003