Welcome to WebmasterWorld Guest from 54.221.73.186

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Only those on the list are allowed entrance
All others OUT
7 grnidone 5:29 pm Jan 2, 2004
txt text for website
txt
4 krikri 5:11 pm Jan 2, 2004
Check my robots.txt please
Don't want to block adsense bot
3 conroy 2:10 pm Jan 2, 2004
How to ban YahooSeeker?
Used robots file, but it doesn't obey.
2 zf007 3:59 am Jan 1, 2004
deny a spider from visiting a single page
3 Michael_Weir 3:08 pm Dec 29, 2003
Disallow indexing on dynamic files
Disallow indexing on dynamic files
6 premzero 2:58 am Dec 26, 2003
banning "vscooter" but allowing "scooter"
4 keyplyr 9:15 pm Dec 25, 2003
robots.txt not showing...
4 mubarak_zeb 3:37 pm Dec 19, 2003
Getting 403 Forbidden for robots.txt
Can anyone help
11 GodLikeLotus 4:59 am Dec 18, 2003
robots.txt use and hosting provider problem
robots.txt use and hosting provider problem
2 david_cool 6:55 pm Dec 16, 2003
robots.txt on phpbb forum
2 RammsteinNicCage 4:06 pm Dec 12, 2003
Spiders get / and /robots.txt but no more!
Is my robots.txt killing me?
6 dwilson 8:55 pm Dec 11, 2003
[404] HTTP Page robots.txt not found 404 error
4 Strike 12:44 pm Dec 10, 2003
changing robots.txt
replacing robots.txt
2 darex 8:24 pm Dec 9, 2003
Can we prevent a form link from being spidered?
form href prevent spider indexing
3 Frannie 2:32 pm Dec 9, 2003
Robots and directory hierachies
4 finlay 5:57 pm Dec 8, 2003
wildcard disallow for pages with varibles
would this work for pages ending with?keep= : Disallow: *keep=*?
2 The_Rabbit 5:25 pm Dec 8, 2003
disallowing based on part of a query string
3 jibbajabba 4:39 pm Dec 2, 2003
Whois.sc Probe/Bot
Prevent Snapshot Cache
1 CustomFit 3:41 am Dec 2, 2003
Hiding robots.txt
21 keyplyr 1:45 am Dec 2, 2003
Did you re-test?
3 hybridx 2:51 pm Nov 28, 2003
Weird Code Return from Googlebot when robots.txt was spidered
3 5x54u 9:14 pm Nov 25, 2003
Problem with robots.txt
4 Nisse 8:08 am Nov 25, 2003
robots and dynamic url
How to ban robots from a dynamic url
3 Ledfish 8:09 pm Nov 23, 2003
Googlebot Wildcard
5 payday 11:26 am Nov 22, 2003

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members