Welcome to WebmasterWorld Guest from 54.205.210.125

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Do we need robots at all? New member here
Confused as to whether I need to add robots txt
5 plenusvita 9:23 pm Dec 2, 2006
Block all spiders, but only allow well known spiders?
2 foxfox 3:19 pm Dec 2, 2006
msnbot visits robots.txt and exits
7 pemba76 12:13 pm Nov 28, 2006
Upload of incorrect robots.txt
What a screw up - any permanent damage?
4 TinkyWinky 10:12 am Nov 28, 2006
Does excluding Googlebot create a penalty?
If other bots are allowed, is this a red flag to Google
2 AndyA 9:26 pm Nov 27, 2006
what to put into robots.txt?
11 Hitman3266 5:09 pm Nov 27, 2006
How long does it take for SEs to learn about a robots'txt
4 bouncybunny 12:04 pm Nov 20, 2006
Yahoo! Slurp Now Supports Wildcards in robots.txt
17 OldWolf 2:54 am Nov 20, 2006
Robots.txt and password-protection
3 hwy2hel 10:56 pm Nov 16, 2006
Yahoo robots txt
nedd to disallow all urls with?
2 cheaperholidays 4:54 pm Nov 16, 2006
robots.txt? OR .htaccess?
disallowing or protecting directories
2 johnjoyce 4:27 pm Nov 16, 2006
Wildcards in robots.txt for forum software
6 Asia_Expat 1:40 am Nov 14, 2006
robots.txt syntax
'$' and '?'
2 sssweb 8:40 pm Nov 11, 2006
robots.txt help
2 miracle 7:16 pm Nov 8, 2006
Link Checker / Site map generator
One that observes robots.txt and meta tags
5 fish_eye 5:33 am Nov 7, 2006
Disallow by ISP
Where to get a list of ISPs and the countries they are in?
1 youfoundjake 7:13 pm Nov 3, 2006
Made the mistake of putting sub folders I want allowed
disallow everthing but?
2 Sunnyvale 7:07 pm Oct 31, 2006
Google remove url tool for supplemental urls a waste of time?
Should I bother removing duplicate supplementals?
3 Tonerman 4:20 pm Oct 30, 2006
allow major engines, block the fleas
and to add further control
5 youfoundjake 5:16 pm Oct 26, 2006
Dissallowing Dynamic Content
Blocking "index.php?"
2 trillianjedi 6:58 pm Oct 25, 2006
Combining htaccess and robots.txt?
7 Lady18 11:37 pm Oct 24, 2006
disallowing a 'staging server'
2 rden17 4:11 pm Oct 24, 2006
.htaccess
2 ssgumby 9:53 pm Oct 21, 2006
cloaking robots.txt
to avoid hacking
4 idolw 12:26 pm Oct 18, 2006
How to deal with a robots.txt hack attack
Guidelines to mitigate the damage from an attack
3 Whitey 10:17 pm Oct 17, 2006

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members