Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||2||miracle||7:16 pm Nov 8, 2006|
|Link Checker / Site map generator
One that observes robots.txt and meta tags
|5||fish_eye||5:33 am Nov 7, 2006|
|Disallow by ISP
Where to get a list of ISPs and the countries they are in?
|1||youfoundjake||7:13 pm Nov 3, 2006|
|Made the mistake of putting sub folders I want allowed
disallow everthing but?
|2||Sunnyvale||7:07 pm Oct 31, 2006|
|Google remove url tool for supplemental urls a waste of time?
Should I bother removing duplicate supplementals?
|3||Tonerman||4:20 pm Oct 30, 2006|
|allow major engines, block the fleas
and to add further control
|5||youfoundjake||5:16 pm Oct 26, 2006|
|Dissallowing Dynamic Content
|2||trillianjedi||6:58 pm Oct 25, 2006|
|Combining htaccess and robots.txt?
||7||Lady18||11:37 pm Oct 24, 2006|
|disallowing a 'staging server'
||2||rden17||4:11 pm Oct 24, 2006|
||2||ssgumby||9:53 pm Oct 21, 2006|
to avoid hacking
|4||idolw||12:26 pm Oct 18, 2006|
|How to deal with a robots.txt hack attack
Guidelines to mitigate the damage from an attack
|3||Whitey||10:17 pm Oct 17, 2006|
this bot is ripping through our site
|2||ssgumby||6:36 pm Oct 14, 2006|
|Google indexing weird links
ones with // in them
|4||netchicken1||12:54 am Oct 13, 2006|
|Please welcome your new moderator: goodroi
||20||engine||6:35 am Oct 11, 2006|
|robots.txt problem and subdirs / wildcards
||2||disgust||1:44 pm Oct 10, 2006|
|Google Sitemaps Robots.txt Question
||2||Tonerman||5:31 pm Oct 9, 2006|
|Naming da' bots
Same place, different face
|1||Reno||11:58 am Oct 7, 2006|
Anyone seen this bot before?
|2||jexx||7:44 pm Oct 3, 2006|
|Disallow Coldfusion Pages Only
Stop cfm pages from being crawled
|2||stcrim||3:18 pm Oct 3, 2006|
|Removing specific file type
||8||MisterT||11:40 pm Sep 28, 2006|
|Robots - Allowing them
rather than disallow
|2||SanDiegoPaul||10:30 pm Sep 28, 2006|
|Robots.txt & meta
Robots meta tage or just a robots.txt
|4||pazang||7:45 pm Sep 27, 2006|
|Do excluded pages still count as links-in
||3||johnlod||8:14 pm Sep 26, 2006|
|Order of lines in robots.txt file
Can you control a bot with the order of the lines?
|3||bouncybunny||3:37 am Sep 23, 2006|