Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Google Sitemaps Robots.txt Question
||2||Tonerman||5:31 pm Oct 9, 2006|
|Naming da' bots
Same place, different face
|1||Reno||11:58 am Oct 7, 2006|
Anyone seen this bot before?
|2||jexx||7:44 pm Oct 3, 2006|
|Disallow Coldfusion Pages Only
Stop cfm pages from being crawled
|2||stcrim||3:18 pm Oct 3, 2006|
|Removing specific file type
||8||MisterT||11:40 pm Sep 28, 2006|
|Robots - Allowing them
rather than disallow
|2||SanDiegoPaul||10:30 pm Sep 28, 2006|
|Robots.txt & meta
Robots meta tage or just a robots.txt
|4||pazang||7:45 pm Sep 27, 2006|
|Do excluded pages still count as links-in
||3||johnlod||8:14 pm Sep 26, 2006|
|Order of lines in robots.txt file
Can you control a bot with the order of the lines?
|3||bouncybunny||3:37 am Sep 23, 2006|
|robots & blog
||2||zuses||2:54 pm Sep 22, 2006|
|Keeping robots out of cgi bin
What's the best format?
|10||Reno||6:00 pm Sep 21, 2006|
|What user agent does Overture use?
when they ping your page for quality
|2||Bewenched||8:35 pm Sep 20, 2006|
||2||Northstar||7:50 pm Sep 20, 2006|
|Need Help With Robots.txt Question
||4||BlueLeaf||4:23 am Sep 19, 2006|
|Contrary advice over robots.txt validity
Who is right Google or the other?
|3||netchicken1||1:05 pm Sep 18, 2006|
|Disallow page disallows variables, too?
example.php and example.php?id=10
|3||4string||8:50 pm Sep 17, 2006|
||3||Boogalou||4:07 am Sep 17, 2006|
|Disallowing access to feed URLs
Should I disallow access to feed and trackback URLs
|1||kektex||4:45 am Sep 16, 2006|
|Exclude all robots on alias domains?
Good idea or not?
|1||jtara||5:52 pm Sep 15, 2006|
|Strange way of writing robots.txt
Has anybody use this?
|2||Cyclob||3:04 pm Sep 15, 2006|
Error issued by Sitemaps
|5||apprentice||7:43 pm Sep 9, 2006|
|IS there any tool to find out the particular page got spidered?
|3||geobals||8:50 pm Sep 7, 2006|
|Remove www. before sitename
||4||tigersoft||5:14 pm Sep 7, 2006|
|I think google can't access my pages.
||6||rj87uk||10:16 am Sep 4, 2006|
|Ban IP range
||3||hostlead||9:18 pm Sep 2, 2006|