Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Are there any tricky uses of robot.txt file?
robot.xt file to be indexed by SEs?
|4||yuvika||8:47 am Aug 22, 2006|
|New spider that is actually good.
found a "good" new spider
|1||jcrimi||12:10 am Aug 22, 2006|
problem disallowing pages in home directory
|2||surfpro203||2:23 am Aug 16, 2006|
|Flash web site - need for robots.txt?
I have a web site in Flash - is it necessary to have a robots.txt file?
|2||Cratima||1:05 pm Aug 15, 2006|
|Https? Where'd that come from?
Site dropped like a rock
|2||Dragonlady26||1:56 am Aug 12, 2006|
|Please double check my robots.txt
||1||lee_sufc||9:30 am Aug 7, 2006|
Is the robots.txt file absolutely necessary
|5||camchoice||6:57 pm Aug 6, 2006|
|'robot.txt' file to be indexed by SEs?!
are there any tricky uses of robot.txt file?
|2||yuvika||6:56 pm Aug 6, 2006|
|Disallow index.html in Robots.txt
Is it safe to disallow an index.html file of a website
|3||pixeladdict||8:48 pm Aug 4, 2006|
|disallow every 'filename' except.
disallow every 'filename' except...
|2||decbrad||4:12 am Aug 4, 2006|
|Robots.txt Exclusion vs. Nofollow Tag
||3||JasperSEO||1:39 am Aug 1, 2006|
|Keep robots away from site at least two weeks
||1||zoth||2:43 pm Jul 29, 2006|
|Robots.txt exclusion help
Robots.txt exclusion help
|3||webguy7||4:57 am Jul 22, 2006|
is it necessary to exclude .php pages
|6||deborahbaker||11:53 am Jul 13, 2006|
|Robots exclusion revisted
How far have we come with controlling robots?
|3||pixel_juice||11:12 pm Jul 7, 2006|
|Need to redirect bots from subdomain to main site
||1||nemo1208||3:28 pm Jul 7, 2006|
|Help with Spiders
Cutting back on the bandwidth
|2||devnet||4:56 am Jul 5, 2006|
|Is there any harm in not having a robots.txt file at all?
||2||1Lit||10:55 am Jul 4, 2006|
|90% of my traffic is spiders and bots.
What to do -
|4||webpro00801||6:41 am Jul 3, 2006|
|two duplicate sites .com .net
confused on how to handle mirror sites
|4||jcnh74||9:33 pm Jul 1, 2006|
|Google 5000 character limit for robots.txt?
||6||physics||8:53 pm Jul 1, 2006|
|how to block all robots except for 2?
|7||ionchannels||12:30 pm Jun 29, 2006|
Any guesses as to when the SE's will display our listings again.
|4||kgormat||7:42 pm Jun 24, 2006|
How would the following be viewed by a spider?
|4||kgormat||3:29 pm Jun 24, 2006|
using the robots meta correctly
|1||Skylo||12:47 pm Jun 20, 2006|