Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Do i need robot.txt on my site
robot.txt-is it needed & how to add it
|5||Amitbud||1:46 pm Aug 25, 2006|
|Prevent spider from following external links
to keep Amazon items out of search engine database
|2||Reno||7:11 am Aug 25, 2006|
|Robots txt to remove pages
Does google still look at urls blocked
|3||cheaperholidays||7:56 am Aug 24, 2006|
|Where do I find the Robot Validation Tool?
robot validator validation robots
|7||myebux||9:45 pm Aug 23, 2006|
|Need advice on my robots.txt file
new to robots.txt, can someone check this file
|4||jwa55121||5:07 pm Aug 22, 2006|
|Are there any tricky uses of robot.txt file?
robot.xt file to be indexed by SEs?
|4||yuvika||8:47 am Aug 22, 2006|
|New spider that is actually good.
found a "good" new spider
|1||jcrimi||12:10 am Aug 22, 2006|
problem disallowing pages in home directory
|2||surfpro203||2:23 am Aug 16, 2006|
|Flash web site - need for robots.txt?
I have a web site in Flash - is it necessary to have a robots.txt file?
|2||Cratima||1:05 pm Aug 15, 2006|
|Https? Where'd that come from?
Site dropped like a rock
|2||Dragonlady26||1:56 am Aug 12, 2006|
|Please double check my robots.txt
||1||lee_sufc||9:30 am Aug 7, 2006|
Is the robots.txt file absolutely necessary
|5||camchoice||6:57 pm Aug 6, 2006|
|'robot.txt' file to be indexed by SEs?!
are there any tricky uses of robot.txt file?
|2||yuvika||6:56 pm Aug 6, 2006|
|Disallow index.html in Robots.txt
Is it safe to disallow an index.html file of a website
|3||pixeladdict||8:48 pm Aug 4, 2006|
|disallow every 'filename' except.
disallow every 'filename' except...
|2||decbrad||4:12 am Aug 4, 2006|
|Robots.txt Exclusion vs. Nofollow Tag
||3||JasperSEO||1:39 am Aug 1, 2006|
|Keep robots away from site at least two weeks
||1||zoth||2:43 pm Jul 29, 2006|
|Robots.txt exclusion help
Robots.txt exclusion help
|3||webguy7||4:57 am Jul 22, 2006|
is it necessary to exclude .php pages
|6||deborahbaker||11:53 am Jul 13, 2006|
|Robots exclusion revisted
How far have we come with controlling robots?
|3||pixel_juice||11:12 pm Jul 7, 2006|
|Need to redirect bots from subdomain to main site
||1||nemo1208||3:28 pm Jul 7, 2006|
|Help with Spiders
Cutting back on the bandwidth
|2||devnet||4:56 am Jul 5, 2006|
|Is there any harm in not having a robots.txt file at all?
||2||1Lit||10:55 am Jul 4, 2006|
|90% of my traffic is spiders and bots.
What to do -
|4||webpro00801||6:41 am Jul 3, 2006|
|two duplicate sites .com .net
confused on how to handle mirror sites
|4||jcnh74||9:33 pm Jul 1, 2006|