Welcome to WebmasterWorld Guest from 54.211.136.250

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Do i need robot.txt on my site
robot.txt-is it needed & how to add it
5 Amitbud 1:46 pm Aug 25, 2006
Prevent spider from following external links
to keep Amazon items out of search engine database
2 Reno 7:11 am Aug 25, 2006
Robots txt to remove pages
Does google still look at urls blocked
3 cheaperholidays 7:56 am Aug 24, 2006
Where do I find the Robot Validation Tool?
robot validator validation robots
7 myebux 9:45 pm Aug 23, 2006
Need advice on my robots.txt file
new to robots.txt, can someone check this file
4 jwa55121 5:07 pm Aug 22, 2006
Are there any tricky uses of robot.txt file?
robot.xt file to be indexed by SEs?
4 yuvika 8:47 am Aug 22, 2006
New spider that is actually good.
found a "good" new spider
1 jcrimi 12:10 am Aug 22, 2006
robot.txt
problem disallowing pages in home directory
2 surfpro203 2:23 am Aug 16, 2006
Flash web site - need for robots.txt?
I have a web site in Flash - is it necessary to have a robots.txt file?
2 Cratima 1:05 pm Aug 15, 2006
Https? Where'd that come from?
Site dropped like a rock
2 Dragonlady26 1:56 am Aug 12, 2006
Please double check my robots.txt
1 lee_sufc 9:30 am Aug 7, 2006
Robots.txt necessary
Is the robots.txt file absolutely necessary
5 camchoice 6:57 pm Aug 6, 2006
'robot.txt' file to be indexed by SEs?!
are there any tricky uses of robot.txt file?
2 yuvika 6:56 pm Aug 6, 2006
Disallow index.html in Robots.txt
Is it safe to disallow an index.html file of a website
3 pixeladdict 8:48 pm Aug 4, 2006
disallow every 'filename' except.
disallow every 'filename' except...
2 decbrad 4:12 am Aug 4, 2006
Robots.txt Exclusion vs. Nofollow Tag
3 JasperSEO 1:39 am Aug 1, 2006
Keep robots away from site at least two weeks
1 zoth 2:43 pm Jul 29, 2006
Robots.txt exclusion help
Robots.txt exclusion help
3 webguy7 4:57 am Jul 22, 2006
using robots.txt
is it necessary to exclude .php pages
6 deborahbaker 11:53 am Jul 13, 2006
Robots exclusion revisted
How far have we come with controlling robots?
3 pixel_juice 11:12 pm Jul 7, 2006
Need to redirect bots from subdomain to main site
1 nemo1208 3:28 pm Jul 7, 2006
Help with Spiders
Cutting back on the bandwidth
2 devnet 4:56 am Jul 5, 2006
Is there any harm in not having a robots.txt file at all?
2 1Lit 10:55 am Jul 4, 2006
90% of my traffic is spiders and bots.
What to do -
4 webpro00801 6:41 am Jul 3, 2006
two duplicate sites .com .net
confused on how to handle mirror sites
4 jcnh74 9:33 pm Jul 1, 2006