Welcome to WebmasterWorld Guest from 54.166.117.130

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Are there any tricky uses of robot.txt file?
robot.xt file to be indexed by SEs?
4 yuvika 8:47 am Aug 22, 2006
New spider that is actually good.
found a "good" new spider
1 jcrimi 12:10 am Aug 22, 2006
robot.txt
problem disallowing pages in home directory
2 surfpro203 2:23 am Aug 16, 2006
Flash web site - need for robots.txt?
I have a web site in Flash - is it necessary to have a robots.txt file?
2 Cratima 1:05 pm Aug 15, 2006
Https? Where'd that come from?
Site dropped like a rock
2 Dragonlady26 1:56 am Aug 12, 2006
Please double check my robots.txt
1 lee_sufc 9:30 am Aug 7, 2006
Robots.txt necessary
Is the robots.txt file absolutely necessary
5 camchoice 6:57 pm Aug 6, 2006
'robot.txt' file to be indexed by SEs?!
are there any tricky uses of robot.txt file?
2 yuvika 6:56 pm Aug 6, 2006
Disallow index.html in Robots.txt
Is it safe to disallow an index.html file of a website
3 pixeladdict 8:48 pm Aug 4, 2006
disallow every 'filename' except.
disallow every 'filename' except...
2 decbrad 4:12 am Aug 4, 2006
Robots.txt Exclusion vs. Nofollow Tag
3 JasperSEO 1:39 am Aug 1, 2006
Keep robots away from site at least two weeks
1 zoth 2:43 pm Jul 29, 2006
Robots.txt exclusion help
Robots.txt exclusion help
3 webguy7 4:57 am Jul 22, 2006
using robots.txt
is it necessary to exclude .php pages
6 deborahbaker 11:53 am Jul 13, 2006
Robots exclusion revisted
How far have we come with controlling robots?
3 pixel_juice 11:12 pm Jul 7, 2006
Need to redirect bots from subdomain to main site
1 nemo1208 3:28 pm Jul 7, 2006
Help with Spiders
Cutting back on the bandwidth
2 devnet 4:56 am Jul 5, 2006
Is there any harm in not having a robots.txt file at all?
2 1Lit 10:55 am Jul 4, 2006
90% of my traffic is spiders and bots.
What to do -
4 webpro00801 6:41 am Jul 3, 2006
two duplicate sites .com .net
confused on how to handle mirror sites
4 jcnh74 9:33 pm Jul 1, 2006
Google 5000 character limit for robots.txt?
6 physics 8:53 pm Jul 1, 2006
how to block all robots except for 2?
possible?
7 ionchannels 12:30 pm Jun 29, 2006
Re-Indexing Timetable
Any guesses as to when the SE's will display our listings again.
4 kgormat 7:42 pm Jun 24, 2006
Disallow: /#example.html#
How would the following be viewed by a spider?
4 kgormat 3:29 pm Jun 24, 2006
Robot behaviour
using the robots meta correctly
1 Skylo 12:47 pm Jun 20, 2006

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month