Welcome to WebmasterWorld Guest from 54.147.63.124

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
no robots.txt
6 textex 5:50 pm Feb 8, 2005
Please help me with robots.txt
8 karantiq 1:18 pm Feb 8, 2005
e-SocietyRobot
e-SocietyRobot new IP...
2 beauzero 5:47 pm Feb 4, 2005
Robot.txt
<meta name="Robots" content="index,follow">
4 Trojanfx 11:37 pm Feb 3, 2005
Can I Disallow Domain Hosts?
I want to disallow spiders from hosts, like foo.mydomain.com
1 SoCalSam 7:24 pm Jan 30, 2005
Robots.txt conundrum!
4 energylevel 6:36 pm Jan 30, 2005
Help I'm trying to get my site on search engines such as google
Help on how to get your site on search engines such as google
9 Mako3 8:32 am Jan 30, 2005
Correct way to exclude dynamic pages
question about differing methods of excluding dynamic pages
8 heuristick 6:50 pm Jan 28, 2005
are the robots.txt examples still valid from WebmasterWorld?
1 keywordguru 2:59 pm Jan 28, 2005
Suggestions On 'Low Value' Bots
Which bots provide little to no value to the web site.
2 netcent 6:24 pm Jan 24, 2005
Blocking a file in the main folder?
4 Ept103 3:33 am Jan 23, 2005
robot.txt needed to be crawled by google?
bot leaves after not finding one
8 thegreatpretender 4:25 pm Jan 22, 2005
Blocking sherlock
quick question to confirm format
11 Reno 2:03 am Jan 20, 2005
Search Engines doesn't work with my robots.txt file
2 nirpet 10:38 pm Jan 18, 2005
google follows links disallowed
Have I done something wrong?
5 shady 4:54 pm Jan 18, 2005
Please clarify this for the last time
Subfolder, particualr bots exclusion
1 damlag 10:12 pm Jan 11, 2005
exclude subdirectories
3 seoerer 11:08 pm Jan 7, 2005
Robot.txt Generator
should i put the results on my website?
3 rj87uk 10:54 am Jan 6, 2005
Not sure how to handle space character
5 Jon_King 1:48 am Jan 6, 2005
Problem of syntax and ordering
1 esllou 11:50 am Jan 4, 2005
Will robots.txt affect IE or other browsers?
A little confuse...
2 iProgram 4:34 am Jan 4, 2005
what are the names of the image bots?
I can't ban them if I don't know their names
8 stapel 5:17 pm Jan 2, 2005
Blocking images folder from spiders
Good or bad idea?
2 twist 10:17 pm Jan 1, 2005
Sudden Increase in robots.txt Hits
3 Rob_Moore 10:04 pm Dec 26, 2004
Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites
4 BillingOrchard 9:41 pm Dec 23, 2004

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members