Welcome to WebmasterWorld Guest from 54.146.41.3

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Please clarify this for the last time
Subfolder, particualr bots exclusion
1 damlag 10:12 pm Jan 11, 2005
exclude subdirectories
3 seoerer 11:08 pm Jan 7, 2005
Robot.txt Generator
should i put the results on my website?
3 rj87uk 10:54 am Jan 6, 2005
Not sure how to handle space character
5 Jon_King 1:48 am Jan 6, 2005
Problem of syntax and ordering
1 esllou 11:50 am Jan 4, 2005
Will robots.txt affect IE or other browsers?
A little confuse...
2 iProgram 4:34 am Jan 4, 2005
what are the names of the image bots?
I can't ban them if I don't know their names
8 stapel 5:17 pm Jan 2, 2005
Blocking images folder from spiders
Good or bad idea?
2 twist 10:17 pm Jan 1, 2005
Sudden Increase in robots.txt Hits
3 Rob_Moore 10:04 pm Dec 26, 2004
Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites
4 BillingOrchard 9:41 pm Dec 23, 2004
About robots.txt file
Confusion in robots.txt file...i need hep..
7 venuc 5:32 am Dec 21, 2004
Googlebot Spoofing
How do you tell the real googlebot from spoofers?
3 teaperson 6:29 pm Dec 20, 2004
Tricky Robots Txt Help,
Two sites one robot
3 badboybart 2:50 pm Dec 20, 2004
redirect without confusing spiders
1 houseofsecrets 4:55 pm Dec 13, 2004
Robot.txt file deleted by accident what now?
2 Hoppo 11:15 pm Dec 12, 2004
robots.txt and meta tags
can I use both?
2 kdawg 11:23 pm Dec 11, 2004
Oddly long robots.txt file
Seeking explanation for weird robots.txt file
7 bendede 9:31 pm Dec 9, 2004
Hidden Text?
Assistance Please....
4 Kasey22 8:44 pm Dec 7, 2004
Google wildcard
Is this allowed?
3 maha 5:02 am Dec 6, 2004
How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder
3 walkman 9:01 pm Dec 2, 2004
Robots.txt Newbie Question
4 paulpsd 12:01 am Dec 2, 2004
redirecting domain
3 tomasz 9:31 pm Dec 1, 2004
subdomains and robots.txt
2 gettin_there 8:54 pm Dec 1, 2004
User-agent: ...
3 gettin_there 3:06 pm Dec 1, 2004
Crawling of non-linked pages
Over 18's area
3 theleveller 2:43 pm Dec 1, 2004

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members