Welcome to WebmasterWorld Guest from 54.211.17.91

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Sudden Increase in robots.txt Hits
3 Rob_Moore 10:04 pm Dec 26, 2004
Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites
4 BillingOrchard 9:41 pm Dec 23, 2004
About robots.txt file
Confusion in robots.txt file...i need hep..
7 venuc 5:32 am Dec 21, 2004
Googlebot Spoofing
How do you tell the real googlebot from spoofers?
3 teaperson 6:29 pm Dec 20, 2004
Tricky Robots Txt Help,
Two sites one robot
3 badboybart 2:50 pm Dec 20, 2004
redirect without confusing spiders
1 houseofsecrets 4:55 pm Dec 13, 2004
Robot.txt file deleted by accident what now?
2 Hoppo 11:15 pm Dec 12, 2004
robots.txt and meta tags
can I use both?
2 kdawg 11:23 pm Dec 11, 2004
Oddly long robots.txt file
Seeking explanation for weird robots.txt file
7 bendede 9:31 pm Dec 9, 2004
Hidden Text?
Assistance Please....
4 Kasey22 8:44 pm Dec 7, 2004
Google wildcard
Is this allowed?
3 maha 5:02 am Dec 6, 2004
How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder
3 walkman 9:01 pm Dec 2, 2004
Robots.txt Newbie Question
4 paulpsd 12:01 am Dec 2, 2004
redirecting domain
3 tomasz 9:31 pm Dec 1, 2004
subdomains and robots.txt
2 gettin_there 8:54 pm Dec 1, 2004
User-agent: ...
3 gettin_there 3:06 pm Dec 1, 2004
Crawling of non-linked pages
Over 18's area
3 theleveller 2:43 pm Dec 1, 2004
Ask Jeeves Question
Need to ban a bot
2 Marcus_Aurelius 10:13 am Dec 1, 2004
Webmaster World's Robots.txt
Why don't they allow msnbot?
6 Rollo 6:11 pm Nov 30, 2004
Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
9 AlucardSpyderWriter 8:17 pm Nov 28, 2004
How would I disallow Google...
...and allow all others?
3 Rollo 5:35 pm Nov 28, 2004
Robot.txt specialists: will Disallow: /? work?
need to block google from ...com/?tracking.
1 walkman 7:46 am Nov 28, 2004
robots.txt-newbie needs help
Is this correct?
4 cybervette 5:18 pm Nov 26, 2004
Changing your Mind with robots.txt
Will googlebot come back to the party?
2 Spine 4:41 pm Nov 26, 2004
dynamic folders
1 apan 12:24 pm Nov 25, 2004