Welcome to WebmasterWorld Guest from 23.22.46.195

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
domain.com vs www.domain.com
different sites
9 HitProf 4:01 pm Jul 7, 2004
Do i need the txt file?
if i want to allow everyone
4 markbiz 12:59 am Jul 5, 2004
I'm not understanding the different makes of bots
Can I stop some, while allowing others?
2 tomequin 10:11 pm Jul 1, 2004
Yahoo! Slurp - robots.txt
Not obeyng robots.txt
5 EarWig 1:59 pm Jun 29, 2004
Making robots?
4 lifedeuce 1:22 pm Jun 29, 2004
Disallow pages in a subdirectory
Basic help please
3 pjuk 8:40 pm Jun 24, 2004
site setup for spider?
5 Gemelli_no1 12:51 pm Jun 23, 2004
needed?
is it completely neccisary
3 natty 1:34 pm Jun 15, 2004
Robots dot text for all user files?
robots.txt all public_html
4 kj6loh 7:21 pm Jun 11, 2004
wildcards in the Disallow field,
non standard?
3 pmac 4:19 pm Jun 11, 2004
where to place robots.txt?
robots newbie alert
6 Duke_of_Url 10:17 pm Jun 10, 2004
How should I use a robots.txt file on a virtual account?
I want to ban spiders from one site only
3 soccer_star 2:01 pm Jun 10, 2004
Robots.txt for yahoo and google.com
google and yahoo robots.txt
3 adisakmai 4:47 am Jun 10, 2004
Prevent Google's "cached" link from appearing
but still stay in the search results
4 digitalv 4:51 pm Jun 8, 2004
Disallowing specific query strings
3 duke33 3:05 pm Jun 8, 2004
Is it a spider?
2 topher 5:38 am Jun 8, 2004
Preventing robot access to all subdirectories within a directory
2 Leonardi 12:16 am Jun 7, 2004
Robots.txt triggers 404 error
Robots.txt triggers 404 error
2 Nout 11:10 am May 29, 2004
Is this robots.txt ok?
5 esllou 12:32 pm May 26, 2004
Removing "query-stringed" pages from Google
And other search engines using robotx.txt
3 MatthewHSE 2:23 pm May 25, 2004
Stop Spiders Crawling Message Board Edit
Cgi Factory Message Board
7 mn1dbp 12:36 pm May 25, 2004
Google still lists disallowed pages
How long do disallowed pages stay in the index?
5 jam13 10:56 am May 23, 2004
large robots.txt file
What size will cause a crawler to choke in the file
7 DoppyNL 9:43 pm May 21, 2004
Using robot.txt to keep unwanted backlinking
Is this an effective means
2 Daily_Sparring 5:14 pm May 21, 2004
importance of a robots.txt file
11 hughie 12:27 am May 21, 2004

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month