Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|domain.com vs www.domain.com
|9||HitProf||4:01 pm Jul 7, 2004|
|Do i need the txt file?
if i want to allow everyone
|4||markbiz||12:59 am Jul 5, 2004|
|I'm not understanding the different makes of bots
Can I stop some, while allowing others?
|2||tomequin||10:11 pm Jul 1, 2004|
|Yahoo! Slurp - robots.txt
Not obeyng robots.txt
|5||EarWig||1:59 pm Jun 29, 2004|
||4||lifedeuce||1:22 pm Jun 29, 2004|
|Disallow pages in a subdirectory
Basic help please
|3||pjuk||8:40 pm Jun 24, 2004|
|site setup for spider?
||5||Gemelli_no1||12:51 pm Jun 23, 2004|
is it completely neccisary
|3||natty||1:34 pm Jun 15, 2004|
|Robots dot text for all user files?
robots.txt all public_html
|4||kj6loh||7:21 pm Jun 11, 2004|
|wildcards in the Disallow field,
|3||pmac||4:19 pm Jun 11, 2004|
|where to place robots.txt?
robots newbie alert
|6||Duke_of_Url||10:17 pm Jun 10, 2004|
|How should I use a robots.txt file on a virtual account?
I want to ban spiders from one site only
|3||soccer_star||2:01 pm Jun 10, 2004|
|Robots.txt for yahoo and google.com
google and yahoo robots.txt
|3||adisakmai||4:47 am Jun 10, 2004|
|Prevent Google's "cached" link from appearing
but still stay in the search results
|4||digitalv||4:51 pm Jun 8, 2004|
|Disallowing specific query strings
||3||duke33||3:05 pm Jun 8, 2004|
|Is it a spider?
||2||topher||5:38 am Jun 8, 2004|
|Preventing robot access to all subdirectories within a directory
||2||Leonardi||12:16 am Jun 7, 2004|
|Robots.txt triggers 404 error
Robots.txt triggers 404 error
|2||Nout||11:10 am May 29, 2004|
|Is this robots.txt ok?
||5||esllou||12:32 pm May 26, 2004|
|Removing "query-stringed" pages from Google
And other search engines using robotx.txt
|3||MatthewHSE||2:23 pm May 25, 2004|
|Stop Spiders Crawling Message Board Edit
Cgi Factory Message Board
|7||mn1dbp||12:36 pm May 25, 2004|
|Google still lists disallowed pages
How long do disallowed pages stay in the index?
|5||jam13||10:56 am May 23, 2004|
|large robots.txt file
What size will cause a crawler to choke in the file
|7||DoppyNL||9:43 pm May 21, 2004|
|Using robot.txt to keep unwanted backlinking
Is this an effective means
|2||Daily_Sparring||5:14 pm May 21, 2004|
|importance of a robots.txt file
||11||hughie||12:27 am May 21, 2004|