Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Disallowing specific query strings
||3||duke33||3:05 pm Jun 8, 2004|
|Is it a spider?
||2||topher||5:38 am Jun 8, 2004|
|Preventing robot access to all subdirectories within a directory
||2||Leonardi||12:16 am Jun 7, 2004|
|Robots.txt triggers 404 error
Robots.txt triggers 404 error
|2||Nout||11:10 am May 29, 2004|
|Is this robots.txt ok?
||5||esllou||12:32 pm May 26, 2004|
|Removing "query-stringed" pages from Google
And other search engines using robotx.txt
|3||MatthewHSE||2:23 pm May 25, 2004|
|Stop Spiders Crawling Message Board Edit
Cgi Factory Message Board
|7||mn1dbp||12:36 pm May 25, 2004|
|Google still lists disallowed pages
How long do disallowed pages stay in the index?
|5||jam13||10:56 am May 23, 2004|
|large robots.txt file
What size will cause a crawler to choke in the file
|7||DoppyNL||9:43 pm May 21, 2004|
|Using robot.txt to keep unwanted backlinking
Is this an effective means
|2||Daily_Sparring||5:14 pm May 21, 2004|
|importance of a robots.txt file
||11||hughie||12:27 am May 21, 2004|
Wildcard userAgent last entry in robots.txt file?
|2||pageoneresults||2:34 pm May 20, 2004|
|Question about robots meta tag, as well as robots.txt....
||3||xsirrionx||11:48 am May 20, 2004|
|How to validate robots.txt on "https" site?
Please help me validate my "https" site
|2||CodeCrunch||12:29 am May 20, 2004|
|Wild chars *.* allowed in "Disallow" of Robots.txt?
or anything else like *.txt, etc.
|2||CodeCrunch||12:26 am May 20, 2004|
|robots.txt restrictions conflictual with site's Terms
Which of them should prevail?
|4||cristiscu||6:52 am May 19, 2004|
|Comments at the end of a line bad style, why?
||3||zachariah||6:52 am May 18, 2004|
||6||cabowabo||3:44 am May 13, 2004|
|Meta Tags & Robots.txt
||4||tcustom||1:57 am May 13, 2004|
|Why is Google still indexing my SWF files?
||2||ichthyous||6:10 am May 10, 2004|
|Deny or Not to Deny /images
Denying /images folder
|9||quebrada||12:19 am May 6, 2004|
|proper way of changing site
||2||jmdb71||5:11 pm May 1, 2004|
|How to get spider to visit parts of site more frequently
Is this possible?
|5||Pimpernel||10:35 am Apr 30, 2004|
||2||zyshen||5:19 pm Apr 27, 2004|
|Google ignoring robots.txt
I'm seeing lots of my 'Disallowed' pages in their index
|4||Amygdala||4:54 am Apr 24, 2004|