Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|large robots.txt file
What size will cause a crawler to choke in the file
|7||DoppyNL||9:43 pm May 21, 2004|
|Using robot.txt to keep unwanted backlinking
Is this an effective means
|2||Daily_Sparring||5:14 pm May 21, 2004|
|importance of a robots.txt file
||11||hughie||12:27 am May 21, 2004|
Wildcard userAgent last entry in robots.txt file?
|2||pageoneresults||2:34 pm May 20, 2004|
|Question about robots meta tag, as well as robots.txt....
||3||xsirrionx||11:48 am May 20, 2004|
|How to validate robots.txt on "https" site?
Please help me validate my "https" site
|2||CodeCrunch||12:29 am May 20, 2004|
|Wild chars *.* allowed in "Disallow" of Robots.txt?
or anything else like *.txt, etc.
|2||CodeCrunch||12:26 am May 20, 2004|
|robots.txt restrictions conflictual with site's Terms
Which of them should prevail?
|4||cristiscu||6:52 am May 19, 2004|
|Comments at the end of a line bad style, why?
||3||zachariah||6:52 am May 18, 2004|
||6||cabowabo||3:44 am May 13, 2004|
|Meta Tags & Robots.txt
||4||tcustom||1:57 am May 13, 2004|
|Why is Google still indexing my SWF files?
||2||ichthyous||6:10 am May 10, 2004|
|Deny or Not to Deny /images
Denying /images folder
|9||quebrada||12:19 am May 6, 2004|
|proper way of changing site
||2||jmdb71||5:11 pm May 1, 2004|
|How to get spider to visit parts of site more frequently
Is this possible?
|5||Pimpernel||10:35 am Apr 30, 2004|
||2||zyshen||5:19 pm Apr 27, 2004|
|Google ignoring robots.txt
I'm seeing lots of my 'Disallowed' pages in their index
|4||Amygdala||4:54 am Apr 24, 2004|
|Strange bot keep on visiting my site and don't obey my robots.txt file
What to do?
|2||Compworld||8:01 pm Apr 22, 2004|
|revisit after so many days
||2||tommygunner||3:21 am Apr 21, 2004|
|index.php vs index.html
Will This Work i Wonder
|2||yummycarp||11:28 pm Apr 20, 2004|
|Banning a script using robots.txt
how do do this
|4||Imaster||9:51 pm Apr 20, 2004|
|Robots.txt - Why?
||3||warlordbb||2:08 pm Apr 20, 2004|
|Can IMDB do this?
charge per unwanted request
|3||jrap||10:19 pm Apr 19, 2004|
Google keeps crawling page
|2||anxvariety||2:27 pm Apr 19, 2004|
|Froogle wants my images!
||3||seadog||5:59 am Apr 14, 2004|