Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Where to put robots in website?
I has doubt where to put robots.
|3||viva||10:00 am Sep 26, 2004|
|robots.txt and .htaccess
||1||HuhuFruFru||12:02 am Sep 26, 2004|
|Excluding Sub Folders
When you exclude a folder, are folders inside that excluded also?
|3||damlag||8:39 pm Sep 25, 2004|
|Can someone help me write a robots.txt?
||5||budbiss||11:50 pm Sep 19, 2004|
How to avoid spidering of anything with this prefix
|2||nuevojefe||10:26 pm Sep 19, 2004|
|5||wolfline||5:10 am Sep 17, 2004|
robot.txt working against me?
|5||seofan||9:36 pm Sep 8, 2004|
|who can give me a website log file sample to see?
website log file
|3||Tellist||5:19 pm Sep 4, 2004|
|Google CHTML Proxy disobeying robots.txt
|5||yosmc||12:16 pm Aug 31, 2004|
|Google tells me to change my robots.txt
Should I follow the advice from Google adSense?
|2||Bonusbana||2:44 pm Aug 27, 2004|
|Master List of Malicious Robots
Is there an active list of the bots we should block?
|5||Eric_Lander||4:09 pm Aug 25, 2004|
|robots.txt security problems?
||2||helohelo||3:44 pm Aug 25, 2004|
|301 Redirect domain?
How to handle robots.txt?
|5||Mr_Roberto||1:13 am Aug 24, 2004|
|Creating robots.txt file.
||6||kabanero||8:34 pm Aug 18, 2004|
|Blocking just one parameter via robots.txt
Is this even possible?
|2||werty||7:31 pm Aug 18, 2004|
|Problem with spiderbots scrounging for robot.txt
||3||Mr_N||6:49 am Aug 10, 2004|
|i have a subdomain I dont want indexed
is robots.txt the answer?
|4||chrisandsarah||1:21 pm Aug 8, 2004|
|robots.txt appearing in my log files
But I have n't got one there!
|5||uk_webber||2:59 pm Aug 5, 2004|
|To Robots.txt or not
Do we use both?
|4||obsos||11:20 pm Aug 1, 2004|
|msn search violating robots.txt!
`disallowed` images appear in preview on result page
|26||DoppyNL||4:15 pm Aug 1, 2004|
|Robots text file or noindex,nofollow
|4||mthorpe||4:18 am Jul 31, 2004|
|google allow - disallow all others
||6||jimmy19||7:20 am Jul 30, 2004|
|MSN and Yahoo both missing robots.txt
||4||bppilot||3:26 pm Jul 29, 2004|
|Yahoo robots.txt problem
|4||yowza||9:53 pm Jul 27, 2004|
|Google listing disallowed pages
|2||sonik||6:44 pm Jul 22, 2004|