Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Frequencey of GET /robots.txt
How often should a crawler/spider/robot ask for robots.txt?
|9||Dijkgraaf||5:08 am Jun 18, 2005|
|Need Clarification On Directory Exclusions
||10||jonrichd||5:00 am Jun 18, 2005|
|Robots.txt and Mod Rewrite
Which url does the robot read?
|2||eflouret||4:52 am Jun 18, 2005|
|how to ban Google from indexing ALL .php files
I'm using rewrite and don't want a dupe penalty
|16||walkman||5:12 pm Jun 15, 2005|
|boitho.com bot violating robots.txt
Specifically requested only forbidden files ( 1 2 3 4 )
|111||jazzguy||8:30 pm Jun 14, 2005|
|robots.txt and Google on intranet
robots.txt not working
|4||John488||8:11 pm Jun 13, 2005|
|Specific Spiders do need more bandwidth?
Leaving Gbot or Msn does other robots take more bandwidth when crawling?
|2||bnvrc||12:39 pm Jun 13, 2005|
|robots.text ..please help
help with robots.text file
|2||dmiller63||12:35 pm Jun 13, 2005|
||8||branmh||6:34 pm Jun 12, 2005|
|Google Robots.txt Wildcard Feature
Please take a look at my robot.txt
|4||seunosewa||7:02 am Jun 11, 2005|
|Separate robots.txt file for secure site
How do i block one and not the other?
|3||Vimes||3:34 am Jun 10, 2005|
||15||ownerrim||4:25 am Jun 8, 2005|
|If I disallow /filename.php ...
will google still index /filename.php?whatever
|7||Trisha||7:05 am Jun 7, 2005|
|Agent Name for MSN
Where can I find this information.
|8||Pat1975||5:55 am Jun 3, 2005|
General robots.txt queries
|7||makzan||11:49 pm Jun 2, 2005|
|Disallow /file.php same as Diallow file.php?
||6||One_on_One||3:43 am Jun 2, 2005|
from Competitors Analysis Tools...
|4||ambiorix_rb||1:21 pm Jun 1, 2005|
|SEO / robots.txt question
unintentional cross-linking penalty
|1||wolfgang||6:27 pm May 31, 2005|
|Blocking a Sub Domain Directory
||3||is300||8:25 am May 30, 2005|
|Order allow,deny or deny,allow
Not sure which order is best...
|3||whoisgregg||9:16 pm May 27, 2005|
|Search Engine Crawlers
||2||dayo||8:39 pm May 26, 2005|
|Spidering of my code pages
Wrong pages being spidered and losing referals
|5||djfelip||11:18 pm May 25, 2005|
|robots4.txt -- I'm confused
Confused about what is allowed in this file
|3||Goose_68||8:46 pm May 24, 2005|
|2||webmz||3:10 am May 22, 2005|
||18||Dexie||4:14 pm May 21, 2005|