homepage Welcome to WebmasterWorld Guest from 54.211.47.170
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconwho can give me a website log file sample to see?
  website log file
3Tellist 5:19 pm on Sep 4, 2004 (gmt 0)
newpost thread iconGoogle CHTML Proxy disobeying robots.txt
  Why?
5yosmc 12:16 pm on Aug 31, 2004 (gmt 0)
newpost thread iconGoogle tells me to change my robots.txt
  Should I follow the advice from Google adSense?
2Bonusbana 2:44 pm on Aug 27, 2004 (gmt 0)
newpost thread iconMaster List of Malicious Robots
  Is there an active list of the bots we should block?
5Eric_Lander 4:09 pm on Aug 25, 2004 (gmt 0)
newpost thread iconrobots.txt security problems?
 
2helohelo 3:44 pm on Aug 25, 2004 (gmt 0)
newpost thread icon301 Redirect domain?
  How to handle robots.txt?
5Mr_Roberto 1:13 am on Aug 24, 2004 (gmt 0)
newpost thread iconCreating robots.txt file.
 
6kabanero 8:34 pm on Aug 18, 2004 (gmt 0)
newpost thread iconBlocking just one parameter via robots.txt
  Is this even possible?
2werty 7:31 pm on Aug 18, 2004 (gmt 0)
newpost thread iconProblem with spiderbots scrounging for robot.txt
 
3Mr_N 6:49 am on Aug 10, 2004 (gmt 0)
newpost thread iconi have a subdomain I dont want indexed
  is robots.txt the answer?
4chrisandsarah 1:21 pm on Aug 8, 2004 (gmt 0)
newpost thread iconrobots.txt appearing in my log files
  But I have n't got one there!
5uk_webber 2:59 pm on Aug 5, 2004 (gmt 0)
newpost thread iconTo Robots.txt or not
  Do we use both?
4obsos 11:20 pm on Aug 1, 2004 (gmt 0)
newpost thread iconmsn search violating robots.txt!
  `disallowed` images appear in preview on result page
26DoppyNL 4:15 pm on Aug 1, 2004 (gmt 0)
newpost thread iconRobots text file or noindex,nofollow
  Whats better?
4mthorpe 4:18 am on Jul 31, 2004 (gmt 0)
newpost thread icongoogle allow - disallow all others
 
6jimmy19 7:20 am on Jul 30, 2004 (gmt 0)
newpost thread iconMSN and Yahoo both missing robots.txt
 
4bppilot 3:26 pm on Jul 29, 2004 (gmt 0)
newpost thread iconYahoo robots.txt problem
  what redirect?
4yowza 9:53 pm on Jul 27, 2004 (gmt 0)
newpost thread iconGoogle listing disallowed pages
  this normal?
2sonik 6:44 pm on Jul 22, 2004 (gmt 0)
newpost thread iconrobots.txt for different language sites
  can use charset tag to allow/disallow?
4a2ztranslate 11:51 pm on Jul 20, 2004 (gmt 0)
newpost thread iconabout subdomain and robots
  should i copy robots to all subdomains?
2webon 3:33 am on Jul 18, 2004 (gmt 0)
newpost thread iconrobots.txt question
 
3motorcycle 5:30 pm on Jul 17, 2004 (gmt 0)
newpost thread iconDisallowing Robot's Effect on Sub-Domain
  Will it disallow the sub domains as well
2bppilot 5:53 am on Jul 14, 2004 (gmt 0)
newpost thread iconWhy is Google following my exclusions?
  Robots.txt file says dont go and Google does!
5anxvariety 5:49 am on Jul 14, 2004 (gmt 0)
newpost thread iconWhy are my .swf files still being indexed by Googlebot?
 
3ichthyous 6:26 pm on Jul 13, 2004 (gmt 0)
newpost thread iconexcluding dynamic pages in all search engines
  excluding dynamic pages in robots.txt
3toddmc04 2:49 pm on Jul 13, 2004 (gmt 0)
Displaying Page 55 of 71<< Back 1 ... 45 46 47 48 49 50 51 52 53 54 [ 55 ] 56 57 58 59 60 61 62 63 64 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved