homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconPreviously Indexed Files-Want To Exclude Now
  RobotsTxt Change
3buview1 12:23 pm on Jul 16, 2005 (gmt 0)
newpost thread iconGoogle only hits robots and index
  Google only hits robots and index
10rostom 3:51 pm on Jul 15, 2005 (gmt 0)
newpost thread iconDisallowing & in url
 
2Wolvereness 2:54 pm on Jul 12, 2005 (gmt 0)
newpost thread iconrobots.txt verses sitemap.html verses OPML file
 
2Neal 8:39 pm on Jul 11, 2005 (gmt 0)
newpost thread iconJust added robots.txt file
 
4soccrt 4:41 am on Jul 11, 2005 (gmt 0)
newpost thread iconblock all except
 
2Radial 9:31 pm on Jul 7, 2005 (gmt 0)
newpost thread iconrobots.txt with? in URL
 
16jozomannen 11:17 am on Jul 6, 2005 (gmt 0)
newpost thread iconHow do i add robot.txt?
 
10genny 4:38 am on Jul 6, 2005 (gmt 0)
newpost thread iconInclude files, spiders, and robots.txt
 
4rottsnroses 11:24 pm on Jul 3, 2005 (gmt 0)
newpost thread iconHow to Stop BecomeBot?
 
12shilmy 3:09 pm on Jun 29, 2005 (gmt 0)
newpost thread iconGoogle ignoring my robots.txt?
 
9homesby 12:17 am on Jun 29, 2005 (gmt 0)
newpost thread iconRobots meta tag *CORRECT* syntax
  Space* or no space for "index,*follow", and case sensitive?
6Clint 12:56 pm on Jun 28, 2005 (gmt 0)
newpost thread iconQuestion about Robots with multiple similar sites
  Question about Robots with multiple similar sites
2ccash 9:54 pm on Jun 27, 2005 (gmt 0)
newpost thread iconRobots/Slurp, etc - what does this mean
  I am a newbie - not sure what this means
4ramnyc 3:01 pm on Jun 22, 2005 (gmt 0)
newpost thread iconDoes adding robot text help your site get crawled more often?
  Advantages of robot text?
6moneyangel13 8:04 am on Jun 22, 2005 (gmt 0)
newpost thread iconwildcards in page names
  Disallow: /store/scripts/emailFriend.asp*
5jimsthoughts 3:54 pm on Jun 21, 2005 (gmt 0)
newpost thread iconHow can I allow robots to access my pages
  inktomi is following links it should not
4expat123 10:08 am on Jun 21, 2005 (gmt 0)
newpost thread iconUpdating robots.txt and Crawlers behaviour
  How to tell bots to come back when you changed robots.txt banning them
6silverbytes 6:57 pm on Jun 20, 2005 (gmt 0)
newpost thread iconslurp ignoring disallow?
  -or have i got it wrong?
4tallis 12:09 pm on Jun 20, 2005 (gmt 0)
newpost thread iconQuestion about blocking 1 robot, but allow all others
 
8capsmaster 3:26 pm on Jun 18, 2005 (gmt 0)
newpost thread iconFrequencey of GET /robots.txt
  How often should a crawler/spider/robot ask for robots.txt?
9Dijkgraaf 5:08 am on Jun 18, 2005 (gmt 0)
newpost thread iconNeed Clarification On Directory Exclusions
 
10jonrichd 5:00 am on Jun 18, 2005 (gmt 0)
newpost thread iconRobots.txt and Mod Rewrite
  Which url does the robot read?
2eflouret 4:52 am on Jun 18, 2005 (gmt 0)
newpost thread iconhow to ban Google from indexing ALL .php files
  I'm using rewrite and don't want a dupe penalty
16walkman 5:12 pm on Jun 15, 2005 (gmt 0)
newpost thread iconboitho.com bot violating robots.txt[4] ( 1 2 3 4 )
  Specifically requested only forbidden files
111jazzguy 8:30 pm on Jun 14, 2005 (gmt 0)
Displaying Page 46 of 72<< Back 1 ... 36 37 38 39 40 41 42 43 44 45 [ 46 ] 47 48 49 50 51 52 53 54 55 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved