homepage Welcome to WebmasterWorld Guest from 54.204.127.191
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconBlocking Spiders
  How to block spiders from crawling your site
6Brian0275 9:15 pm on Jul 25, 2011 (gmt 0)
newpost thread iconwhich part of "disallow" did you not understand?
 
9lucy24 2:46 am on Jul 25, 2011 (gmt 0)
newpost thread iconwww, www21, wwww, etc. Displaying Same Content & Indexed By SEs
  What to do?
5getxb 6:48 am on Jul 22, 2011 (gmt 0)
newpost thread iconHow to manage this in robots.txt
  disallow a folder but include one page
10pankajj 1:36 pm on Jul 21, 2011 (gmt 0)
newpost thread iconDisallowed Googlebot-Image bot still spidering
 
7cyberdyne 8:47 am on Jul 9, 2011 (gmt 0)
newpost thread iconArchive.org
  for sites not crawled since 2006
8Neil_McRae 6:04 pm on Jul 1, 2011 (gmt 0)
newpost thread iconGooglebot not respecting custom crawl rate
 
13rowan194 3:38 am on Jun 22, 2011 (gmt 0)
newpost thread iconsitemap for images on subdomain
 
4AlexB77 6:50 am on Jun 11, 2011 (gmt 0)
newpost thread iconWhat's the difference between bing bot and Microsoft?
  Microsoft IP staying too long...banned for now, what's with that?
3erlandc 10:21 pm on Jun 4, 2011 (gmt 0)
newpost thread iconGoogle, Bing and Yahoo agree on Microformats?
  Was that discussed here: schema.org?
5pontifex 11:00 pm on Jun 3, 2011 (gmt 0)
newpost thread iconrobots.txt wildcard syntax?
  is this syntax ok?
5maha 9:39 pm on May 31, 2011 (gmt 0)
newpost thread iconIndex Page Removal, Serious Problem Please Help
  robots.txt or whats the issue? non-technical person here
6faust1 10:25 am on May 29, 2011 (gmt 0)
newpost thread iconGenerating a huge sitemap on shared host
  Creating a sitemap with thousands of URL'son a shared hosting plan.
1BrainDed 5:54 pm on May 23, 2011 (gmt 0)
newpost thread iconrobots.txt - disallow any url containing.
  using robots.txt and dynamic urls for phpbb3 forum
7BrainDed 10:28 pm on May 18, 2011 (gmt 0)
newpost thread iconWhy Indexing count Fluctuates?
 
2ihavegotyou 11:35 pm on May 17, 2011 (gmt 0)
newpost thread iconMy SITEMAPS are getting indexed--pls help implementing this tag
 
12donna130 1:16 pm on May 14, 2011 (gmt 0)
newpost thread iconthe good, the bad and the above the law?
 
1lucy24 10:03 pm on May 12, 2011 (gmt 0)
newpost thread iconblocking dynamic parts of forum
 
8viggen 6:42 pm on May 10, 2011 (gmt 0)
newpost thread iconSky IP requesting all files alphabetically
 
3cyberdyne 3:42 pm on May 7, 2011 (gmt 0)
newpost thread iconallowing & disallowing (robots.txt)
 
8mike2010 11:38 pm on May 4, 2011 (gmt 0)
newpost thread iconGoogle, robots, disallow
 
3cyberdyne 6:01 pm on May 3, 2011 (gmt 0)
newpost thread iconWebsite not showing in any search engine
 
21jesstp 3:20 pm on Apr 17, 2011 (gmt 0)
newpost thread iconupdating sitemaps - how often?
 
5iWebTrack 10:38 am on Apr 17, 2011 (gmt 0)
newpost thread iconhow many pages should be added in xml site map
 
2getxmlsitemap 10:25 am on Apr 17, 2011 (gmt 0)
newpost thread iconRedireted URls showing up in Robots.txt Errors
 
3raj1094 8:58 am on Apr 13, 2011 (gmt 0)
Displaying Page 7 of 71<< Back 1 2 3 4 5 6 [ 7 ] 8 9 10 11 12 13 14 15 16 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved