Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|How to manage this in robots.txt
disallow a folder but include one page
|10||pankajj||1:36 pm Jul 21, 2011|
|Disallowed Googlebot-Image bot still spidering
||7||cyberdyne||8:47 am Jul 9, 2011|
for sites not crawled since 2006
|8||Neil_McRae||6:04 pm Jul 1, 2011|
|Googlebot not respecting custom crawl rate
||13||rowan194||3:38 am Jun 22, 2011|
|sitemap for images on subdomain
||4||AlexB77||6:50 am Jun 11, 2011|
|What's the difference between bing bot and Microsoft?
Microsoft IP staying too long...banned for now, what's with that?
|3||erlandc||10:21 pm Jun 4, 2011|
|Google, Bing and Yahoo agree on Microformats?
Was that discussed here: schema.org?
|5||pontifex||11:00 pm Jun 3, 2011|
|robots.txt wildcard syntax?
is this syntax ok?
|5||maha||9:39 pm May 31, 2011|
|Index Page Removal, Serious Problem Please Help
robots.txt or whats the issue? non-technical person here
|6||faust1||10:25 am May 29, 2011|
|Generating a huge sitemap on shared host
Creating a sitemap with thousands of URL'son a shared hosting plan.
|1||BrainDed||5:54 pm May 23, 2011|
|robots.txt - disallow any url containing.
using robots.txt and dynamic urls for phpbb3 forum
|7||BrainDed||10:28 pm May 18, 2011|
|Why Indexing count Fluctuates?
||2||ihavegotyou||11:35 pm May 17, 2011|
|My SITEMAPS are getting indexed--pls help implementing this tag
||12||donna130||1:16 pm May 14, 2011|
|the good, the bad and the above the law?
||1||lucy24||10:03 pm May 12, 2011|
|blocking dynamic parts of forum
||8||viggen||6:42 pm May 10, 2011|
|Sky IP requesting all files alphabetically
||3||cyberdyne||3:42 pm May 7, 2011|
|allowing & disallowing (robots.txt)
||8||mike2010||11:38 pm May 4, 2011|
|Google, robots, disallow
||3||cyberdyne||6:01 pm May 3, 2011|
|Website not showing in any search engine
||21||jesstp||3:20 pm Apr 17, 2011|
|updating sitemaps - how often?
||5||iWebTrack||10:38 am Apr 17, 2011|
|how many pages should be added in xml site map
||2||getxmlsitemap||10:25 am Apr 17, 2011|
|Redireted URls showing up in Robots.txt Errors
||3||raj1094||8:58 am Apr 13, 2011|
|name of the website in title tag
||4||iWebTrack||7:36 am Apr 12, 2011|
|Meta.Txt is it real?
Is Meta.Txt an accepted protocol?
|2||iPullRank||10:47 am Apr 7, 2011|
|Help: htaccess rewrite rule
||14||bramley||7:27 am Apr 3, 2011|