Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|How can I block these kinds of urls?
||3||starchild||1:45 am Aug 2, 2011|
|sitemap for mobile versions
||2||sam222||2:11 am Jul 30, 2011|
|Free tools for creating a sitemap
||11||Fotiman||12:13 pm Jul 27, 2011|
How to block spiders from crawling your site
|6||Brian0275||9:15 pm Jul 25, 2011|
|which part of "disallow" did you not understand?
||9||lucy24||2:46 am Jul 25, 2011|
|www, www21, wwww, etc. Displaying Same Content & Indexed By SEs
What to do?
|5||getxb||6:48 am Jul 22, 2011|
|How to manage this in robots.txt
disallow a folder but include one page
|10||pankajj||1:36 pm Jul 21, 2011|
|Disallowed Googlebot-Image bot still spidering
||7||cyberdyne||8:47 am Jul 9, 2011|
for sites not crawled since 2006
|8||Neil_McRae||6:04 pm Jul 1, 2011|
|Googlebot not respecting custom crawl rate
||13||rowan194||3:38 am Jun 22, 2011|
|sitemap for images on subdomain
||4||AlexB77||6:50 am Jun 11, 2011|
|What's the difference between bing bot and Microsoft?
Microsoft IP staying too long...banned for now, what's with that?
|3||erlandc||10:21 pm Jun 4, 2011|
|Google, Bing and Yahoo agree on Microformats?
Was that discussed here: schema.org?
|5||pontifex||11:00 pm Jun 3, 2011|
|robots.txt wildcard syntax?
is this syntax ok?
|5||maha||9:39 pm May 31, 2011|
|Index Page Removal, Serious Problem Please Help
robots.txt or whats the issue? non-technical person here
|6||faust1||10:25 am May 29, 2011|
|Generating a huge sitemap on shared host
Creating a sitemap with thousands of URL'son a shared hosting plan.
|1||BrainDed||5:54 pm May 23, 2011|
|robots.txt - disallow any url containing.
using robots.txt and dynamic urls for phpbb3 forum
|7||BrainDed||10:28 pm May 18, 2011|
|Why Indexing count Fluctuates?
||2||ihavegotyou||11:35 pm May 17, 2011|
|My SITEMAPS are getting indexed--pls help implementing this tag
||12||donna130||1:16 pm May 14, 2011|
|the good, the bad and the above the law?
||1||lucy24||10:03 pm May 12, 2011|
|blocking dynamic parts of forum
||8||viggen||6:42 pm May 10, 2011|
|Sky IP requesting all files alphabetically
||3||cyberdyne||3:42 pm May 7, 2011|
|allowing & disallowing (robots.txt)
||8||mike2010||11:38 pm May 4, 2011|
|Google, robots, disallow
||3||cyberdyne||6:01 pm May 3, 2011|
|Website not showing in any search engine
||21||jesstp||3:20 pm Apr 17, 2011|