Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Google indexing blocked content
||6||fatpeter||5:52 pm Sep 10, 2011|
|Category and Tag pages in Sitemap
listing noindex pages in sitemap
|2||cloud36||6:54 am Sep 9, 2011|
|Robots.txt for Wordpress and Vbulletin together - Help
Robots.txt for Wordpress and Vbulletin - Help
|7||anticipation||11:17 pm Sep 3, 2011|
|Google's Search Result Display Pattern
||5||webexpert2u||3:42 am Aug 31, 2011|
|robots.txt disallow searching reoccuring folders
||6||jchristi||2:14 am Aug 30, 2011|
|how to disallow feed in robot.txt readable by all bots
||5||bhavana||4:42 am Aug 17, 2011|
|Sitemap for out of stock products
||2||sam222||6:43 am Aug 9, 2011|
|what we do for our robots
||6||lucy24||2:22 am Aug 7, 2011|
|Question on Sitelinks
|3||sledge81||10:26 pm Aug 2, 2011|
|How can I block these kinds of urls?
||3||starchild||1:45 am Aug 2, 2011|
|sitemap for mobile versions
||2||sam222||2:11 am Jul 30, 2011|
|Free tools for creating a sitemap
||11||Fotiman||12:13 pm Jul 27, 2011|
How to block spiders from crawling your site
|6||Brian0275||9:15 pm Jul 25, 2011|
|which part of "disallow" did you not understand?
||9||lucy24||2:46 am Jul 25, 2011|
|www, www21, wwww, etc. Displaying Same Content & Indexed By SEs
What to do?
|5||getxb||6:48 am Jul 22, 2011|
|How to manage this in robots.txt
disallow a folder but include one page
|10||pankajj||1:36 pm Jul 21, 2011|
|Disallowed Googlebot-Image bot still spidering
||7||cyberdyne||8:47 am Jul 9, 2011|
for sites not crawled since 2006
|8||Neil_McRae||6:04 pm Jul 1, 2011|
|Googlebot not respecting custom crawl rate
||13||rowan194||3:38 am Jun 22, 2011|
|sitemap for images on subdomain
||4||AlexB77||6:50 am Jun 11, 2011|
|What's the difference between bing bot and Microsoft?
Microsoft IP staying too long...banned for now, what's with that?
|3||erlandc||10:21 pm Jun 4, 2011|
|Google, Bing and Yahoo agree on Microformats?
Was that discussed here: schema.org?
|5||pontifex||11:00 pm Jun 3, 2011|
|robots.txt wildcard syntax?
is this syntax ok?
|5||maha||9:39 pm May 31, 2011|
|Index Page Removal, Serious Problem Please Help
robots.txt or whats the issue? non-technical person here
|6||faust1||10:25 am May 29, 2011|
|Generating a huge sitemap on shared host
Creating a sitemap with thousands of URL'son a shared hosting plan.
|1||BrainDed||5:54 pm May 23, 2011|