Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Problems crawling my pages for XML Sitemap
Spider simulators (all of them) seem to struggle
|1||Munster||12:15 pm Nov 5, 2009|
|How to stop search engines caching my webpages?
||11||gcan||4:31 am Nov 2, 2009|
redirect in htaccess file
|3||anyone||3:53 pm Oct 29, 2009|
|Will URLs blocked in robots.txt remove currently indexed pages?
||6||limitup||12:20 am Oct 28, 2009|
|Noindex and also Changeing 2 million URL's
||2||azn_romeo_4u||10:53 am Oct 27, 2009|
|Blocked Shot - Robots.txt
blocking numerous duplicate pages using robots.txt
|6||underglass||10:50 am Oct 27, 2009|
|Need a Sitemap Generator
||2||eljuan||8:52 pm Oct 26, 2009|
|site map generator program
site map generator
|7||Roobik||12:27 pm Oct 26, 2009|
|Webmaster Tools Issues - Need Help
Webmaster tools showing data that does not reflect in Google
|3||ranadeep||7:16 am Oct 22, 2009|
|meta name="allow search" content="yes"
do i really need this tag now?
|4||supafresh||1:52 am Oct 20, 2009|
|Question concerning Disallow and *
||4||augustwind||1:44 am Oct 20, 2009|
|Which Is The Better Way To Handle Site Title
Which is the better way to handle the site title.
|4||Robert||11:40 pm Oct 18, 2009|
|Webmaster URL Removal Issues
Robots file/URL removal issues
|10||MarkWW||3:24 pm Oct 13, 2009|
|Different content on the same URL
Users language is stored in a cookie - website displays different content o
|3||gcan||9:23 pm Oct 12, 2009|
|Why search engines are showing the wrong meta title & description
Metadata was correct two months ago but now search engine showing wrong.
|4||Kash||2:36 pm Oct 12, 2009|
|Question about robots.txt file
It is blank
|2||iGLOBAL_NiCK||2:45 am Oct 12, 2009|
|allowing google bot only to index URL with parameters
||2||ulmudmezig||7:12 pm Oct 8, 2009|
|Google Webmaster Tools - Domain Not Found
||1||kflanagan||9:25 am Oct 6, 2009|
|Robots.txt and allow and $
|3||djingo||12:43 pm Oct 1, 2009|
|Index pages: site:www.abc.com vs Google Sitemap(Webmastertools)
Why there's a huge difference between the two?
|2||Singhvineet||1:28 pm Sep 29, 2009|
|Google not obeying robots.txt
Disallow: /*? not going through
|4||smallcompany||4:48 am Sep 26, 2009|
|Successful 301 Redirection?
Switching a website to an entirely new technology, URLS, etc
|5||supermanu||2:39 am Sep 23, 2009|
|Is Pinging Google a crime ?
||5||kinley||8:46 am Sep 20, 2009|
||7||seomonster||4:11 am Sep 15, 2009|
Include Robots.txt in my site
|2||ravikant||12:00 pm Sep 11, 2009|