jeffposaka
msg:3683637 | 2:10 am on Jun 26, 2008 (gmt 0) |
There are no files or folders that you need to block? It is hard to believe. I think meta tag is not needed across the site if you are on top of the robots.txt file.
|
jdMorgan
msg:3683643 | 2:29 am on Jun 26, 2008 (gmt 0) |
If you wish to allow robots to access everything, then you don't need the <meta name="robots" tag at all, and your robots.txt file can be blank. I recommend that even if it is blank, you should still have the file there, since this will prevent many 404 errors in your logs from robots trying to fetch it if it's missing. It may take several more days or even a couple of weeks for your site to recover mostly, and longer to recover fully. Always review your work carefully and test it. There are several files in which you cannot afford to make mistakes, and robots.txt is one of them... Jim
|
Senthil
msg:3684935 | 7:30 am on Jun 27, 2008 (gmt 0) |
I had the same experience but I got reindexed within few days... Better update/create your Google Sitemap/XML file and resubmit it to Google Webmasters Tools...
|
catman
msg:3709383 | 4:42 pm on Jul 28, 2008 (gmt 0) |
Google stopped indexing my site on July 11, and I found a robots.txt file that I did not place on the site that appeared to be blocking all search engines. We removed the file altogether. Is it possible that my host added this file, and will Google now begin indexing the site with no robots.txt in place? The site is: <snip> Thanks for your help. [edited by: goodroi at 3:32 am (utc) on July 29, 2008]
[edit reason] Please no specific URLs [/edit]
|
JamieBrown
msg:3709397 | 4:50 pm on Jul 28, 2008 (gmt 0) |
| I had the same experience but I got reindexed within few days... |
|
I've seen it take a few weeks. I think it depends on how popular your site was to start with. I would recommend:
1) Resubmit your sitemap
2) Do a bit of good old fashioned link building What is your webmaster tools analysis looking like right now? Does it have the latest robots.txt or is it showing the old version?
|
catman
msg:3709417 | 5:06 pm on Jul 28, 2008 (gmt 0) |
Thanks for your help. The analysis reads 404 not found. Does this mean I need to create a robots.txt page and leave it blank, or will the search engines index the page anyway? I've never submitted a sitemap and have always pulled up very highly on search engines, I think largely because I have an extremely long list of quality links. Thanks again, your help is most appreciated
|
JamieBrown
msg:3709457 | 5:33 pm on Jul 28, 2008 (gmt 0) |
If the robots.txt page in Webmaster Tools says 404 not found then it can't find the robots.txt file on your site. Have you deleted it? If so, that's a good thing (for the moment) - Google should start indexing all of the URLs on your domain. No robots.txt generally means "index everything" (as long as you don't have <meta name="robots" content="noindex,nofollow"> on your pages).
|
catman
msg:3709871 | 1:07 am on Jul 29, 2008 (gmt 0) |
Thanks again for your help. Yes, we deleted the robots.txt file. I'll most likely get some help in creating a new one once Google starts indexing my pages.
|
|
