Neil_McRae - 5:44 pm on Jun 28, 2011 (gmt 0)
Does anyone know any tricks for getting a more recent entry to appear for a site that hasn't been crawled since 2006? It seems there are a lot of websites out there that were crawled more frequently up until then. I contacted IA asking them if they take payments to update a website's entry. They wrote back saying that if the site is well linked it is likely to be crawled again and they do not charge for their services. I don't currently have a robots.txt file in the root directory on my homepage. Do I need to have one or could I possibly have something else enabled that might be preventing crawlers from retreiving copies of my website?
Thanks for any information you are able to give me about this.