Forum Moderators: open
The answer may be as simple as cost. Now that Yahoo is doing their own thing, Google may have to conserve resources at bit.
-s-
Get inbound links, have an easy to read site and build traffic.
"If you build it, google will come"
My cache is rarely over 2 days old. I find it amazing how than can index so quickly!
[webmasterworld.com...]
And don't kid yourself about the cost of crawling and making use of the crawled pages...
-s-
BryonM said,
"Where are you seeing this? I'm crawled every day and i only have a few pr4 sites.
Get inbound links, have an easy to read site and build traffic.
"If you build it, google will come"
My cache is rarely over 2 days old. I find it amazing how than can index so quickly! ""
Very nice to hear that googlebot is still visiting your site regularly...However, it doesn't always seem to be the case...I've got a site with about 20,000 unique links with a PR6-7 with lots of unique content and all pages are google friendly...Googlebot is not crawling the site anymore and it now shows a 1 month old cache...
Our latest Press Release (yesterday) was found by googlebot within one hour and included in their news feed immediately.
Most often it is the Mediapartners-Google/2.1 which is visiting but there was a major visitation last night and this morning from the 64.68 guys.
As for being crawled and indexed - I have seen what seem to be changes with my sites. I think it was around the time of Florida when pages that were picked up by Freshbot began to stick in the index. Over the past month, I've had a few pages picked up by freshbot, indexed for a few days, then dropped. Pre-Florida, this was the typical pattern, and back then, it wasn't until the big monthly update occurred that all of the crawled were included.
Another thing I've noticed is that I've actually had many new pages spidered over the past several days (not mediabot, which I thought is just for adsense?), but they are not showing up in the index, with or without fresh tags. Over the past few months, new pages would typically show up around 24 hours after being crawled - not so this last week.
I don't know what to make of this, or if there is any reason to make anything of it, but I figured I'd just share my experience.
FYI, the sites that are getting crawled have links from PR 6 sites.
If it was a money issue, it would not have such an immediate effect.
All those servers of Google's that do the crawl are also general purpose computers. They are able to use them for other things when they feel the need. It just might be that they pulled a bunch of them for use fighting some evil or another that people complain about here from time to time.
Googlebot has definitely slowed down on my site, but it is still there. I'm not worried.
As for new sites, google still claims that you should expect them to be indexed in between one and two months. Y'all are just getting spoiled with freshbot lately.
So far that's the only page that's fresh and many of the others show in the top ten results for their keywords. I keep thinking that every day some will start showing up, but nothing.
Maybe the cost isn't in crawling sites, but in updating the databases.
Has anyone put up a new site or new pages in a new directory in the last 14 days that has been crawled and included by Google?
Included meaning you can find them with some type of seaarch on google.
Please do not include news sites as they are spidered differently...
-s-
I assume it has to do with the 1 billion new pages that Google has to crawl. Their ressources probably cannot handle as much fresh content anymore.
I put up two new sites on February 26th. Neither is related to news in any way.
The concept for the sites did not even exist on the morning of the 26th and they were fully deployed by 5:00 that night.
As of today, (Feb 29th) both of them are showing up in the SERPS for a variety of search terms and they have fresh tags for February 27th.
Has anyone put up a new site or new pages in a new directory in the last 14 days that has been crawled and included by Google?
I published 100+ pages about a city in Germany on February 26. Two pages had links from my home page, and both were in the Google index when I checked last night.
The remaining pages don't seem to be indexed yet, although Googlebot has been so active that I suspect they've been crawled. (I'm also waiting for some other new pages to get indexed; these are pages from about a week ago that don't have links from the home page.)
In general, I'd say that Google isn't adding "fresh" pages to the index as quickly as it did a while back, but the delays don't appear to be extreme. Anyway, this thread began with the premise that Google is "not crawling or not crawling much." That isn't true, at least for those of us who have seen Googlebot in our logs every day.
As far them not indexing sites for the last two weeks I feel is due to the fact that the current algo is just a temporary "make everyone happy for a few weeks" algo. They wanted webmasters to be praising Google for bringing their site back from the dead while the whole Yahoo switch thing was starting up.
I think very soon we will be back to Florida/Austin results and the indexing will pick back up again.
I'd expect the reverse. With Yahoo no longer using Google, this means that Google is facing much greater competition than before. If Google went to a smaller index, this would mean more searcher dissatisfaction, and them switching to other SEs.
Where do these myths come from? Somebody has a site that doesn't get crawled, and they don't take a look at the serps where in the past several days we have been seeing fresh tags FROM THE SAME DAY. Yesterday at 6pm west coast time I had hundreds of Feb 28 fresh tags.
What Google seems to be doing less of is crawling huge (25,000+), poorly seo'ed sites, but aside from that Googlebot is far more active than usual for me... and the objective fact of the daily fresh tags is something that simply can't be ignored.
===
3:10 pacific, and seeing Feb 29 fresh tags
This was not some wild question based on one site doing poorly
-s-
Expecting new sites to be fully crawled within two weeks is expecting a lot at best. Index pages within two weeks, yes I'd expect that, but anything more is very optimistic. And new directories on existing domains... that just depends on the linking. A page off an index page might get hit immediately, a directory four levels deep might not get picked up for a long time.
There is a huge difference between adding new pages to the index, and very actively crawling domains that googlebot already values.... and adding a directory to a PR7 site, and adding one to a PR3 site.
-s-
... and I responded about new stuff.
Two brand new sites, non-existent before February 26th.
Two of them. Crawled the next day and included.
The directory that contains that review was crawled today and has a feb 29 fresh date. I am assuming that it was crawled because of the cookie crumbs on the page.
