Forum Moderators: open
First of all, I have to admire the knowledge that some of you have on the overall functionality of Google...incredible!
I am pretty much new to all that, but I have learnt a lot reading many posts.
One thing is still unclear for me.
I have created a brand new site, and submitted to Google a month ago. Last week, on the 3rd, 4th and 5th of July 2003, Google came on my site.
My idea of this was that Google would actually look at unique files like mydomain.com/something.html but in my reports it shows that Google looked at my robots.txt file a few times.
So first: is this okay? Will Google come back to look at my multiple page URLs?
And second: how long should it take for my site to show up in Google? A week, a month or more?
I thank you all in advance for sharing your experience with this.
Bobosse
Next, sit back and relax. You'll show up in the index eventually.
Consider yourself lucky that you didn't start your web site about 4 months ago like I did, just before all the strange update problems -- the first deep crawl from Google didn't happen until last week.
Welcome to WW,
Google seems to be picking up pages quicker now and you can get indexed within hours :) - however I would not panic if you are not this probably largely depends on links to your site and freshness.
Assuming that Google did have a deep(er) crawl last week and assuming that there will be a more traditional update then the crawl from last week could appear in 2-3 weeks (or so - maybe less maybe more)
Hard to say with the latest developments at Google but good luck and I am sure you will appear in the index sooner or later - after all it looks like Google found you OK - so it should work out in the end.
This as of this morning... So, the 'traditional update' is, for me (provided these pages stay 'stuck' in the serps), looking like the dinosaurs...
Quick question: How do you know what pages are actually crawled by Googlebot? Since I only see my robots.txt file to have been indexed, will I see the actual pages to be indexed by the bot later on this month? is it a 2 steps process for Google to index pages?
What are the "SERPS"?
Bobosse
Now, what if I change a page URL.
Let's say I had a contact.html page last week when Google came, but yesterday I have changed this for contact-us.html.
Will Google look for contact.html? Will Google also take the contact-us.html?
I may have done a mistake to change a few file names...did I?
Thanks!
Bobosse
Thanks -
jd
I thought when google does its deepcrawl and comes to visit your site it will grab all the pages that exist. However when I was speaking to my webmaster he stated that they only got about half of my pages and are still working on getting the rest.
Also the weird thing about it all is that I am getting the same amount of traffic before my site was indexed on google that I am now after the index. Logically this does not make sense at all. I say this because several of my keywords are rank high and these keywords are a popular search according to the numbers.
Is it because Google may still be unsettled and my backlinks may be popping in and out? Please advise.
Showed up with no fresh tags.
Don't know how this works into the rolling update.
I created a site three months ago. Got the same combination of links that have worked in the past (DMOZ, YAHOO, state organization, national organization, affiliated national organizations, local directories, etc)
I manually submitted our front (index) page and that's all that has been spidered or added to the index. The bots' stroll by, request robots.txt (which is there) but then move on without scanning the site. [We use the sample robots.txt file from Searchengineworld, basically]
Has Google changed it's criteria for non commercial sites? Is there some threshhold before the bots will crawl our site?
You should only need at least one incoming link of at least PR4 to get you started. Run the pages through [validator.w3.org...] in case there is something tripping the spiders up. Next, find a long phrase on any one of your pages then do a search for that phrase on Google. See if someone else has duplicated your content and is stopping you being listed.
There are other things to try, but those are the two I would do first.
Fearless: sounds like there are problems with the site.
This is my fourth one. There is nothing significantly changed in terms of site creation, file layout, etc.
A site first online 2003-05-04 dropped cleanly into the index on 2003-06-16 with all pages listed (still no PR though).
That may be true, however, there have been other posts from people in similar straits as myself.
You should only need at least one incoming link of at least PR4 to get you started.
I've got well over that, plus DMOZ and Yahoo.
Run the pages through [validator.w3.org...] in case there is something tripping the spiders up. Next, find a long phrase on any one of your pages then do a search for that phrase on Google. See if someone else has duplicated your content and is stopping you being listed.
Nope, not the problem.
G-guy has responded to similar posts on other threads and he always says about what he did above
Over time, Google should find more of your pages, especially as more people link to you when they find your site.
I note that the dictionary defines "should" as "ought to, but not necessarily will.""
Googleguy's posts on this topic have been remarkably vague. Clearly, something has changed [regarding new non commercial modest size sites] and apparently he's not at liberty to discuss it, other than to offer reassurances so vague that nothing can be read into them.
Two sites ... one commercial, one non-commercial
Commercial is linked from dmoz, yahoo, business.com, joeant, and two of my own on-topic PR7 and PR7 sites. Also an ad on a PR9 site [very on topic] and a PR6 [also on topic]
Just index page grabbed in 2 months, and the one being shown itself is 1 month old.
Page Rank: 1
Pages Indexed: 1
Total Pages: 204
Non commercial .. also in Yahoo, zeal, goguides, joeant, plus also a mention on the official site!
Just index page grabbed.
Page Rank: 0 [n/a]
Pages Indexed: 1
Total Pages: 10,521
The sites are very clean, zero spam, not 'overly' optimised. I have several other sites on this same server and IP [its a dedicated] and they are getting crawled left and right so it cannot be an IP ban.
Manually submitted too, and emailed webmaster@ [no response]
*sigh*
as more people link to you when they find your site
He is obviously implying that backlinks are what we need.
More and more of the logical, (legitimate) high ranking backlinks for my sites are script generated. I've already mentioned that "jump menu" links don't register with the google bots. (Even though the "links" query isn't working for my latest site) just by looking at the results for "widget+county+widget+party" several of our linking sites are returned.
But none of the ones from jump menus OR php pages show up! (Depsite their high "significance.")
Hang in there Ahmed.
I'm pullin' for ya. We're all in this together!"
Its weird ... suddenly about 80 pages were added to the index .. .but only 3 of them have a cache to show, and the main index page still shows the 4 week old version :-/
[newb question - when/why is there no cache to be shown for a page?]
I thought Google deepcrawls for a particular month picks up all spiderable pages on your site that you have at that time? Why is it then that only half of my spiderable pages were picked up by google during last(June) month and not all of them?
Am I incorrect of this, or does it take google several months to pick up all pages that are spiderable?
My site is only a couple of months old, could this be why?
When will I know exactly how well my site has done in regards to google PR, indexing etc....
Does someone out there have an answer for me?
Conclusion - for our new site is that it took 2 weeks for our new pages to show up following the first deep crawl of the site.
[edited by: Fearless at 6:48 pm (utc) on July 13, 2003]
site:yourdomain.org -qwerrew
I think that another problem for us under the new regimen, (besides jump menus, php and asp backlinks) is that ALL of our backlinks are to the site not to individual pages.
(and it's likely to stay that way.....)
Googleguy was nice enough to provide this google query (sort of like the "links:" query) that shows exactly what Google has indexed of your sitesite:yourdomain.org -qwerrew
-Bob
Replace the yourdomain with, well, your domain.
.
You can also add &num=100 to the end of the search URL in the browser address bar, to get 100 results per page, and add &filter=0 as well if you want to see all of the results.
My sites approx 2 months old and Google has about half my pages. Anyone have the answer?
Certainly allow 2 or 3 months, maybe more on bigger sites.
