homepage Welcome to WebmasterWorld Guest from 54.237.95.6
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 32 message thread spans 2 pages: 32 ( [1] 2 > >     
Nuts and Bolts Tips
How many of these do you know?
GoogleGuy




msg:150167
 6:07 am on Jul 21, 2003 (gmt 0)

Okay, I'm waiting for something to finish compiling, so I thought I'd write up a few webmaster tips. Most of these are on our help pages somewhere, but not many people know all of these tidbits. So here goes:

Tip #1: Use If-Modified-Since (IMS). IMS lets your webserver tell Googlebot whether a page has changed since the last time the page was fetched. If the page hasn't changed, we can re-use the content from the last time we fetched that page. That in turn lets the bot download more pages and save bandwidth. I highly recommend that you check to see if your server is configured to support If-Modified-Since. It's an easy win for static pages, and sometimes even pages with parameters can benefit from IMS.

Tip #2: You can use wildcards in robots.txt, and patterns can end in '$' to indicate the end of a name. So if you don't want Googlebot to fetch any PDF files, for example, you could say
Disallow: /*.pdf$
Don't forget that in the robots.txt file, all url patterns need a "/" anchor to be valid. That's a pretty common webmaster error (maybe the most common robots.txt mistake), so keep it in mind and save yourself some angst. :)

Tip #3: Googlebot also permits an "Allow" directive in robots.txt. This lets you specifically flag areas that are okay to crawl. When there are two directives that could apply, we follow the longest (i.e. most specific directive). See
[google.com...]
for an example.

Tip #4: Avoid session ID's. If you can, use fewer dynamic parameters and stay away from the parameter "id=" in urls--Googlebot tries to stay away from things that might be session ID's.

Tip #5: Make sure that you can reach every page on your site with a text browser like lynx. That's the best way to make sure that a spider can follow links to all of your pages. Site maps can be a really good way to help users and spiders get down into different parts of your site.

Some of these tips work mainly with Googlebot, but I hope that they help. Anybody else with nuts and bolts tips for site architecture, crawling, or robots.txt--throw 'em in! :)

 

skipfactor




msg:150168
 3:07 pm on Jul 21, 2003 (gmt 0)

Tip #2: You can use wildcards in robots.txt, and patterns can end in '$' to indicate the end of a name. So if you don't want Googlebot to fetch any PDF files, for example, you could say
Disallow: /*.pdf$

Well whack me in the head...SO...
Disallow: /*.pdf
will disallow *.pdf*, that is, anything with pdf in the prefix of the extension is disallowed right? (filename.pdf and filename.pdfx would be disallowed)

and

Disallow: /*.pdf$
will disallow *.pdf only (filename.pdf is disallowed and filename.pdfx would be allowed)

giggle




msg:150169
 3:24 pm on Jul 21, 2003 (gmt 0)

Thanks for the pointers GG.

Just to clarify; when you say stay away from ID= in a URL do you mean avoid using a variable like that even if you're not using any session variables?

tschild




msg:150170
 3:29 pm on Jul 21, 2003 (gmt 0)

Regarding #2 and #3: as these (wildcards and Allow:) are extensions of the robots.txt standard it would be well to keep in mind that these tips won't work for all or most search engines other than Google.

bhartzer




msg:150171
 3:31 pm on Jul 21, 2003 (gmt 0)

Thanks, GG, for telling us about these tips.

I would also like to add a link to a thread about if-modified-since: [webmasterworld.com...]

GoogleGuy




msg:150172
 4:22 pm on Jul 21, 2003 (gmt 0)

I believe that's right, skipfactor. I'll try to ask around and post if it's not. :)

claus




msg:150173
 4:31 pm on Jul 21, 2003 (gmt 0)

oh yes, valid code ;) I've found that most spiders like well prepared food. Not only the html, but also scripts, especially link/page redirect scripts (serverside that is, not JS/VB/ECMA... just forget these navigation-wise).

And lynx, that's a great point, this way you even support some screen readers as well as the bots :)

/claus

dmorison




msg:150174
 4:40 pm on Jul 21, 2003 (gmt 0)

That in turn lets the bot download more pages and save bandwidth. I highly recommend that you check to see if your server is configured to support If-Modified-Since. It's an easy win for static pages, and sometimes even pages with parameters can benefit from IMS.

Would it be possible to elaborate as to how Googlebot handles first time queries with respect to If-Modified-Since and "Last-Modified".

The RFC's imply that a client should only use this header if it holds an actual "Last-Modified" value received in response to a previous request - this is to avoid problems of time-synchronisation between hosts.

I've heard wind that Googlebot makes up an If-Modified-Since value that is ages ago and simply "hopes for the best", in which case i'm not sure I want to risk it.

I'm developing a forum system that has a "mod_rewritten" archive section that could certainly benefit both parties from using this header. The URI looks something like:

/archive/2003/07/21/1.html

and it would be a trivial matter to serve 304 ("Not Modified) if a request for this page offers an If-Modified-Since date greater than the date to which the page refers. Not sure I want to risk it though...

athinktank




msg:150175
 5:03 pm on Jul 21, 2003 (gmt 0)

GG,
Thanks for reiterating the basics. I have a question regarding session ids in the query string. Will Googlebot handle a cookie based session id system? All of our applications set a session id when a user first enters the site. The user, if they allow cookies, will have the session id set in this cookie. If they do not allow cookies, or refuse the cookie, we will write the session id in the query string.

Will Googlebot take the cookie and use the session id throughout its session?

Thanks

Yidaki




msg:150176
 5:24 pm on Jul 21, 2003 (gmt 0)

>Will Googlebot take the cookie and use the session id throughout its session?

Allthough i'm not GoogleGuy (honestly! ;) i can confirm that Google doesn't take/read/stores cookies at all.

athinktank, most session based software packages (forums, cms' etc.) put the session id into the url if cookies are disabled at the client site. Therefor GoogleGuy said: turn off (did he say cloak off?) session id's if googlebot visits your site. Often this is done with a single "exclude user agent line" within your session creation code.

Reason is pretty easy to understand: session id's are unique numbers and therefor a page that uses session id's in the request uri will get duplicated again and again everytime googlebot initiates a new visit at a session based site.

Yidaki




msg:150177
 5:27 pm on Jul 21, 2003 (gmt 0)

>Tip #5: Make sure that you can reach every page on your site with a text browser like lynx.

...or do the same with one click using the Search Engine Spider Simulator [searchengineworld.com] at sew. ;)

mil2k




msg:150178
 5:37 pm on Jul 21, 2003 (gmt 0)

Regarding Site architecture just thought to add that if you have a new site and your emphasis is on getting more pages crawled than go for Breadth rather than Depth.

coolasafanman




msg:150179
 5:53 pm on Jul 21, 2003 (gmt 0)

I have 2 varsions of a page:
[domain.com...]
and
[domain.com...]

how do I tell Google to read /blah, but to not crawl /blah?something=*

Thanks, and excellent post.

dmorison




msg:150180
 5:56 pm on Jul 21, 2003 (gmt 0)

how do I tell Google to read /blah, but to not crawl /blah?something=*

You could add the HTML:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

to your page where something <> "", although Googlebot might go and apply the META to the page that exists without the?something, so i'd be a bit carefull.

coolasafanman




msg:150181
 6:08 pm on Jul 21, 2003 (gmt 0)


Googlebot might go and apply the META to the page that exists without the?something, so i'd be a bit carefull.

yeah, that's what i'm worried about.

tschild




msg:150182
 6:39 pm on Jul 21, 2003 (gmt 0)


I have 2 varsions of a page:
[domain.com...]
and
[domain.com...]
how do I tell Google to read /blah, but to not crawl /blah?something=*

That should be covered with an entry

Disallow: /blah?something

in your robots.txt.

coolasafanman




msg:150183
 6:56 pm on Jul 21, 2003 (gmt 0)

k thanks :-)

cjtripnewton




msg:150184
 4:36 pm on Jul 25, 2003 (gmt 0)

Bumping this useful thread.

Redguy




msg:150185
 6:30 pm on Jul 26, 2003 (gmt 0)

I am in a bit of a pickle...My store uses SIDs so in order to make sure googlebot could get to the pages it needed to with out getting lost in the SIDs I served the googlebot a page that would not create SIDs, The page looked very similar to my homepage but simply had some description text and links to all my products. After talking with a good number of people some said this practice was cloaking and some said it was OK.

Never the less I just noticed that I fell off of the 1st page for my most popular keyword. So I have been loosing sleep at night thinking that I pissed off the googlebot when all I was trying to do was help it get through site and not get tangled in a web of SIDs. As a result I removed the "cloaked" page and now I am left with what I am sure will be a SID mess. Any suggestions? I am very new to the world of SEO so any advice will be greatly appreciated.

figment88




msg:150186
 6:38 pm on Jul 26, 2003 (gmt 0)

For tip number 3, does Google look explicitly for "id=" paramater or does it also use partial matching. For example, if I use a paramter like "jobid=" am I asking for trouble.

rcjordan




msg:150187
 6:43 pm on Jul 26, 2003 (gmt 0)

>Tip #4: Avoid session ID's.

Let me bookmark that one, GG. We had a big go-round after you confirmed this in that CMS thread a while back.

vincevincevince




msg:150188
 6:54 pm on Jul 26, 2003 (gmt 0)

For tip number 3, does Google look explicitly for "id=" paramater or does it also use partial matching. For example, if I use a paramter like "jobid=" am I asking for trouble.

ALARM BELLS Urgent mass variable changing...

Nick_W




msg:150189
 7:01 pm on Jul 26, 2003 (gmt 0)

>>Avoid session ID's

So, we should show Googlebot pages that don't contain session id's and (assuming our sites require it) show our users pages that do...?

Isn't that cloaking? hehehehehe ;)

Nick

WebGuerrilla




msg:150190
 8:24 pm on Jul 26, 2003 (gmt 0)


Detecting Googlebot in order to serve a session/cookie free page is not considered cloaking.

Yidaki




msg:150191
 8:25 pm on Jul 26, 2003 (gmt 0)

>Isn't that cloaking? hehehehehe ;)

huhuh, wake up! ;)

Here's a ancient statement of GG (Dec 4, 2002):
Google and Session Killing [webmasterworld.com]

Everybody knows I'm pretty anti-cloaking, but WebGuerilla has already made strong points why it's okay to drop a session id for Google.<snip snap>
This is just my personal take, but allowing Googlebot to crawl without requiring session id's should not run afoul of Google's policy against cloaking. I encourage webmasters to drop session id's when they can. I would consider it safe. Fair enough?

Hope that helps,
googleguy


Nick_W




msg:150192
 8:26 pm on Jul 26, 2003 (gmt 0)

Says who? - Details please WG..

Added:
>>This is just my personal take (GG from that thread)

Nick

[edited by: Nick_W at 8:27 pm (utc) on July 26, 2003]

Yidaki




msg:150193
 8:27 pm on Jul 26, 2003 (gmt 0)

see above, Nick. ;)

Nick_W




msg:150194
 8:28 pm on Jul 26, 2003 (gmt 0)

...and I just edited heheh!

Nick

Yidaki




msg:150195
 8:31 pm on Jul 26, 2003 (gmt 0)

Well, yes - his personal take ... hmm, we don't start this discussion again about GoogleGuy's relation to Google Inc., or!? ;)

btw: my personal experience is that google doesn't have any problem with my "cloaked" forums - it eats 'em.

<added>nevertheless, good edit, Nick</added>

Nick_W




msg:150196
 8:35 pm on Jul 26, 2003 (gmt 0)

Yeah, my statement was clearly a little 'tongue in cheek' (which I thought I made clear with the 'hehehehehe' apologies if it wasnt clear enough...) - I cloak for that all the time and have never had a problem...

Nick

This 32 message thread spans 2 pages: 32 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved