Forum Moderators: open

Message Too Old, No Replies

Robots.txt with dynamic URLs?

Will googlebot understand?

         

eaden

3:14 pm on Jan 26, 2003 (gmt 0)

10+ Year Member



I am looking to limit some search engine requests after some dynamic urls appeared in the index that really shouldn't be there, e.g. reply to a forum.

The robots exclusion standard isn't to clear on this, but i'm wondering :

# robots.txt
User-agent: *
Disallow: /forum.php?action=post

will that work?
Or will googlebot ignore everything after the ? and not index forum.php at all?

I'm only interested in googlebot, as the results may differ between search engines.

Brett_Tabke

4:11 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



It _should_ work with Google - cross fingers.

jamesyap

5:39 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



> will that work?
Definitely, it will block everything same or start with

/forum.php?action=post

means it will not crawl

/forum.php?action=post
/forum.php?action=post&what=iswhat
/forum.php?action=postoffice

but it will crawl

/forum.php?what=iswhat&action=post

remember it is a matching game. ;)

GoogleBot love dynamics pages, they are delicious!

eaden

1:24 am on Jan 29, 2003 (gmt 0)

10+ Year Member



O.k. I'd really like to know if anyone does this, because if I get it wrong, it's the equilivant of doing
Disallow: /mypage.php - which is _all_ my content!