Forum Moderators: open

Message Too Old, No Replies

problematic urls

problematic urls

         

magicsoftware

9:28 am on Nov 24, 2002 (gmt 0)

10+ Year Member



does anyone know if such urls can be followed by spiders?

[mydomain.com...] Who=tech%5El1%5EseDeveloper&enZone=tech&enVersion=0&enretain=branch
(no space - that was edited in.)

the spiders all reach and index my root-level static files but don't seem to follow meta or js redirects (redirecting to such urls), nor follow any of the links!

thanks very much.

[edited by: heini at 10:35 am (utc) on Nov. 24, 2002]

andreasfriedrich

3:23 pm on Nov 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to WebmasterWorld [webmasterworld.com], magicsoftware.

In this message [webmasterworld.com] GoogleGuy said that Google is getting better about crawling dynamic urls. But from membersī experience this seems to be true for pages with a high page rank only. Furthermore the higher the number of parameter in the URL the less likely will SEs spider your pages.

Your URIs seem to be rather long. I donīt think that they will be spidered.

Some relevant threads:
theory on dynamic pages [webmasterworld.com]
We Can Finally Put it to Rest, No If's, Ands, or Buts. [webmasterworld.com]
PR and dynamic URLs [webmasterworld.com]

This will give you even more to read:
site search for dynamic urls [searchengineworld.com]

Andreas

magicsoftware

2:41 pm on Nov 25, 2002 (gmt 0)

10+ Year Member



Thanks a lot, Andreas!
I read through everything I'm surprised and glad that there is such a place where theories on SEs' behavior can be brought up and discussed.

Currently I'm checking a simple workaround, which I couldn't find on this site and may be worth mentioning: most SEs will not follow javascript or Meta refreshes. but seems that they will index a dynamic page when replied with a server redirection such as response.sendRedirect(url) (in JSP, or whatever the equivalent is in ASP). the redirection can be to a dynamic page that can otherwise only be reachable by a long url, so this may be the only way to get them to index these pages.

I'm studying this - and i have a feeling it works, at least for Google. I'll update you when i have more accurate results.

BTW i'm not sure this is the right forum for this discussion. maybe this thread should be moved.