Forum Moderators: open

Message Too Old, No Replies

Will Google deepcrawl this kind of URL?

         

Jesse_Smith

5:23 am on Mar 4, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Will Google deepcrawl a site where the file names are like this?

/file.cgi?input_item=B123456789&input_search_type=ItemSearchÏ

OZZY2662

5:38 am on Mar 4, 2003 (gmt 0)

10+ Year Member



This may exceed the maximum length of URL that google will index.

jomaxx

5:51 am on Mar 4, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



My two cents...

1. Yes, Google can crawl .cgi's with parameters, as long as this is not disallowed by the robots.txt file.
2. It may take more indexing cycles for a site using this format for all its pages to get fully indexed.
3. What the 7734 is the last character in that sample URL?

Jesse_Smith

5:56 am on Mar 4, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Now I shrunk it down to

file.cgi?item=A123456789&type=Search

The * part of the URL shouldn't of been there. Not sure how those marks got added.

jomaxx

7:21 am on Mar 4, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Should have added that the URL doesn't look that long.

I don't know if there's a practical limit on what Google can spider at all, but I had no trouble finding a page in the index whose URL is 98 characters long, excluding the "http://".