Forum Moderators: open

Message Too Old, No Replies

Google and .cgi pages

I thought they weren't being crawled, but it looks like they are

         

MatthewHSE

3:01 pm on Mar 29, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Can anyone clear up a little problem I'm having understanding Google and .cgi pages?

I thought Google couldn't crawl .cgi pages. But, in conducting a Google search of my own site, I was startled to find several .cgi pages in the search results. Not only that, but several of the results were actually the results page of my own site search!

So is this the Google search bar at work, indexing pages on my site that my visitors have viewed, or is Google now actually crawling and indexing .cgi's? If so that's great news!

ciml

5:45 pm on Mar 29, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google has crawled 'dynamic' looking pages for quite a long time now, but in their efforts to expand the index the number of URLs with a '?' has increased over the last few months.

bufferzone

6:16 pm on Mar 29, 2004 (gmt 0)

10+ Year Member



Google crawls a lot of things. Generally if it on the web server it can be found on Google. Take a hard look on your Cgi scripts, some of them might be used by hackers and the like to do things you don’t want them to be able to do and some of them might contain vulnerabilities that can be used. Wit a little ingenuity you will be able to find a lot more then cgi scripts on Google. Things like passwords and usernames in clear text if you know how to search, Nice!

abates

10:02 pm on Mar 29, 2004 (gmt 0)

10+ Year Member



CGI pages on my site have been indexed in Google for years. You may find pages aren't being indexed if they have an "id" argument, and I think CGI pages with arguments don't show PR.