Forum Moderators: DixonJones

Message Too Old, No Replies

Submittin an HTTP request "manually"

         

pawel

7:16 pm on May 14, 2003 (gmt 0)

10+ Year Member



Let's say a page is cloaked and I get a different content than what SE get.
How can I submit (on Win 2000 ) an HTTP request, so as I pretend that I really am the Google bot?
In other words,
(1) how should such a request look like,
(2) what to do to send HTTP requests without the use of agents like IE?
Thanks,
Pawel

takagi

7:22 pm on May 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



What about a Search Engine Spider Simulator [searchengineworld.com]?

bcc1234

7:18 am on May 13, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Get a telnet client and connect to the port 80 on that host.
Once the connection is established type:
GET /filename HTTP/1.1
Host: hostname
User-Agent: agent
Whatever: whatever

then hit enter twices and read the response.

count_zer0

10:24 am on May 22, 2003 (gmt 0)

10+ Year Member



My site has a tool which allows you to retrieve the raw HTTP data from a site. You can also set the User-Agent string. Go to the site in my profile and select "HTTP Viewer".

Note that most professional cloaking is done using IP detection, not User-Agent detection, so you'll only catch out the amateurs.