Any PERL gurus in the house - LWP question?

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • goldrush
    Will trade SE hits for CJ hits
    • Apr 2006
    • 661
    #1

    Any PERL gurus in the house - LWP question?

    I'm a newbie to LWP and trying to get my HTTP headers to properly emulate my browser. However, LWP keeps adding a TE header which I want to get rid of. So far I have:
    Code:
    #!/usr/bin/perl

use warnings;
use strict;
use LWP::UserAgent;

# this page just prints the headers sent by the client
my $url = 'http://localhost/go.php';

# initialize browser and set user agent
my $browser = LWP::UserAgent->new();
$browser->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');

# create new http request, clear headers and add a new header
my $request = HTTP::Request->new(GET => $url);
$request->clear;
$request->header('Connection' => 'keep-alive');

# print the http headers before making the request
my $headers = $request->headers_as_string;
print "Headers before http request are:\n";
print "$headers";

# send the http request and print the headers echoed back by  $url
my $response = $browser->request($request);
my $contents = $response->decoded_content;
print "Returned headers are:\n";
print "$contents";
    The output is:
    Code:
    Headers before http request are:
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Returned headers are:
Connection: keep-alive, TE, close
Host: localhost
TE: deflate,gzip;q=0.3
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
    Why is LWP adding the TE header and the extra parts to the connection header? How can I stop this behaviour?
    217303611
    Tags: None
    • flashbang
      Confirmed User
      • May 2006
      • 767
      #2
      14.39 TE
      The TE request-header field indicates what extension transfer-codings it is willing to accept in the response and whether or not it is willing to accept trailer fields in a chunked transfer-coding. Its value may consist of the keyword "trailers" and/or a comma-separated list of extension transfer-coding names with optional accept parameters (as described in section 3.6).

      TE = "TE" ":" #( t-codings )
      t-codings = "trailers" | ( transfer-extension [ accept-params ] )
      The presence of the keyword "trailers" indicates that the client is willing to accept trailer fields in a chunked transfer-coding, as defined in section 3.6.1. This keyword is reserved for use with transfer-coding values even though it does not itself represent a transfer-coding.

      Examples of its use are:

      TE: deflate
      TE:
      TE: trailers, deflate;q=0.5
      The TE header field only applies to the immediate connection. Therefore, the keyword MUST be supplied within a Connection header field (section 14.10) whenever TE is present in an HTTP/1.1 message.



      Your TGP traffic belongs to you again       |
      X Rated Words & Hard Links

        Comment

        • flashbang
          Confirmed User
          • May 2006
          • 767
          #3
          First find out what it is, then figure out why



          Your TGP traffic belongs to you again           |
          X Rated Words & Hard Links

            Comment

            • goldrush
              Will trade SE hits for CJ hits
              • Apr 2006
              • 661
              #4
              Hmmm so the TE stuff is actually being added by the server because the message is HTTP/1.1? So the headers being sent by LWP::UserAgent client are actually correct? I am using the PHP function apache_request_headers() on a page on localhost to echo back the request headers.
              217303611

                Comment

                • drjones
                  Confirmed User
                  • Oct 2005
                  • 908
                  #5
                  Does it actually cause an issue for you to have that there? If not probably best to just leave it.

                  You can remove content from the header easily enough:

                  Code:
                  $request->remove_header('TE');
                  ICQ: 284903372

                    Comment

                    • goldrush
                      Will trade SE hits for CJ hits
                      • Apr 2006
                      • 661
                      #6
                      Quick solution:

                      Edit /usr/lib/perl5/site_perl/5.8.8/LWP/Protocol/http by hand, and change this part of the new_socket subroutine:
                      Code:
                      local($^W) = 0;  # IO::Socket::INET can be noisy
    my $sock = $self->socket_class->new(PeerAddr => $host,
					PeerPort => $port,
					Proto    => 'tcp',
					Timeout  => $timeout,
					KeepAlive => !!$conn_cache,
					SendTE    => 1,
					$self->_extra_sock_opts($host, $port),
				       );
                      Manually change the value of SendTE to 0, and you will no longer have to worry about TE headers being sent. I'm trying to sort out a more elegant solution though if anyone can explain to this LWP noob!
                      217303611

                        Comment

                        Previous template Next
                        Working...