GoFuckYourself.com - Adult Webmaster Forum - Any PERL gurus in the house

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - Any PERL gurus in the house - LWP question? (https://gfy.com/showthread.php?t=629619)

Any PERL gurus in the house - LWP question?

I'm a newbie to LWP and trying to get my HTTP headers to properly emulate my browser. However, LWP keeps adding a TE header which I want to get rid of. So far I have:

Code:

#!/usr/bin/perl



use warnings;

use strict;

use LWP::UserAgent;



# this page just prints the headers sent by the client

my $url = 'http://localhost/go.php';



# initialize browser and set user agent

my $browser = LWP::UserAgent->new();

$browser->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');



# create new http request, clear headers and add a new header

my $request = HTTP::Request->new(GET => $url);

$request->clear;

$request->header('Connection' => 'keep-alive');



# print the http headers before making the request

my $headers = $request->headers_as_string;

print "Headers before http request are:\n";

print "$headers";



# send the http request and print the headers echoed back by  $url

my $response = $browser->request($request);

my $contents = $response->decoded_content;

print "Returned headers are:\n";

print "$contents";

The output is:

Code:

Headers before http request are:

Connection: keep-alive

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Returned headers are:

Connection: keep-alive, TE, close

Host: localhost

TE: deflate,gzip;q=0.3

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Why is LWP adding the TE header and the extra parts to the connection header? How can I stop this behaviour?

14.39 TE
The TE request-header field indicates what extension transfer-codings it is willing to accept in the response and whether or not it is willing to accept trailer fields in a chunked transfer-coding. Its value may consist of the keyword "trailers" and/or a comma-separated list of extension transfer-coding names with optional accept parameters (as described in section 3.6).

TE = "TE" ":" #( t-codings )
t-codings = "trailers" | ( transfer-extension [ accept-params ] )
The presence of the keyword "trailers" indicates that the client is willing to accept trailer fields in a chunked transfer-coding, as defined in section 3.6.1. This keyword is reserved for use with transfer-coding values even though it does not itself represent a transfer-coding.

Examples of its use are:

TE: deflate
TE:
TE: trailers, deflate;q=0.5
The TE header field only applies to the immediate connection. Therefore, the keyword MUST be supplied within a Connection header field (section 14.10) whenever TE is present in an HTTP/1.1 message.

First find out what it is, then figure out why :)

Hmmm so the TE stuff is actually being added by the server because the message is HTTP/1.1? So the headers being sent by LWP::UserAgent client are actually correct? I am using the PHP function apache_request_headers() on a page on localhost to echo back the request headers.

Does it actually cause an issue for you to have that there? If not probably best to just leave it.

You can remove content from the header easily enough:

Code:

$request->remove_header('TE');

Quick solution:

Edit /usr/lib/perl5/site_perl/5.8.8/LWP/Protocol/http by hand, and change this part of the new_socket subroutine:

Code:

local($^W) = 0;  # IO::Socket::INET can be noisy

    my $sock = $self->socket_class->new(PeerAddr => $host,

                                        PeerPort => $port,

                                        Proto    => 'tcp',

                                        Timeout  => $timeout,

                                        KeepAlive => !!$conn_cache,

                                        SendTE    => 1,

                                        $self->_extra_sock_opts($host, $port),

                                       );

Manually change the value of SendTE to 0, and you will no longer have to worry about TE headers being sent. I'm trying to sort out a more elegant solution though if anyone can explain to this LWP noob!