Any PERL gurus in the house - LWP question?

[goldrush]

I'm a newbie to LWP and trying to get my HTTP headers to properly emulate my browser. However, LWP keeps adding a TE header which I want to get rid of. So far I have:

#!/usr/bin/perl

use warnings;
use strict;
use LWP::UserAgent;

# this page just prints the headers sent by the client
my $url = 'http://localhost/go.php';

# initialize browser and set user agent
my $browser = LWP::UserAgent->new();
$browser->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');

# create new http request, clear headers and add a new header
my $request = HTTP::Request->new(GET => $url);
$request->clear;
$request->header('Connection' => 'keep-alive');

# print the http headers before making the request
my $headers = $request->headers_as_string;
print "Headers before http request are:\n";
print "$headers";

# send the http request and print the headers echoed back by  $url
my $response = $browser->request($request);
my $contents = $response->decoded_content;
print "Returned headers are:\n";
print "$contents";

The output is:

Headers before http request are:
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Returned headers are:
Connection: keep-alive, TE, close
Host: localhost
TE: deflate,gzip;q=0.3
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

Why is LWP adding the TE header and the extra parts to the connection header? How can I stop this behaviour?

[flashbang]

14.39 TE
The TE request-header field indicates what extension transfer-codings it is willing to accept in the response and whether or not it is willing to accept trailer fields in a chunked transfer-coding. Its value may consist of the keyword "trailers" and/or a comma-separated list of extension transfer-coding names with optional accept parameters (as described in section 3.6).

TE = "TE" ":" #( t-codings )
t-codings = "trailers" | ( transfer-extension [ accept-params ] )
The presence of the keyword "trailers" indicates that the client is willing to accept trailer fields in a chunked transfer-coding, as defined in section 3.6.1. This keyword is reserved for use with transfer-coding values even though it does not itself represent a transfer-coding.

Examples of its use are:

TE: deflate
TE:
TE: trailers, deflate;q=0.5
The TE header field only applies to the immediate connection. Therefore, the keyword MUST be supplied within a Connection header field (section 14.10) whenever TE is present in an HTTP/1.1 message.

[flashbang]

First find out what it is, then figure out why

[goldrush]

Hmmm so the TE stuff is actually being added by the server because the message is HTTP/1.1? So the headers being sent by LWP::UserAgent client are actually correct? I am using the PHP function apache_request_headers() on a page on localhost to echo back the request headers.

[drjones]

Does it actually cause an issue for you to have that there? If not probably best to just leave it.

You can remove content from the header easily enough:

$request->remove_header('TE');

[goldrush]

Quick solution:

Edit /usr/lib/perl5/site_perl/5.8.8/LWP/Protocol/http by hand, and change this part of the new_socket subroutine:

local($^W) = 0;  # IO::Socket::INET can be noisy
    my $sock = $self->socket_class->new(PeerAddr => $host,
					PeerPort => $port,
					Proto    => 'tcp',
					Timeout  => $timeout,
					KeepAlive => !!$conn_cache,
					SendTE    => 1,
					$self->_extra_sock_opts($host, $port),
				       );

Manually change the value of SendTE to 0, and you will no longer have to worry about TE headers being sent. I'm trying to sort out a more elegant solution though if anyone can explain to this LWP noob!