Gibberish displayed when content has unicode, can an option be added to control utf8 flag into Log::Dispatch ?

[hadjiprocopis]

I get gibberish on my console when running the following, with or without binmode'ing STDERR:

use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw( debug_ua );

# gibberish with or without it
#binmode STDERR, ':encoding(UTF-8)';

my $ua = LWP::UserAgent->new();
my $logger = debug_ua($ua, 5);
# greek content
#my $res = $ua->get('https://tvxs.gr');
my $res = $ua->get('https://www.greekfontsociety-gfs.gr/about');

What works for me (well ok I get question-marks for a few characters) is to tell $logger to use a new Log::Dispatch with the utf8 flag turned off, as follows:

use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw( debug_ua );
use Log::Dispatch;

my $ua = LWP::UserAgent->new();
my $logger = debug_ua($ua, 5);

$logger->logger(Log::Dispatch->new(
  outputs => [
    [ 'Screen', min_level => 'debug', newline => 1, utf8 => 0, ],
  ],
));
 
# greek content
#my $res = $ua->get('https://tvxs.gr');
my $res = $ua->get('https://www.greekfontsociety-gfs.gr/about');

Is there a simpler way to display the unicode content correctly?

Or can you allow controlling utf8 via LWP::ConsoleLogger::Easy?

[oalders]

Oof. This is not great. As much as I hate to say it, I think we'd probably need to control this via an environment variable, so that LWP::ConsoleLogger::Everywhere would also be able to benefit from it.

If you're motivated to put a pull request together, I'm happy to look at it.

Are you able to test out LWP::ConsoleLogger::Everywhere with the LWPCL_LOGFILE variable set? I'd be interested to know what the output in the file looks like for you.

[hadjiprocopis]

Thank you!

with this:

use LWP::UserAgent;
use LWP::ConsoleLogger::Everywhere;

my $ua = LWP::UserAgent->new();
# greek content
my $res = $ua->get('https://www.greekfontsociety-gfs.gr/about');
#my $res = $ua->get('https://tvxs.gr');

and this,

LWPCL_LOGFILE=foo.log perl ./gibberish-debugua-with-everywhere.pl

I got this on STDERR

Wide character in print at /opt/perlbrew/perls/perl-5.38.2/lib/site_perl/5.38.2/Log/Dispatch/File.pm line 150.
Wide character in print at /opt/perlbrew/perls/perl-5.38.2/lib/site_perl/5.38.2/Log/Dispatch/File.pm line 150.

and this log file:

foo.log

Which renders unicode fine except for the very few questionmark chars (e.g. the contents of theTitle response header, not sure if it is a glitch of the site, BUT the contents of body, e.g . its <title> render fine so it seems that headers are treated somewhat differently and wrongly?)

PLUS the 2 warnings about wide characters as above.

Curiously, when not saving to the log file via env-var but redirection (linux terminal), I do not get the Wide chars warning but the Title header is gibberish (αιÏεία ÎλληνικÏν) often seen in double encoding. AND the contents of body renders fine.

perl ./gibberish-debugua-with-everywhere.pl &> foo2.log

foo2.log

So, it seems LWP::ConsoleLogger::Everywhere behaves much better except for headers, or , at least, the Title header.

[oalders]

I am not a subject matter expert when it comes to encodings, so I'm happy to defer to you on a solution. I'll also ping @simbabque here, as he wrote Everywhere.pm and he'll be sitting across from me for a few days. 😄