utf8::all breaks Perl black magic of while ( readdir )

[van-de-bugger]

$ cat ./test.pl 
#!/usr/bin/perl

use strict;
use warnings;

$\ = "\n";
opendir( my $dh, '.' ) or die;
while ( readdir $dh ) {
    print;
};

$ ./test.pl 
.
..
0
test.pl

Note the file named "0" in the test output.

Now let's use utf8::all module:

$ perl -Mutf8::all ./test.pl 
Use of uninitialized value $_ in print at ./test.pl line 9.

Use of uninitialized value $_ in print at ./test.pl line 9.

Oops. There is no output at all, but two warnings that $_ is not defined. utf8::all replaces readdir with own implementation, _utf8_readdir, which is not a subject for Perl black magic: Perl does not assign the result of _utf8_readdir to the $_.

If I slightly rewrite the test to avoid using $_, the situation is even worse:

$ cat ./test.pl 
#!/usr/bin/perl

use strict;
use warnings;

opendir( my $dh, '.' ) or die;
while ( my $entry = readdir $dh ) {
    print "$entry\n";
};

$ ./test.pl 
.
..
0
test.pl

$ perl -Mutf8::all ./test.pl 
.
..

No warning, for the first look everything seems ok… but entry "0" and all entries after it are not printed. Silently.

The reason is the same: Perl black magic doesn't work with _utf8_readdir, and while loop checks condition for "trueness", not for "definedness", while "0" evaluates to false.

I have no idea whether it possible to fix the problem. But anyway, the utf8::all documentation should document this side effect in the section "CAVEATS".

Versions:

$ perl -v

This is perl 5, version 42, subversion 2 (v5.42.2) built for x86_64-linux-thread-multi
(with 15 registered patches, see perl -V for more detail)

Copyright 1987-2026, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at https://www.perl.org/, the Perl Home Page.

$ perl -Mutf8::all -e 'print $utf8::all::VERSION, "\n";'
0.024

With the latest version 0.026 results are the same.

[Leont]

Yeah that's annoying. I have an idea of how to fix that, but it's all very non-trivial.

[van-de-bugger]

I was able to hack utf8::all so my trivial test works the same if utf8::all included or not. Here is the fix:

package utf8::all::TrueStr {
    use overload
        '""'     => sub { ${ $_[ 0 ] } },
        '0+'     => sub { ${ $_[ 0 ] } },
        'bool'   => sub { 1 },
        fallback => 1,
        ;
    sub new {
        my ( $class, $str ) = @_;
        return defined( $str ) ? bless( \$str, $class ) : undef;
    };
};

sub _utf8_readdir(*) { ## no critic (Subroutines::ProhibitSubroutinePrototypes)
    my $pre_handle = shift;
    my $hints = (caller 0)[10];
    my $handle = ref($pre_handle) ? $pre_handle : qualify_to_ref($pre_handle, caller);
    if (not $hints->{'utf8::all'}) {
        return $_ = utf8::all::TrueStr->new(CORE::readdir($handle));                           # <<<=== NOTE
    } else {
        $UTF8_CHECK |= Encode::LEAVE_SRC if $UTF8_CHECK; # Enforce LEAVE_SRC
        if (wantarray) {
            return map { $_ ? $_UTF8->decode($_, $UTF8_CHECK) : $_ } CORE::readdir($handle);
        } else {
            $_ = CORE::readdir($handle);                                                       # <<<=== NOTE
            return defined $_ ? utf8::all::TrueStr->new($_UTF8->decode($_, $UTF8_CHECK)) : $_; # <<<=== NOTE
        }
    }
}

I added utf8::all::TrueStr class and fixed 3 lines in _utf8_readdir function.

However, I am not sure what is better — break existing black Perl magic or introduce new one.

[HayoBaan]

@van-de-bugger thanks for the issue report and suggested fix. It breaks the tests though; the not utf8::all branch needs wantarray handling too. Also: doesn't the same issue apply to the other overridden methods (readlink, glob) as well?

But like you said, I'm unsure whether we should implement this or add a note to the documentation. However, that fact that the loop without using $_ breaks at files named 0, is worrisome and something that needs fixing regardless (and that is also something your true string wrapper addresses).

@Leont what are your thoughts on this? Did you have a similar fix in mind?

[Leont]

@Leont what are your thoughts on this?

I don't like this amount of magic, and it's quite likely to break something (e.g. something checking if its argument is a reference).

Did you have a similar fix in mind?

Not at all. I was considering doing opcode wrapping instead of wrapping things in functions. This involves a lot of XS though.

[HayoBaan]

@Leont what are your thoughts on this?

I don't like this amount of magic, and it's quite likely to break something (e.g. something checking if its argument is a reference).

Yes, that's a bit problematic, but the current implementation has some things going wrong too (things I wasn't aware of until now). I'm especially unhappy by how 0 filenames behave differently from the original.

Did you have a similar fix in mind?

Not at all. I was considering doing opcode wrapping instead of wrapping things in functions. This involves a lot of XS though.

If you can help me there, I would be grateful; my XS experience is virtually non-existent.

[van-de-bugger]

@HayoBaan, yes, you are right. I didn't run utf8::all tests.

Also: doesn't the same issue apply to the other overridden methods (readlink, glob) as well?

The Perl documentation says:

If the condition expression of a while statement is based on any of a group of iterative expression types then it gets some magic treatment. The affected iterative expression types are readline, the <FILEHANDLE> input operator, readdir, glob, the <PATTERN> globbing operator, and each. If the condition expression is one of these expression types, then the value yielded by the iterative operator will be implicitly assigned to $_. If the condition expression is one of these expression types or an explicit assignment of one of them to a scalar, then the condition actually tests for definedness of the expression's value, not for its regular truth value.
— https://perldoc.perl.org/perlsyn

So, the issue is also applicable to glob, but not to readlink.

I'm especially unhappy by how 0 filenames behave differently from the original.

BTW, TrueStr can be applied only to directory entries named "0". There is no need to apply TrueStr to all entries:

package utf8::all::TrueStr {
    use overload
        '""'     => sub { ${ $_[0] } },
        '0+'     => sub { ${ $_[0] } },
        'bool'   => sub { 1 },
        fallback => 1;
    sub new {
        my ($class, $str) = @_;
        return defined($str) ? bless(\$str, $class) : undef;
    };
};

sub _utf8_readdir(*) { ## no critic (Subroutines::ProhibitSubroutinePrototypes)
    state $zero = utf8::all::TrueStr->new('0');
    state $null = utf8::all::TrueStr->new( '');
    my $pre_handle = shift;
    my $hints  = (caller 0)[10];
    my $handle = ref($pre_handle) ? $pre_handle : qualify_to_ref($pre_handle, caller);
    if (not $hints->{'utf8::all'}) {
        if (wantarray) {
            return CORE::readdir($handle);
        } else {
            $_ = CORE::readdir($handle);
            return $_ || (defined $_ ? ($_ eq '0' ? $zero : $null) : $_);
        };
    } else {
        $UTF8_CHECK |= Encode::LEAVE_SRC if $UTF8_CHECK; # Enforce LEAVE_SRC
        if (wantarray) {
            return map { $_ ? $_UTF8->decode($_, $UTF8_CHECK) : $_ } CORE::readdir($handle);
        } else {
            return ($_ = CORE::readdir($handle))
                ? ($_ = $_UTF8->decode($_, $UTF8_CHECK))
                : defined $_ ? ($_ eq '0' ? $zero : $null) : $_;
        }
    }
}

This variant passes all the existing utf8::all tests, but it is obvious that entry named "0" should be added to "corpus" directory and few new tests are required.

If you get an agreement on the approach (pure Perl or XS), I could try to fix/add tests in case of pure Perl variant.