Perl6's newline translation behavior has been fixed

I am very happy to report that Bug #126881 for perl6: Newline handling is broken on Windows was recently resolved:

Now:

• We do translation of \r\n graphemes to \n on all input read as text except sockets, independent of platform.
• We do translation of all \n graphemes to \r\n on text output to handles except sockets, on Windows only.
• \n is now, unless use newline is in force, always \x0A
• We don’t do any such translation when using .encode/.decode, and of course when reading/writing Bufs to files, providing an escape hatch from translation if needed.

If I understand the changes correctly, this is the logical way to handle to handle newline conventions for text streams:

In text mode, carriage return–linefeed combinations are translated into single linefeeds on input, and linefeed characters are translated to carriage return–linefeed combinations on output.

…

Open in binary (untranslated) mode; translations involving carriage-return and linefeed characters are suppressed.

With this change in place, let’s try the tests that previously failed when the source file used LF for line endings:

D:\Src\rakudo> perl t\harness --moar t\spec\S02-literals\heredocs.t
t\spec\S02-literals\heredocs.t .. ok
All tests successful.
Files=1, Tests=22,  1 wallclock secs ( 0.05 usr +  0.03 sys =  0.08 CPU)
Result: PASS

D:\Src\rakudo> perl t\harness --moar t\spec\S02-literals\quoting.t
t\spec\S02-literals\quoting.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/184 subtests

Test Summary Report
-------------------
t\spec\S02-literals\quoting.t (Wstat: 256 Tests: 184 Failed: 1)
  Failed test:  126
  Non-zero exit status: 1
Files=1, Tests=184,  4 wallclock secs ( 0.11 usr +  0.03 sys =  0.14 CPU)
Result: FAIL

What happens if I do:

chcp 65001?

Everything works:

D:\Src\rakudo> perl t\harness --moar t\spec\S02-literals\quoting.t
t\spec\S02-literals\quoting.t .. ok
All tests successful.
Files=1, Tests=184,  5 wallclock secs ( 0.06 usr +  0.03 sys =  0.09 CPU)
Result: PASS

Since the test involves q:x and UTF-8, that counts as a full pass.

In fact, most of the tests I listed as failing now pass. And, the remaining failures seems to be unrelated to EOL conversions.

For example:

D:\Src\rakudo>perl t\harness t\spec\S32-io\IO-Socket-Async.t
t\spec\S32-io\IO-Socket-Async.t.. Dubious, test returned 255 (wstat 65280, 0xff00)
Failed2/6 subtests

TestSummary Report
-------------------
t\spec\S32-io\IO-Socket-Async.t(Wstat: 65280 Tests: 4 Failed: 0)
Non-zero exit status: 255
Parse errors: Bad plan.  You planned 6 tests but ran 4.
Files=1,Tests=4,  3 wallclock secs ( 0.08 usr +  0.02 sys =  0.09 CPU)
Result:FAIL

D:\Src\rakudo>perl t\harness t\spec\S32-io\IO-Socket-Async.t --verbosity=3
t\spec\S32-io\IO-Socket-Async.t..
1..6
ok1 - Async listen on bogus hostname
ok2 - Async connect to unavailable server breaks promise
ok3 - Async connect to available server keeps promise
ok4 - Echo server
ok5 - Discard server
ok6 - bytes-supply
ok
Alltests successful.
Files=1,Tests=6,  3 wallclock secs ( 0.08 usr +  0.01 sys =  0.09 CPU)
Result:PASS

D:\Src\rakudo> perl t\harness t\spec\S32-io\IO-Socket-Async.t
t\spec\S32-io\IO-Socket-Async.t .. ok
All tests successful.
Files=1, Tests=6,  2 wallclock secs ( 0.08 usr +  0.01 sys =  0.09 CPU)
Result: PASS

Interesting and possibly related to RT#126992.

I am very happy at this point. Any other problems that remain seem to be of a practical, implementation related nature, and they don’t all have to be addressed for an exciting Perl 6 launch.

The behind-the-scenes conversion of \n in string literals to \r\n sequences was a serious philosophical issue of making what has always been a property of streams a mandatory property of literal strings, and needed to be addressed. I appreciate the speed with which the issue was resolved.

I am looking forward to trying out the release version soon after Christmas. Perl6 developers are busy, working hard on putting various finishing touches. Good luck!

And, Merry Christmas!

PS: You can discuss this post on /r/perl.

PPS: For reference, see Perl6’s newline conversion behavior seems to be by design and Newline translation in Perl6 is broken for a brief background on my realization of how newlines were being handled in Perl6 on Windows prior to this change.