Non-ANSI paths cause rustc to fail to link on windows

[InnocentusLime]

Hello!

After installing the brand new rust 1.24 toolset on my windows machine the rustc started refusing to compile any of the projects that were linking to libraries from crates.io.

Both gnu and msvc toolchains linkers claim that they can't find some files (the msvc for example can't find .o files).

Reinstalling rust toolset and removing all cache doesn't help. The only thing that stops that issue is rolling back to 1.23. And that makes me think that is has something to do with rustc 1.24.

Thank you in advance.

[InnocentusLime]

UPD:
It seems that the cause of that error is rustc's new way to change amount of generated codegen units.

[Mark-Simulacrum]

cc @retep998 @aturon @alexcrichton

It seems odd that this would be the case and our bots didn't catch it. @InnocentusLime Could you provide the exact command and the log that you get?

[kovaxis]

I'm having a similar issue on windows, the MSVC link.exe fails to link with fatal error LNK1181: cannot open input file 'C:\Users\Andrighetti Márquez\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\lib\libstd-8f33fb13bb6c9735.rlib' which might be related to the Unicode characters in the path, but doesn't fail on 1.23 or 1.21. Reinstalling Rust/clearing caches didn't fix it either.
Full error message after running a standard cargo run
By the way, the 'missing file' does indeed exist, copy-pasting the path into the explorer tries to open it.

[ollie27]

I believe #47507 is what has triggered the error @negamartin is seeing. The linker-arguments file that rustc outputs is encoded in UTF-8 with no BOM but according to https://docs.microsoft.com/en-gb/cpp/build/reference/unicode-support-in-the-compiler-and-linker#linker-response-files-and-def-files :

Response files and DEF files can be either UTF-16 with a BOM, or ANSI.

So link.exe is interpreting the file as ANSI and so any none ASCII paths are going to fail.

[Mark-Simulacrum]

I'd suspect that's the root cause here for the original reporter as well. We probably want to encode into UTF-16... but I'm not entirely sure how easy that would be to do.

[retep998]

We already have OsStrExt for encoding into UTF-16 and we do it all the time when calling system functions. The hardest part would be writing [u16] data to a File given that Write only accepts [u8].

EDIT: Actually... the hardest part might be the fact that technically rustc can attempt to invoke link.exe from a non-Windows platform where the windows specific OsStrExt isn't available, in which case there's nothing sane you can do with the OsString arguments if they contain non-unicode bytes.

[InnocentusLime]

@Mark-Simulacrum

The compilation was invoked using
cargo build

If you still need it, here's a big piece (not the full log) of log that was printed on the screen:
https://pastebin.com/XRC50xTe

tl;dr
Here's the end of it:
= note: Non-UTF-8 output: LINK : fatal error LNK1104: \xed\xe5 \xf3\xe4\xe0\xe5\xf2\xf1\xff \xee\xf2\xea\xf0\xfb\xf2\xfc \xf4\xe0\xe9\xeb \"C:\\Users\\\xd0\x98\xd0\xbd\xd0\xbd\xd0\xbe\xd0\xba\xd0\xb5\xd0\xbd\xd1\x82\xd0\xb8\xd0\xb9\\Desktop\\projects\\project-neko-penguin\\target\\debug\\deps\\host_module-3f8d4e6f409b7c53.exe\"\r\n error: aborting due to previous error

[retep998]

@InnocentusLime If you set your console to be UTF-8 using chcp 65001 it would probably be easier to read the error messages from link.exe.

[InnocentusLime]

@retep998 Unfortunately, after I tried the chcp 65001 the compiler kpet giving me the unicode chars as \xxx.

[alexcrichton]

Thanks for the report @InnocentusLime as well as the extra information @negamartin! Additionally thanks @ollie27 for pinpointing this, that sounds like exactly the issue!

I believe this will be a relatively easy bug to fix in rustc itself. The file that we pass to link.exe is created here and it sounds like if sess.target.target.options.is_like_msvc is true then we should automatically encode the file as UTF-16 (with a BOM) rather than as UTF-8. While I don't think there's an existing function for doing that I think we can add one to the backend around there which does this.