"doctests working group" for T-rustdoc?

[QuietMisdreavus]

There's been a lot of activity in trying to improve doctests recently. Most notably, a couple contributors outside the Rustdoc Team have really stepped up to help out: @euclio and @rep-nop. It would be cool to coordinate improvements to doctests so we can plan out our changes and not step on each other's toes.

(I'll try to fill this thread with some ideas that i remember seeing or wanting, but full discussion may have to wait until the All-Hands on February 4-8 so i have dedicated time to think about it. In the meantime, feel free to pitch ideas here!)

[GuillaumeGomez]

👍

[euclio]

I'd like to remove the panic from doc test error handling. Currently, if you have a doc test that fails or panics, you get output with 2 panics in it:

running 1 test
test src/lib.rs - Foo (line 1) ... FAILED

failures:

---- src/lib.rs - Foo (line 1) stdout ----
thread 'src/lib.rs - Foo (line 1)' panicked at 'test executable failed:

thread 'main' panicked at 'explicit panic', src/lib.rs:3:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.

', librustdoc/test.rs:367:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    src/lib.rs - Foo (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

[QuietMisdreavus]

A couple large items that i can think of:

• I'd like to rework how the code conversion happens. Right now, there are still a lot of text-based checks or conversions, and the entire process happens before any macros have a chance to run in the doctest. This means that some situations can still confound rustdoc (attributes spread across multiple lines, macros which create the main function on their own) and cause a doctest to fail when it actually should create a proper executable. I see two ways to fix this, but they won't be easy:
  • Create a new AST in libsyntax that allows expressions in module scope, so that we can parse "standard" doctests to an AST while still allowing doctest with complex macros to have a chance to expand them. This will likely require a contact on the compiler team to tell us how infeasible it is, or how would be the best way around it.
  • Run a kind of "two-phase" macro, to eagerly perform main-wrapping before macro expansion (as a proc-macro or equivalent), then to inspect the AST after macro expansion to see whether the expanded code inserted a main function of its own. This may be a little more cumbersome to reason about, but it's also a lot easier to implement, as it requires no extra work in libsyntax.
• Once rustdoc: Add option to persist doc test executables rust-lang/rust#56189 is merged, we should start considering what is necessary to help people cross-compile doctests and run them in the target environment.
  • @cramertj has requested that we compile doctests into a single executable, like how rustc does for unit tests. This will require some new structural work to mimic the generated main function that libtest creates (or perhaps work to just use that code directly). I think it'll be less work than i originally thought, but it will still require some study and footwork to put it together, especially since the current model is to compile the test as part of its "execution", but this will require a sort of two-phase model, where compilation and execution are handled separately. (Granted, this is currently the model for unit tests, so it's not that much of a stretch, mentally.)
  • A couple of the items i mentioned in the roadmap thread also apply here, namely, the ability to save the source instead of/in addition to the executable, and possibly the ability to run tests with a "runner" script. These will help bridge the gap to being able to run doctests on target systems, but without the "single executable" feature, it may be more trouble than it's worth.

[cramertj]

cc @rtsuk

[cramertj]

This will require a sort of two-phase model, where compilation and execution are handled separately.

Yes, I also need this because I want to be able to cross-compile doctests and run them on a different platform from the host compiler.

[QuietMisdreavus]

Yes, I also need this because I want to be able to cross-compile doctests and run them on a different platform from the host compiler.

Right, and this is quite close to being possible once the --persist-doctests flag is ready, though it will compile each doctest separately. Creating the "single executable" model is another layer on top of that, where we'll have to separate the compilation and execution in the code itself, rather than just skipping execution if we cross-compile the doctest.

[repnop]

Sounds pretty reasonable to me 👍

I'd be more than happy to tackle some of the remaining text-based stuff you mention since I've already done it once!