[Parquet] ALP encoder/decoder support

[sdf-jkl]

Which issue does this PR close?

• Part of Implement ALP encoding #10541
• Closes [Parquet] Prototype ALP encoding #8748 .
• Related spec PR: GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification parquet-format#557

Rationale for this change

check issue

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[alamb]

amaaazing

[sdf-jkl]

@alamb I worked with codex on replicating the c++ implementation reviewing commit by commit.

I'll do one final read myself, but this should be ready for initial review.

[alamb]

Thank you -- I will put this on my short list to reivew I was out last week

[devanbenz]

A few comments

[sdf-jkl]

@devanbenz Sorry for taking so long to address your comments. I'll make sure to work on it this week!

[alamb]

The final spec is

• GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification parquet-format#557

[devanbenz]

Comment about header regarding publication of spec.

[sdf-jkl]

Thanks @devanbenz, I've finally addressed your reviews!

[devanbenz]

Thanks @devanbenz, I finally addressed your reviews!

Please cargo fmt the code 🫡

[sdf-jkl]

I swear I did 😿
Had to run

cargo fmt -p parquet --  --config skip_children=true `find ./parquet -name "*.rs" \! -name format.rs`

5e5c8d2

[sdf-jkl]

Well, clearly having better performnce for the blog would be good :) But at the moment I have no idea where we stand compared to ZSTD in the rust implementation so I would probably build the benchmark scripts first and then decide if we needed to optimize more before publishing

The results would be dataset dependent. Not sure we can get a reliable benchmark using random generated ones.

[alamb]

The results would be dataset dependent. Not sure we can get a reliable benchmark using random generated ones.

Yes for sure -- I think we can start with using data from the paper as a starting point (they list several datasets)

[sdf-jkl]

Yes for sure -- I think we can start with using data from the paper as a starting point (they list several datasets)

Some of them are here - https://github.com/cwida/ALP/tree/main/benchmarks

I'll work on the benchmarks PR

[alamb]

Yes for sure -- I think we can start with using data from the paper as a starting point (they list several datasets)

Some of them are here - https://github.com/cwida/ALP/tree/main/benchmarks

I'll work on the benchmarks PR

I think there is two usecases:

1. A benchmark for the blog post (probably we can create this with a custom CLI / repo, though it could be a PR we never merge in this repo too)
2. A benchmark (cargo bench style) for optimizing performance in this repo

[devanbenz]

Just the one comment, spent the morning looking over this PR and playing around with it locally. It is a very large body of work, looks good to me. My comment is non-blocking but maybe something to look in to.

[devanbenz]

Can this ever be called when an encoder has no values set or null value? For example, if you create a new encoder with AlpEncoder::<DoubleType>::new() and call flush_buffer all while later pages have well behaved data. Will it cause issues? Could you add a test for this if it's possible?

[devanbenz]

Maybe something like:

#[test]
fn test_empty_first_page() {
    let mut encoder = AlpEncoder::<DoubleType>::new();
    // First page flushed with no values
    let empty = encoder.flush_buffer().unwrap();
    assert_eq!(empty.len(), ALP_HEADER_SIZE);

    // Later pages carry well-behaved decimal data.
    let values: Vec<f64> = (0..1500).map(|i| (i as f64) * 0.01).collect();
    encoder.put(&values).unwrap();
    let page = encoder.flush_buffer().unwrap();

    assert!(page.len() < values.len() * 8);
}

[sdf-jkl]

I think it's the same issue @alamb mentioned above -

If the first data page of a column chunk has zero non-null values, build_preset(&[]) fixes the scale at 10^0, so every non-integer value in all later pages of that chunk becomes an exception

Since an all-null leading page is a realistic input, this is probably something we should fix (as a follow on PR)

We'll add this as a follow up

[sdf-jkl]

Added the parquet-rewrite wiring here - 5e9ad1c

[alamb]

FWIW I plan to merge this once the upstream parquet-testing PR is merged. I will prod the upstream PR tomorrow at the parquet sync

[sdf-jkl]

added #10785 to update the parquet-testing dep

[sdf-jkl]

Added another one with just the pinned rev change - #10786

[sdf-jkl]

@alamb

[alamb]

@Jefffrey and @etseidl -- do you think it is ok to merge the Rust ALP implementation for inclusion in arrow/parquet 60?

I think we could be one of the first major open source implementations to ship ALP if we did so

(thank you @sdf-jkl for all your work with this)

[alamb]

BTW if you want to see a sneak preview of a blog post about what ALP is

• Add ALP support blog parquet-site#195

[sdf-jkl]

@alamb @devanbenz thanks for your help 🙏

[etseidl]

@Jefffrey and @etseidl -- do you think it is ok to merge the Rust ALP implementation for inclusion in arrow/parquet 60?

Yes, I don't see a problem with that. It's in "preview" in the spec, and it requires opt-in from users.

I think we could be one of the first major open source implementations to ship ALP if we did so

(thank you @sdf-jkl for all your work with this)

Yes thanks to all for getting this done so soon!

[Jefffrey]

sounds good; i was hoping to take a proper look at this since ive just skimmed it, but kept getting sidetracked by other PRs 😅

[alamb]

sounds good; i was hoping to take a proper look at this since ive just skimmed it, but kept getting sidetracked by other PRs 😅

I think we still have some more time until the 60 release -- so we can wait a while to merge it in