Feat/add dataset build - #37
Conversation
…tion Add script that generate dataset containing commits, author, token and source code mapping, basically all info current at .html files. This dataset will allow easier and better search, querying and aggregation of relevant token-based contribution data.
… docs - Move generate_dataset.py into generate_dataset directory - Add DATASET.md with full column docs, token_type domain, author unification guide - Add cross-reference comments to generate_dataset.py and prettyPrint-author.pl - Parameterize run_pipeline_process.sh (--repo-url, --bfg-jar, --file-filter, --keep-on-failure, etc.) - Add pythonEnv with duckdb to devenv.nix - used at generate_dataset.py run - Add cregit.cast as pipeline successful run proof
| @@ -0,0 +1,1471 @@ | |||
| {"version": 2, "width": 142, "height": 13, "timestamp": 1781727668, "env": {"SHELL": "/usr/bin/bash", "TERM": "xterm-256color"}} | |||
There was a problem hiding this comment.
I believe this is the asciinema recording of the tool running right?
If that's the case, we might want to convert it to a gif/mp4, upload it to a blob file open to the web and only reference it in the README.md. I think that makes more sense than having this file in the source code.
There was a problem hiding this comment.
Yes, it is.
Agreed - doing that
| @@ -0,0 +1,256 @@ | |||
| # CreGit Parquet Dataset | |||
There was a problem hiding this comment.
First of all, great job, I think this is incredible. Having a parquet dataset created by cregit token-level blame is pretty good to conduct analysis.
Said that, I do have a few worries and would like to hear your opinion:
- I don't know if the correct place for the parquet generator is inside the cregit source code, but I also don't know if having this as a separate repository would make sense.
- We are adding one more language in already very diversified source code and now we have duplicate functions which can change and we would have to update them in both languages when we do such. For this point I would like to think what we could do that we first generate all the .db and the .html and then we transform it to parquet without duplicating any function. If we need to add an intermediary step, I'm also okay with it.
If you can give me your opinion on this and how we should handle it, I would love to hear.
| | `blank` | pipeline | Blank/empty token line. | | ||
| | `unknown` | fallback | Token content that did not match any known format. | | ||
|
|
||
| ### Content tokens (`is_structural = 0`) |
There was a problem hiding this comment.
I think we can delete this section, even because cregit might run for other languages.
2 participants
Add script that generate dataset containing commits, author, token and source code mapping, basically all info current at .html files.
This dataset will allow easier and better search, querying and aggregation of relevant token-based contribution data.
Also included: