Cregit whole pipeline takes too long to execute in the whole linux kernel for the first time.
Cregit is able to be a bit more parallelizable accross cores, but since the walker we use for git is sequential we are not able to maximize core use in a multicore computer, but it does not need to be this way, for us to start a tokenization and other steps we only need the actual commit and the last one for the Walker. So we could break the entire git history of a project by time and have multiple "shards" each one running cregit to make use of computer with more cores.
Cregit whole pipeline takes too long to execute in the whole linux kernel for the first time.
Cregit is able to be a bit more parallelizable accross cores, but since the walker we use for git is sequential we are not able to maximize core use in a multicore computer, but it does not need to be this way, for us to start a tokenization and other steps we only need the actual commit and the last one for the Walker. So we could break the entire git history of a project by time and have multiple "shards" each one running cregit to make use of computer with more cores.