get_new_commits crashes

[audrism]

echo tst,https://github.com/ssc-oscar/tst,968cdcf2e6b22fd5f8f95f2c8666f1a976fac0c7,968cdcf2e6b22fd5f8f95f2c8666f1a976fac0c7 | /usr/bin/get_new_commits
path: tst
url: https://github.com/ssc-oscar/tst
new head: 968cdcf2e6b22fd5f8f95f2c8666f1a976fac0c7
old head: 968cdcf2e6b22fd5f8f95f2c8666f1a976fac0c7
no update!
Segmentation fault

[KayGau]

The problem is caused by the incorrectly free a null pointer in line 102. I'll fix the bug and commit a new version

[audrism]

txs.

[audrism]

Btw, current fetch does the full clone of the repo. In other words it does not save on network traffic at all. What would help would be to fetch only objects associated with new commits.

[KayGau]

Hi, Professor! I run a test to test the fetch operation. I added some code to observe how many objects were fetched. First, I create an empty directory, then use the git_remote_fetch to fetch from remote, and I received 21 objects. Then I modify a remote file, the use git_remote_fetch again, I received 3 objects. It looks like that the fetch operation fetches objects associated with new commits

[audrism]

Great, thats how fetch works with a populated git database. But we have an
empty git database with objects are stored elsewhere. Is it possible to modify fetch so that it looks for objects that do not need to be fetched in an external database?

[KayGau]

I'm sorry, Professor, I couldn't get the point. To get new commit objects, we have to fetch from the remote repository. Then what is "looks for objects that do not need to be fetched in an external database"? Could you please describe in more detail? I' sorry......

[audrism]

Lets consider the example you have above:

1. "First, I create an empty directory" (empty git repo I presume?),

2. "then use the git_remote_fetch to fetch from remote, and I received 21 objects."

3. Now lets take all these 21 objects and store hashes in some database, e.g., flat file
   You can use gitListSimp.sh to do that.

4. Create another empty git repo

5. Run fetch on it, but use objects in the external database so that this fetch
   only receives 3 objects not all 24 objects?

[KayGau]

Hi, Professor!

1. In the first step, it indeed create a directory(empty git repo) to store the .git directory. I didn't delete the directory when the program is over.
2. In the third step, I tried the gitListSimp.sh, in the shell, I type in the follwing command: bash gitListSimp.sh /home/kgao/Desktop/test/.git
   and it prints all the commits.
   After run the shell, the directory created in the first step still exists and the .git still stays there. So there is no need to create another empty git repo in step 4, just use the directory created in step1. Run fetch on it, and use objects in the .git in the directory.

[audrism]

The use case is that I have a database of over 11B of git objects, and would like to update it. In the considered example, all of the 21 objects obtained in the first step are
imported into that database, but the repo /home/kgao/Desktop/test/.git is no longer there.
I'd like to retrieve only the new three objects, not all 24 objects.

[KayGau]

Oh, I know the key point. You have collected commits, but have no local repository, so when I fetch, there is no local repo to compare with, I need to compare with the commit in the database. I thought you have local repo……Then I'll need to modify the fetch logic. I'll try!

[audrism]

Thank you. You can assume that any of the existing object hashes and content can be obtained from the database.

[KayGau]

I traced the execution of fetch, and found the git_smart__negotiate_fetch function in the smart_protocol.c file tells the remote what we want and what we have. I modified it to tell the remote that we have the commits that we got last time, but later when executing to the git_smart__download_pack function in the smart_protocol.c file, the fetch operation raises an error! I checked the output, found the remote indeed knew that we have local commits. Then I checked the function, and found that when downloading from the remote, it first writes the odb in the repo to a writepack. The odb is the object database in the repo, which means that to correctly download the new commits, we must have local repo...

[audrism]

Yes, we need to have local repo. The question is if we just need to prepopulate it with the objects we have, or can we replace with an alternative database, e.g., https://github.com/libgit2/libgit2-backends

[KayGau]

OK. By the way, what's the type of the database used? mysql, redis or sqlite?

[audrism]

Its a custom database based on tokyocabinet since none of the above work at that size. You can assume an arbitrary interface to it. The difference from the git native object store is that you need to pass repo url (to get heads associated with a specific repo)