postgres integration?

[nicolasburtey]

looking at the README:

- Wallet and channel state may be persisted to an [SQLite][sqlite] database, to file system, or to a custom back-end to be implemented by the user.

is there any example of the code that would integrate with something like a Postgres or Mysql?
if not, to integrate with Postgres, does it require a fork, or can it be done by implementing one or a few trait?

[tnull]

is there any example of the code that would integrate with something like a Postgres or Mysql?

While there currently is no ready-to-go Postgres-based backend available, we may start working on one in the future. Let me know (here or on Discord tnull#9465) what you'd use it for. Depending on your timeline, we could possibly consider prioritizing working it, or of course you're more than welcome to contribute such an implementation via a PR.

if not, to integrate with Postgres, does it require a fork, or can it be done by implementing one or a few trait?

Writing an implementation should be straightforward, in particular if you'd simply use the SQLite schema and logic as a template. If you build in Rust, you can already just plug your own implementation of the KVStore via Builder::build_with_store. This is currently not exposed in bindings though, so let me know if you'd want to use this in one of the bindings language. It's def. doable, just requires minor changes to LDK's KVStore trait to make it work.

[nicolasburtey]

It would be used as a server directly compiled from rust, not as a client in a mobile app through some bindings (which may be the primary use case of LDK-node?)

To operate a server, having a separate db makes it much easier than having to deal with a file db that needs to be provisioned concurrently with the binary

I'm not suggesting to prioritize this feature but just trying to understand what is possible in the status quo for this project.

But that said, I think having a SQL backend for Lightning nodes has long been something node operator has been looking for.

And maybe that's where its not so simple: if it's using a fully featured SQL backend, but organized over KV pairs, the overall data structure may be sub optimal for a larger node, as it may not have the indexes or other optimizations required. But maybe not. I'm not familiar with this codebase so just guessing

[tnull]

It would be used as a server directly compiled from rust, not as a client in a mobile app through some bindings (which may be the primary use case of LDK-node?)

Ah, good to know! While simplifying non-custodial Lightning on mobile was indeed one of the first targets of LDK Node, it's our goal to make LDK Node fit for a wide variety of use cases, including server use. In fact, we're in the process of adding features with the server use case in mind (notably syncing via bitcoind RPC, LSP spec integration, etc.) and do further work in that direction (more tba. soon). But even as of now it should work nicely in server deployments, as recent adoptions (e.g., Alby, Fedimint) have shown.

To operate a server, having a separate db makes it much easier than having to deal with a file db that needs to be provisioned concurrently with the binary

Yes, I agree, remote storage support is def. something we'll work on. FWIW, we will ship alpha-stage support for the VSS protocol with the upcoming release, and, as mentioned above, intend to add further backends such as Postgres for the user to choose in the future.

I'm not suggesting to prioritize this feature but just trying to understand what is possible in the status quo for this project.

Sure, but let us know if/when you'd want/require it. We usually prioritize new features by user demand. So no promises we'd do it right away, but if would be a prerequisite for you, we may see if we can prioritize it.

But that said, I think having a SQL backend for Lightning nodes has long been something node operator has been looking for.

And maybe that's where its not so simple: if it's using a fully featured SQL backend, but organized over KV pairs, the overall data structure may be sub optimal for a larger node, as it may not have the indexes or other optimizations required. But maybe not. I'm not familiar with this codebase so just guessing

Well, in terms of the persistence layer LDK Node shares LDK's architecture. While Lightning is generally I/O bottlenecked (as you constantly need to persist the full channel state before being able to proceed), even large-scale LDK users haven't found the KVStore-based persistence a particular limitation. We also continuously make improvements to avoid particularly hot keys and to optimize the IO operations necessary, and will also support an async persistence layer in future LDK versions. Happy to discuss this in further detail though if you're interested (e.g., feel free to hit us up on Discord).

[nicolasburtey]

(e.g., Alby, Fedimint)

One of the first comment I got when asking the developers of Alby about readiness for larger scale use was "well it's using sqlite, so it's fine of our usage (small node) but not sure for larger node" ;)

I was not familiar that Fedimint is using LDK. I see this file indeed. Do you know if some fedimint deployments is using this gateway?

Well, in terms of the persistence layer LDK Node shares LDK's architecture. While Lightning is generally I/O bottlenecked (as you constantly need to persist the full channel state before being able to proceed), even large-scale LDK users haven't found the KVStore-based persistence a particular limitation. We also continuously make improvements to avoid particularly hot keys and to optimize the IO operations necessary, and will also support an async persistence layer in future LDK versions. Happy to discuss this in further detail though if you're interested (e.g., feel free to hit us up on Discord).

ok good to know. I'll reach out on Discord when I have more bandwidth to dive into it.

[benthecarman]

fixed by #863