Support mempool.space API changes

[notmandatory]

The mempool.space APIs are diverging from the original ones from blockstream. We need to figure out a way to support both.

API Docs

mempool.space: https://mempool.space/docs/api/rest
blockstream: https://github.com/Blockstream/esplora/blob/master/API.md

[notmandatory]

This issue came up today because @MaxFangX had a production issue because mempool.space removed the /fee-estimates API. Mempool may re-enable the API to give end-users time to update their code, but still TBD on that.

See: https://discord.com/channels/753336465005608961/1359516227235741969/1510389629180969091

[orangesurf]

/fee-estimates will continue to be supported but we recommend using the precise fees API endpoint https://mempool.space/api/v1/fees/precise

[luisschwab]

It would be great if you and Blockstream would come up with a versioning scheme and endpoint like the Electrum Protocol does it.

[oleonardolima]

I feel that having the new trait I discussed with @luisschwab (#200) would benefit this, we can have strict ones, like: EsploraMethods and MempoolSpaceMethods.

[luisschwab]

I feel that having the new trait I discussed with @luisschwab (#200) would benefit this, we can have strict ones, like: EsploraMethods and MempoolSpaceMethods.

Indeed

[tnull]

It would be great if you and Blockstream would come up with a versioning scheme and endpoint like the Electrum Protocol does it.

Fully agree with this sentiment. Given that Blockstream's version is likely not seeing many changes, it could be consider v0 and any mempool-specific changes could be easily added in a backwards compatible manner as part of a new v1 that is simply communicated via an HTTP header field?

[luisschwab]

@orangesurf you're still breaking the API:

~$curl -s https://mempool.space/api/fee-estimates | jq
{
  "1": 1.975,
  "2": 1.975,
  "3": 1.28,
  "4": 1.28,
  "5": 1.28,
  "6": 0.743,
  "7": 0.743,
  "8": 0.743,
  "9": 0.743,
  "10": 0.743,
  "11": 0.743,
  "12": 0.743,
  "13": 0.743,
  "14": 0.743,
  "15": 0.743,
  "16": 0.743,
  "17": 0.743,
  "18": 0.743,
  "19": 0.743,
  "20": 0.743,
  "21": 0.743,
  "22": 0.743,
  "23": 0.743,
  "24": 0.743,
  "25": 0.743,
  "144": 0.2,
  "504": 0.2,
  "1008": 0.1,
  "warning": "This endpoint is deprecated and will be removed in a future release. Please use /api/v1/fees/precise instead."
}

Instead of adding a new field, just use HTTP codes to redirect to your endpoint.

[wiz]

Hey, just saw this thread, yeah, mempool.space is essentially not blockstream compatible, as we hard-forked a long time ago. We re-added the temporary deprecated API at the request of a paying enterprise customer but there should be a separate client crate for mempool enterprise instead of trying to use a blockstream/electrs client with mempool.space

[luisschwab]

@wiz my understanding is that everything under /api/v1/ is Mempool-specific, and everything else is legacy Esplora, right?

We could easily have a mempool feature here for Mempool /api/v1/ methods, but we can't really support diverging responses from the same endpoint.

[junderw]

I tested this PR with the latest rust-esplora-client and it works.

mempool/mempool#6546

---

And now, this is how it will be used:

https://github.com/ark-bitcoin/bark/blob/38f79ffefa057d6323662a22dff5d074a1ac7447/bark/src/chain.rs#L322-L338

1. Ask BDK for fee estimates.
2. Assume that 1, 3, 6 will definitely be in the response (even though esplora can return any combination of the base targets removing any target that responded with error or -1.)
3. Crash when it doesn't. (This app was actually handling errors, which is nice assuming they handle it properly all the way up the stack, but most just do get(&3).unwrap())
4. Then convert to human readable fee rate levels like "fast" "regular" "slow" (ie. the mempool API)

---

So just for emphasis... if they point their app at mempool, we will be converting "fastestFee" to "1" and they convert "1" back to "fast".

[notmandatory]

My summer of bitcoin student @EliteCoder18 is going to create a PoC rust-mempool-client project based on a fork of rust-esplora-client. I've asked him to focus on supporting the blockchain APIs (all the ones listed above Lightning) so we can see what the diff looks like. Even though there is probably a lot of overlap now I suspect it will be cleaner to fork it then try to support both in one crate. But still TBD if that will be the right approach.

[luisschwab]

Will rust-mempool-client also support legacy /api/* endpoints? If so, I think feature gating it is the way to go.

I talked with @oleonardolima about splitting this crate into esplora-types and esplora-client. We could have an additional mempool-types which the client consumes.

[oleonardolima]

Will rust-mempool-client also support legacy /api/* endpoints? If so, I think feature gating it is the way to go.

I talked with @oleonardolima about splitting this crate into esplora-types and esplora-client. We could have an additional mempool-types which the client consumes.

Agree, @notmandatory maybe it's a good idea to guide him at this first.