Extensibility?

[xpe]

I would appreciate if you would share your rationale for CE's design choice(s) regarding extensibility (or lack thereof).

What research have you done with regards to extensibility?

Have you looked at: edn which uses tagged elements as a mechanism for extension?

Serialization Comparisons

Some are quite opinionated :)...

• https://github.com/maximveksler/awesome-serialization
• https://blog.mbedded.ninja/programming/serialization-formats/a-comparison-of-serialization-formats/
• https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats

[kstenerud]

The basic philosophy for CE is that it must provide the minimum required types to serve 80% of use cases without requiring the user to build complicated constructs or special rules for the data to be understood. To me, this means supporting most of the fundamental data types natively.

After the fundamental types, the less commonly used data types can be seamlessly represented using one or more of the primitive types, and can be understood by the receiving end if you have a schema (or partially understood if you don't). This should get coverage to the 90% stage.

The remaining 10% can be custom-encoded, because otherwise we'd be complicating the format for diminishing returns.

EDN is doing pretty close to the same thing, but designates responsibilities more to the encoding format itself rather than to a schema. #myapp/Person {:first "Fred" :last "Mertz"} would be done in CE using a map and a schema, for example. Others such as timestamps and UUIDs have their own dedicated types already in CE, so they'll just work.

Tagging necessarily bloats the encoded data (because you need to preserve the tag data), but I'd call it a fair tradeoff if your goal is to avoid schemas altogether. Other encoding formats (such as protobufs) omit type information entirely from the encoded data in order to save space, which means that the data is completely unreadable without the schema (once again, a fair tradeoff given their goals).

CE takes a middle-of-the-road approach: preserve only enough type information to make it at least minimally understandable to the receiving end without requiring a schema, even if the type doesn't match exactly (for example int vs float). This gives a good encoded size and serialization complexity while remaining fully readable (except when custom data is used, of course).

[xpe]

Thanks for the response. I can tell you've put a lot of thought and effort into Concise Encoding.

First, I'd like to share some feedback about some writing I'm seeing on the CE project. I mention them because they turn me off, because I don't perceive them as accurate.

Like any feedback, I realize these may be perceived as negative or perhaps even harsh. I don't want them to be taken that way. I am offering this feedback because I want to help your project be more attractive. I'm writing this in the sense of "how do you 'sell' CE" -- not in terms of money but in terms of adoption / usage.

The basic philosophy for CE is that it must provide the minimum required types to serve 80% of use cases ...

... the remaining 10% ...

... This should get coverage to the 90% stage.

Whichever percentage you choose, it looks arbitrary and/or overly specific. It also raises questions such as "X% of what, exactly... What use cases are you including?"

So I'd suggest leaving out any percentages.

More generally, I'm suggesting you steer clear of (a) arbitrary claims and (b) any claims that are not truly central to your project's purpose.

Here's why:

1. Maybe X is valid, according to some definition, but you probably don't want to invest effort in justifying that your choice. Doing so will just be a distraction and time-suck.
2. Such claims can make the project seem less thoughtful and credible. For one, I'm tired of the proliferation of unjustified numbers. (You didn't cause this state of affairs, but it is what it is. Better to recognize it and steer clear.)
3. There are plenty of places to have opinions, certainly, but you should pick your battles.

fundamental data types

You linked to the Wikipedia "data types" page with the label "fundamental data types", but I don't know of any consensus on "fundamental data types".

Again, to repeat, having such an 'argument' would be a waste of your time, so it is better to steer clear.

Instead, I'd suggest some of these alternatives:

1. "CE supports these data types... A, B, C, D..." rather than making overly broad, opinionated claims about what data types are "fundamental".
2. "CE supports Y% of the data types offered by any of these well-known serialization frameworks: X, Y, Z"

The point can link to your nicely detailed table where you compare data type support across different serialization formats.

The above recommended alternatives are more credible and detailed. They help motivate developers to use CE.

Tagging necessarily bloats the encoded data

The word "bloat" is pejorative. Using it implies tagging is a bad idea. But that's not true -- it all depends on the design goals (like you said, it can be a fair tradeoff). Saying "increases the file size" gets the point across.

[xpe]

EDN is doing pretty close to the same thing, but designates responsibilities more to the encoding format itself rather than to a schema.

I'm not sure you're getting the "ah ha moment" about why edn's choice here is quite remarkable. (It isn't for the reason you mentioned.)

Have you read over how edn delegates tag interpretation to custom readers? And how these are optional? There are powerful ideas. I don't think I've seen it anywhere else in the data serialization world.

P.S. The format is called edn. Yes, I find it annoying that the project doesn't name itself with all caps. Yes, I'm a stickler. :)

[kstenerud]

I think you may be misinterpreting the purpose of my communication here. It is vague and imprecise, but that's by design.

Most people have neither the time nor the inclination to digest precise data because it requires so much information density in order to be conveyed effectively. I stick to "marketroid speech" for introductions, superficial answers to GH issue questions, and pretty landing pages such as https://concise-encoding.org/ because that's all most people will actually be willing to read before making their decisions. Trust me: you are an exception in your quest for precision (that's not a bad thing - just rare. And this is a numbers game in the end as we all vie for mindshare).

I didn't mean for the word "bloat" to be taken pejoratively. Bloat is a natural consequence of data communications, and different solutions will be accepting of differing levels of bloat for various reasons. edn accepts some bloat to allow for its powerful typing, tag and delegates system. CE accepts some bloat to be mostly readable without requiring a schema. Hell, even Protobufs accepts a tiny bit of bloat to support optional values. Bloat is only bad if it has no good reason to be there.

But in response to the tagging and custom reader system, yes it is powerful and yes it has many benefits. But that niche is not where CE's focus is. All formats must make compromises, and this is a case where CE compromises such expressive power but edn does not - and that's fine. There is no datacomm silver bullet, so each format and protocol makes compromises on some of bandwidth (bloat), breadth of usability, expressive power, processing efficiency, simplicity of implementation, and ease-of-use based on which aspects are most important to them for the niche they're attempting to fill.

[xpe]

Most people have neither the time nor the inclination to digest precise data because it requires so much information density in order to be conveyed effectively.

This sounds like rationalization to me. In practice, there are short and clear ways to avoid arbitrary made-up percentages.

Trust me: you are an exception in your quest for precision

I disagree. Anyhow, I think this is a matter of opinion and perspective.

Whatever your current motivation may be, I'd encourage you to ask this question: Who makes the decisions to adopt a lesser-known serialization format?

My hypothesis is that a significant portion of your audience is going to be pretty seasoned, skeptical, and looking for "just the facts, ma'am". We're developers. We're actively opposed to marketing bloat.

CE has, what, probably less than 0.001% of the adoption of Protocol buffers or Apache Avro? So cater to the decision makers. (my two cents)