Quick Filesystem Reference Matrix

[ross-spencer]

Happened to be referencing this table again today. It's a nice matrix showing which file systems have support for XATTRS, it can be sorted, and shows ~30?!

• https://en.wikipedia.org/wiki/Comparison_of_file_systems#Metadata

I think the main ones looked at for AHAlodeck have been:

• ZFS
• EXT
• BTRFS
• NTFS (partially)

Could be something to do down the line to crib this table for the AHAlodeck docs, maybe including dates that also help folks to discover the most recent implementations. (Also compatibilities)

[pjotrek-b]

Thanks for putting this here.
Indeed, I'm quite happy to see that so many filesystems support the idea of "key/value" information in the filesystem.

I'd add "HPFS" and "APFS" - which I add in theory, because all I know about their xattr/resource-fork behavior is from reading and what people told me who use MacOS.

And "F2FS" as possible candidate to support xattrs on Android.

[pjotrek-b]

Found this, as I was looking for "where I wrote down the byte limits of xattrs per filesystem".
Here we go (again):

Related discussion #14, contain tech details about xattrs in certain filesystems:
The Quest of Finding Out: "How to find out the actual byte-limit of xattrs on /that/ filesystem?"
#14

filesystem	xattr y/n	bit-proof	size limits	remark
ZFS	✔️	✔️	up to ca. 64 kB per key/value pair; unlimited keys	main test+production candidate. "just works"
EXT4	✔️	✔️	up to almost 4096 Byte for all key/value metadata	tight, but works for most "vitally necessary or popular" audio,video,image,document metadata, commonly "embedded"
HFS, APFS	✔️ ❓	✔️ ❓	unlimited (according to docs)	not tested personally /yet/
FAT32 (vfat)	🚫	--	--	Rumor has it, that metadata "can somehow be added", but it's not lib-code compatible with POSIX-style xattrs
NTFS	✅	?	?	NTFS calls xattr-like metadata capabilities "Alternate Data Streams"
F2FS	✔️	❓	~4096 Byte	seems similar to ext4, but haven't tested it /yet/
exFAT	🚫	--	--	Although being a "young" filesystem, they didn't consider xattrs "worth it".
BTRFS	✔️	✔️	~64 kB?, but unclear if limit in number of keys?	Tested a little bit, but ran into "out of diskspace (for xattrs) errors with xattrs working on ZFS

[pjotrek-b]

BeFS (Haiku OS) seems to be using the filesystem already quite "AHAlodeck-like"! 😁 🥳
This proofs that what I'm proposing is possible, and intended by serious filesystem developer(s), like Dominic Giampaolo (former Google, now Apple filesystem expert)

Who wrote the document "Practical File System Design with the Be File System" (by D.Giampaolo, 2004), regarding filesystem attributes (in BeFS).

Quote:

Attribute Details

As defined previously, an attribute is a string name and some arbitrary chunk
of data. In the BeOS, attributes also declare the type of the data stored with
the name. The type of the data is either an integral type (string, integer, or
floating-point number) or it is simply raw data of arbitrary size. The type
field is only strictly necessary to support indexing.

Furthermore:

In deciding what data structure to use to store an attribute, our first temp-
tation might be to define a new data structure. But if we resist that tempta-
tion and look closer at what an attribute must store, we find that the descrip-
tion is strikingly similar to that of a file. At the most basic level an attribute
is a named entity that must store an arbitrary amount of data.

This is quite unique! All other filesystems so far have no means of storing the data-type.

I believe it may give a performance benefit, having the ability to declare that natively in a filesystem.
Feature-wise, it should nowadays be quite possible (with a "context-style" value in an additional xattr), to do the same - with slight overhead of text-and-parsing once to feed it to a search indexer (Typesense, etc).

Reading xattrs (too?) is like reading a directory

Interesting to know:

The functions to list the attributes of a file correspond very closely with
the standard POSIX functions to list the contents of a directory. The open
attribute directory operation initiates access to the list of attributes belong-
ing to a file.