Standard File Format for Storing File Metadata

AmigaOS users can make feature requests in this forum.
Post Reply
User avatar
Daedalus
Posts: 42
Joined: Sat Jun 18, 2011 12:30 am
Location: From Ireland, living in Glasgow
Contact:

Standard File Format for Storing File Metadata

Post by Daedalus »

I haven't been able to find any information on this so I don't think it's available, but might it be an idea to define a standard file format for storing a file's metadata on non-Amiga filesystems so it isn't lost? Reading and writing this file could then be added to the filesystem handlers as required.

For example: If I have a file called MyScript.rexx with the protection bits srwed and the comment "Test Comment" and I copy it to a FAT32 USB drive, then a second file is also created called MyScript.rexx.aosmetadata that looks something like:

--s-rwed
Test Comment
01.07.2015 14:11:33.10

in the same directory. The FAT32 filesystem could handle this transparently, converting it back into proper Amiga metadata during the Examine call or wherever necessary. It would mean that protection bits, comments and timestamps could be preserved easily and transparently, and also be human readable on other systems if needed.

If the fileformat was officially defined, support could be added to things like SMBFS, Fat95 and so on in a compatible way.
User avatar
Daedalus
Posts: 42
Joined: Sat Jun 18, 2011 12:30 am
Location: From Ireland, living in Glasgow
Contact:

Re: Standard File Format for Storing File Metadata

Post by Daedalus »

Thinking about this a bit more, perhaps it could be implemented as a commodity in a similar way to how CrossDOS translates line feeds when reading and writing from/to CrossDOS volumes. That way it could easily be turned on and off by the user for cases where they don't need Amiga attributes saved and so can avoid the performance penalty associated.
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: Standard File Format for Storing File Metadata

Post by tonyw »

I think this is the way that Apple implemented metadata, with their "resource fork" (or metadata file) associated with each normal file. I don't know if they still do this, I haven't played with a Mac for 20 years.

I think you would choose a format that was extensible, eg xml, so that new metadata could be added to the mix at any time without rendering old metadata obsolete. So your example might be encoded as:

<protection> --s-rwed </protection>
<comment> "Test Comment" </comment>
<date> 01.07.2015 </date>
<time> 14:11:33.10 </time>
cheers
tony
User avatar
Daedalus
Posts: 42
Joined: Sat Jun 18, 2011 12:30 am
Location: From Ireland, living in Glasgow
Contact:

Re: Standard File Format for Storing File Metadata

Post by Daedalus »

Yep, XML would be a good idea, once it's a standard format so 3rd parties can implement it too. It would be similar to the resource fork in that it would be handled transparently on relevant filesystems. Macs don't use the resource fork much any more, though it's still supported on Mac filesystems (HFS etc.). OSX instead uses ".DS_Store" files to store the various properties of folders, meaning that things like icon imagery, snapshot information etc. don't go missing on foreign filesystems, and that's very close to what I was thinking of, except for storing the Amiga-specific attributes instead.
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: Standard File Format for Storing File Metadata

Post by xenic »

Daedalus wrote:Thinking about this a bit more, perhaps it could be implemented as a commodity in a similar way to how CrossDOS translates line feeds when reading and writing from/to CrossDOS volumes. That way it could easily be turned on and off by the user for cases where they don't need Amiga attributes saved and so can avoid the performance penalty associated.
I don't think a seperate commodity will work. There are numerous ways to copy files. There is Workbench drag'n drop which can be done by Workbench or by ASyncWB if it is running, There is the C:Copy command and similar 3'rd party commands. There are file managers like Dopus4, Dopus5 and others. I think the only practical way to accomplish the addition of a metadata file would be to the implement it in CrossDOSFileSystem and control it with additions to the existing CrossDOS commodity.

Another approach would be a custom "Copy" program that would add the metadata when files are copied. It would be even easier just to format your USB stick with SFS so that there is no need to create a seperate metadata file. You can also use an archiver like "LHA" to compress the files and preserve all the Amiga file data.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
Daedalus
Posts: 42
Joined: Sat Jun 18, 2011 12:30 am
Location: From Ireland, living in Glasgow
Contact:

Re: Standard File Format for Storing File Metadata

Post by Daedalus »

Yes, a similar implementation to CrossDOS is what I was thinking of since it would need to be implemented at the filesystem level. It could be implemented at a dos.library level too I guess, but that's not a very efficient way of doing things when 95% of the time the dos routines are accessing Amiga filesystems, and every examine, setcomment etc. call would have to be updated to check what filesystem the target was on.

Yeah, I know there are ways to work around it such as using archives or sticking to Amiga filesystems. I was really thinking of my use case where I share data across multiple computers and OSes via removable media, network shares and email. Losing file attributes is something I put up with and is only a minor inconvenience, but it would be nice to have the option of preserving them in such environments.

A custom copy command (or implementation at the application level) sort of defeats the purpose as it's no longer transparent to the user (think of opening the Icon Information window for a file on a network share), but I might knock one up tonight as a proof of concept.

This thought process initially came about while thinking about folder synchronising over the network. There are programs to do this (Foldersync, BackUp), but I might put one together that preserves the file attributes, and then maybe see about integrating that functionality into open-source filesystems such as Fat95 and SMBFS.
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: Standard File Format for Storing File Metadata

Post by tonyw »

I agree that adding the feature to CrossDOS (or smbfs or any interface between AmigaOS and a lesser-featured system) would be the better approach. It's really the only way to keep the data and metadata together.
cheers
tony
Post Reply