Equivalent of mmap

This forum is for general developer support questions.
Post Reply
User avatar
jaokim
Beta Tester
Beta Tester
Posts: 90
Joined: Sat Jun 18, 2011 12:41 am

Equivalent of mmap

Post by jaokim »

I need to implement an Amiga version of mmap, i.e. read in a potentially huge file to memory, and use it as a chunk of memory. To quote the docs: "A direct byte buffer whose content is a memory-mapped region of a file." This is for Java, i.e. Jamiga, and the java.nio.MappedByteBuffer class.

I think have a few options:
  • 1. Just read in the whole file in memory and pretend like nothing. This is my quick-fix idea, and probably what I'll start doing.
    2. Do something smart on my own, i.e. open the file, read in a reasonably amount of data in a buffer, and then read in chunks of the file when needed.
    3. Use the exec/MMU/MapMemory... which seems to be sort of the same thing as mmap. However, the autodocs tell me:
    This function is extremely dangerous to use if you don't know what you're doing. And since the virtual addressing architecture is not publically documented, you do *NOT* know what you're doing. If in doubt, don't use this function, and rely on ExecSG to do the job.
    Obviously I'm in dobut.
    4. As above: rely on ExecSG to do the job. Could I use the extmem stuff for this?
The primary thing is not to read in super duper huge files, but rather not to have the file entirely in memory (i.e. a 100 MB won't take up 100 MB when we just want to read a few bytes from it).
Therefor I'm thinking that extmem is not what I'm looking for... or? I'm thinking that I'd allocate an extmem chunk same size as the file, but then you'd need to read the whole file into the virtual memory, which could be unneccessary if not the entire is to be used.

Option 2 is relatively straight forward to do, although there's a lot of stuff to take into consideration, and if f.i. MMU/MapMemory works reliably, I'd rather not re-invent the wheel. But: I don't know what I'm doing there, apparently.

Do anyone have any ideas, or perhaps other methods I haven't thought of?
User avatar
broadblues
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 600
Joined: Sat Jun 18, 2011 2:40 am
Location: Portsmouth, UK
Contact:

Re: Equivalent of mmap

Post by broadblues »

I don't think there is anyway you can implement a mmap() other than return failure and setting errno to ENOSYS

Programs that need MMap and variations on the theme need to be rewritten.

It's not just loading a file in memory, but ensuring that file is in sync with any changes to that memory.
User avatar
jaokim
Beta Tester
Beta Tester
Posts: 90
Joined: Sat Jun 18, 2011 12:41 am

Re: Equivalent of mmap

Post by jaokim »

Yes, one of the arguments to the open method is just that: read or read/write. So I'm thinking it'd be possible to at least support read only. And perhaps even do some sort of partial write support.
It is stated in the docs that the functionality can have undefined results if the file is written to by something else, and a few other restrictions. So I think a best-effort implementation is sufficient.

Furthermore, if this is already done in MMU/MapMemory, I'd be thrilled to try to get it to work. But that'd require someone to spill the beans on the inner details of the virtual memory implementation.
chris
Posts: 562
Joined: Sat Jun 18, 2011 11:05 am
Contact:

Re: Equivalent of mmap

Post by chris »

I don't know whether this is helpful: https://github.com/chris-y/libgit2/blob ... miga/map.c

It's rather specific to that library, but emulates the needed functionality. It rather depends what your mmap is trying to achieve.
User avatar
jaokim
Beta Tester
Beta Tester
Posts: 90
Joined: Sat Jun 18, 2011 12:41 am

Re: Equivalent of mmap

Post by jaokim »

@chris:

Yes, its something like this I'm thinking of doing. And with your implementation, my assumptions of the functionality seems correct. Thanks!

I don't want to emulate the exact behavior of mmap per se, but rather find a way to implement java.nio.MappedByteBuffer. The reference implementation I'm looking at uses mmap/munmap, but its not entirely clear in what order the C functions are called by the Java stuff.
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: Equivalent of mmap

Post by tonyw »

Since I'm working with file system caches most of the time, this question strikes a chord with me (an Eb, BroadBlues).

Would it be true to say that what you want is a window into the file so that you can read parts of it (eg classes)? That sounds to me to be well suited to a memory-resident disk cache, rather than loading the whole file into memory.

Salassoo has done a lot of work in this field and may be able to help you in creating a disk cache for a given file. I suggest you have a word with him.
cheers
tony
User avatar
jaokim
Beta Tester
Beta Tester
Posts: 90
Joined: Sat Jun 18, 2011 12:41 am

Re: Equivalent of mmap

Post by jaokim »

How should I put it... It's usage will not be internal in the Java virtual machine, it's not for loading classes or any other one-case scenario. It's a functionality available in the Java API, letting developers create a normal byte buffer, backed by a file, without needing to allocate all the memory. It can be used to read huge files without reading them to memory. It has also been used as a mechanism to share memory between processes, with great performance gains:can be more than 10x lower latency than using a Socket over loopback.. Since the memory-to-file writing is handled by the OS, the shared memory doesn't reside in the Java heap, but rather in the real OS memory. So, basically, two processes will be able to poke at the same memory from Java.

The reason I have for implementing it now (it's been throwing an error until now) is that I found a code snippet that uses the reading functionality which hopefully will be usable for the Java GUI stuff (it's about reading fonts).

For now I think the performance gains are unnecessary, since quite a few other areas in Jamiga has other performance issues... But hopefully I'll come back to this.
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: Equivalent of mmap

Post by tonyw »

It still sounds like an ideal application for Salassoo's work. I do suggest you contact him about it.
cheers
tony
Post Reply