How to add Little Endian support to OS? (for easier porting)

A forum for general AmigaOS 4.x support questions that are not platform-specific
User avatar
ChrisH
Beta Tester
Beta Tester
Posts: 920
Joined: Mon Dec 20, 2010 9:09 pm
Contact:

How to add Little Endian support to OS? (for easier porting)

Post by ChrisH »

Tony suggested that I start a thread here to discuss this issue, so here it is.

On AW.net, people were discussing the difficulty (or otherwise) of fixing the Endian issues in the JavaScript part of WebKit, so that Odyssey can be updated. Made more difficult by the up-stream maintainers not being interested in accepting Big Endian (aka PPC) fixes. But this problem is hardly isolated to that - many programs (of varying sizes) assume x86 endianess, and each time the program is ported/updated, someone has to find & fix all those tricky endian issues.

Wouldn't it be so much easier if AmigaOS directly supported running certain tasks in a special Little Endian mode? PPC can apparently already change it's endianess, so this doesn't seem completely far-fetched. Here's what I posted on AW.net:
A crazy idea that's not a short-term solution, but might require less effort in the long term:

What if the (Amiga) OS was modified to allow certain tasks to run in the PPC's "Little Endian" mode? You'd also need to mark memory allocated by it as being "Little Endian", and automatically do endian-swaps when it accessed memory NOT allocated by it. Probably also other situations where endianess would need to be swapped (e.g. calling OS procedures, if the parameters are stored on the stack).

Now surely that is a lot of work, but it would "automagically" mean that any future ports (not just WebKit) would not have any Endianess issues.

Now perhaps someone can tell me why this crazy idea wouldn't be possible? :-)
Tony rebutted that post, so I then replied with:
@tonyw
It wouldn't be possible because the very first Exec or library call that includes a multi-byte variable in a structure is going to fail. A longword 0x12345678 is going to be read backwards (0x78563412) by a task running in the opposite endian-ness.
I partially already raised that issue & it's solution in my first post. Let me elaborate a bit:

A task running in Little Endian mode requests memory from Exec. It gets given memory from special memory pages marked as "Little Endian". The idea here being that the MMU could be used to detect accesses to Little vs Big Endian memory pages, and perform the necessary endian swap. So for Little Endian tasks, the MMU would have to trap access to all non-Little-Endian memory pages (basically all normal Amiga memory), and do a swap.

What I'm not clear on is (1) whether this is actually possible, and (2) how big of performance hit would it cause (probably a fair bit, but as long as L.E. tasks rarely interact with the OS it should be fine).
Last edited by ChrisH on Sun Mar 06, 2016 11:12 pm, edited 2 times in total.
User avatar
ChrisH
Beta Tester
Beta Tester
Posts: 920
Joined: Mon Dec 20, 2010 9:09 pm
Contact:

Re: How to add Little Endian support to OS? (for easier port

Post by ChrisH »

After a bit more thought, I shall try to rebut my own proposal:

What happens when a "Little Endian" task calls an Amiga library? That code still runs in the same task, but expects Big Endian memory layout, so the OS would still dutifully swap it to Little Endian. Crash!

So it would seem that library calls also need to be handled specially, but since Amiga library calls go through a jump table, maybe that would be possible? e.g. Present different jump tables to Little Endian & Big Endian tasks, with the Little Endian tasks being made to call a little wrapper code that (temporarily) makes the OS treat the task as Big Endian. (Then any Little Endian memory pointers passed to the library call would automatically get swapped by the OS.)
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: How to add Little Endian support to OS? (for easier port

Post by xenic »

@ChrisH
Our current memory allocation system has been over 10 years in the making. I don't think the changes you suggest could be implemented in a short period of time even if such changes are feasible. I want an up-to-date compatable browser as much as anyone but I doubt if altering OS4 will be the solution. Maybe a little-endian emulator is possible. If UAE runs big-endian AmigaOS on a little-endian PC then why couldn't the reverse be done?
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
ChrisH
Beta Tester
Beta Tester
Posts: 920
Joined: Mon Dec 20, 2010 9:09 pm
Contact:

Re: How to add Little Endian support to OS? (for easier port

Post by ChrisH »

xenic wrote:Our current memory allocation system has been over 10 years in the making. I don't think the changes you suggest could be implemented in a short period of time even if such changes are feasible.
In some ways OS4's memory system is ideally designed for what I propose, because it ALREADY works at the level of (virtual) MMU memory pages. And while I suspect that it wouldn't be nearly as difficult as you imagine, THAT'S NOT OUR PROBLEM (and frankly we don't know enough to tell how difficult it really would be). I merely want to see if it *could* be possible. It's then up to Hyperion whether they think it's worth the hassle & cost (most probably not, but it will be 100% definitely not if we never discuss the possibility).
I want an up-to-date compatable browser as much as anyone but I doubt if altering OS4 will be the solution.
Please don't misunderstand. I'm NOT suggesting this is the solution to our immediate Odyssey/WebKit endian problem - that was just the initial motivation. Rather I am trying to EXPLORE whether it is POSSIBLE to permanently solve ALL endian porting problems at the OS level. Because endian porting problems are not going away, and seem likely to get worse with time (more projects not caring about PPC/BigEndian, projects getting bigger & more complex, and potentially less Amiga developers with sufficient spare time).

At the very least, it would allow "quick'n'dirty" ports, without worrying about endian issues. And once that works (albiet maybe too slowly due to endian swapping), only then tackling the endian issues (at which point you know for sure any bugs/crashes are due to endian issues, rather than porting issues). That would still make porting so much easier, and opens all sorts of doors (for example running the same code in Big & Little Endian modes, and seeing where they diverge, for example fixing endian issues one thread/task at a time instead of having to do it all at once).
Maybe a little-endian emulator is possible. If UAE runs big-endian AmigaOS on a little-endian PC then why couldn't the reverse be done?
Of course it would be possible... but it wouldn't allow Little Endian programs to run on Big Endian OS (whether or not it is emulated).

BTW, if my proposal can be made to work, then it makes it a bit easier to consider compiling AmigaOS4 for x86 or ARM, and yet still retain compatibility with PPC & 68k programs. (There would still be a lot of work involved, but it would change from being impossible, to merely being very difficult... assuming x86 or ARM have some way of changing Endian mode like PPC does. I think ARM does, but x86 may not.)

In conclusion: If you do not have anything technically constructive to add to this thread, please do not derail it with talk of whether it is too difficult or helpful or could be solved in some completely different way (you can always start a new thread if you wish to discuss that).
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: How to add Little Endian support to OS? (for easier port

Post by tonyw »

Mods: This thread arose out of another on aw.net. I suggested that we discuss it here to avoid the noise on the other forum. Please don't lock it just yet, we need a serious discussion on the subject.

Chris:

As I see it, your question is whether a program written for an LE machine could be compiled and run (unmodified) on a BE machine. Assuming it still uses the PPC instruction set, then presumably it is compiled C/C++ code that has all the instructions (I-space) in BE, while the data (D-space) is a mixture of BE (that produced by the compiler) and LE (that produced by the architecture of the original code).

For instance, suppose we have a program with some tables or other fixed data that has been written for an LE machine and ported to a BE machine. The data in the tables will be read wrongly and must be reordered before it will work (the classic BE/LE problem). The program will compile and run, but will give wrong results, as we all know.

Now, how can we indicate to the program that it is running on a BE machine and must reorder its data? Do we need to reorder all the data or only some of it? Remember that a lot of the data will have been compiled by the compiler and will be in BE mode, so there will be a mixture. We can't just indicate to the OS (kernel, DOS, libs etc) that this program is in LE, because it then has to know which parts are BE and which are LE.

The OS is going to have to perform an intelligent switch in real time, with signals in the code indicating which bits are BE and which are LE. Worse, the switches are going to occur in places that the compiler cannot always anticipate and signal correctly.

I can suggest several data streams (some even in my own code) that are a mixture of byte, halfword and longword data. I can't see any way to avoid having to go through all the code, looking for and either flagging or editing such data. In short, for any means of automatic treatment that I can think of, I can think of simple exceptions that will killl it.

So I think that, no matter how you implement a BE/LE switch, there will be examples that don't work. As a result, we are still going to have to go through the code, line by line, to edit it.
cheers
tony
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: How to add Little Endian support to OS? (for easier port

Post by xenic »

ChrisH wrote: In conclusion: If you do not have anything technically constructive to add to this thread, please do not derail it with talk of whether it is too difficult or helpful or could be solved in some completely different way (you can always start a new thread if you wish to discuss that).
That would make this topic a dictation and not a discussion. I just didn't want to provide a detailed explanation of why it's impossibe to have the memory system handle the conversion between LE and BE. If the data is being accessed as a 32bit integer all 4 bites would need to be reversed. If the data is being accessed as a 16bit integer then only 2 bytes would be reversed. A PPC processor has seperate assembler instructions that can load data as 32bit, 16bit & 8bit values at specified addresses.

For example:
Given that the capital letters A, B, C & D represent bytes in memory, the bytes ABCD addressed as a 32bit integer would be reversed to DCBA. If those bytes are addressed as 2 16bit integers the bytes would be reversed to BA & DC which would be BADC if viewed as a 32 bit integer. The memory system would have no way of determining how the data will be accessed.

Refer to sites like https://en.wikipedia.org/wiki/Endianness or similar.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
ChrisH
Beta Tester
Beta Tester
Posts: 920
Joined: Mon Dec 20, 2010 9:09 pm
Contact:

Re: How to add Little Endian support to OS? (for easier port

Post by ChrisH »

tonyw wrote:As I see it, your question is whether a program written for an LE machine could be compiled and run (unmodified) on a BE machine. Assuming it still uses the PPC instruction set, then presumably it is compiled C/C++ code that has all the instructions (I-space) in BE, while the data (D-space) is a mixture of BE (that produced by the compiler) and LE (that produced by the architecture of the original code).
The "BE (that produced by the compiler)" is an interesting point. I guess an example would be a table of (precomputed) word or long word data included in the code itself.
Now, how can we indicate to the program that it is running on a BE machine and must reorder its data? Do we need to reorder all the data or only some of it? Remember that a lot of the data will have been compiled by the compiler and will be in BE mode, so there will be a mixture. We can't just indicate to the OS (kernel, DOS, libs etc) that this program is in LE, because it then has to know which parts are BE and which are LE.
The program's code will be stored in it's own memory area. Why can't this be a memory page marked as BigEndian, and thus have the MMU detect attempts to read from it (by a LittleEndian task) & automagically perform the swap?

I suppose the real question is whether the MMU can distinguish between reading code & reading data. If it can, then my proposal will work, otherwise it won't (since the instruction code would be endian swapped, leading to gibberish). Since it's possible to prevent a CPU from executing memory pages used for data (security), I imagine it must be able to distinguish between code & data, so it seems likely to me my proposal can be made to work.

OTOH, if the MMU cannot be made to do this job, the problem might still be solvable. There are two possible solutions:

1. We would need the OS to pre-endian-swap the code segments of a Little Endian task. Then the MMU would dutifly swap the code back into something the CPU can understand. But it'd probably be rather slow, so maybe it might be too slow to be useful?

2. From my limited knowledge of (x86) assembler, I know that code & data of a program executable tends to be separated into different sections. I imagine this might be the case for ELF executables too? Then the OS can ensure that data sections are put in different memory pages to code sections, and mark those pages appropriately.
User avatar
ChrisH
Beta Tester
Beta Tester
Posts: 920
Joined: Mon Dec 20, 2010 9:09 pm
Contact:

Re: How to add Little Endian support to OS? (for easier port

Post by ChrisH »

xenic wrote:That would make this ... not a discussion. I just didn't want to provide a detailed explanation of why it's impossibe to have the memory system handle the conversion between LE and BE.
Sorry, but you didn't provide ANY explanation. That is the equivalent to saying "It's not possible, trust me", which isn't a discussion either :-(
If the data is being accessed as a 32bit integer all 4 bites would need to be reversed. If the data is being accessed as a 16bit integer then only 2 bytes would be reversed.
(snip more description of Endian-swapping basics)
That much is obvious, but you don't explain why it is relevant to the discussion. And unfortunately I'm not capable of reading your mind :(

My best guess is that you were trying to imply that the MMU is incapable of distinguishing between 32-bit & 16-bit memory accesses. If you can confirm this is true, then that would obviously prevent my proposal from working. BTW, I effectively asked about this in my second post, albiet in not such detail:
the MMU would have to trap access to all non-Little-Endian memory pages (basically all normal Amiga memory), and do a swap.

What I'm not clear on is (1) whether this is actually possible
Refer to sites like https://en.wikipedia.org/wiki/Endianness or similar.
Sigh. Since you seem to think I am a COMPLETE idiot, I would like to mention that I already implemented my own Endian-swapping procedures in PortablE. And they work just fine when (de)serialising data from/to a file, whether it be on PPC or x86.
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: How to add Little Endian support to OS? (for easier port

Post by xenic »

ChrisH wrote:Sigh. Since you seem to think I am a COMPLETE idiot
That wasn't my intent and you are probably smarter than I am. It's quite possible that I have a misunderstanding of the way MMU and virtual memory works which would lead me to the wrong conclusion that your suggestion wouldn't work.

I'll just say that your statement:
In conclusion: If you do not have anything technically constructive to add to this thread, please do not derail it with talk of whether it is too difficult or helpful or could be solved in some completely different way (you can always start a new thread if you wish to discuss that).
rules me out of this discussion and I have no further comment.
AmigaOne X1000 with 2GB memory - OS4.1 FE
Belxjander
Posts: 314
Joined: Mon May 14, 2012 10:26 pm
Location: 日本千葉県松戸市 / Matsudo City, Chiba, Japan
Contact:

Re: How to add Little Endian support to OS? (for easier port

Post by Belxjander »

@ChrisH: the limitations of a mixed-endian compilation are seriously painful.

BE compiled with mixed LE data... Possibly but seriously painful and then some,
PE and ELF do provide for. ".text" and ".data" sections as required as produced by the compiler,
WHEN NONE OF THE SYMBOLS ARE IMMEDIATE DATA.
Compilers produce a LOT of "li" values in PPC compilation along with 16b and 32b values inlined directly into the ".text" section.
Without any means of marking these inline values, access to any of them immediately veto's use of the MMU to magic-convert the value at runtime. (This also introduces more mixed-endian headaches ...)

For a "little endian" mode, the entire program using this mode would require interfaces for LE mode calling AND the entire program to be switched to LE mode at compilation.
Basically requiring wrapper interfaces and an OS level "Container" implementation to be built into the OS.

If you are going to make container facilities available, then isolate the entire ported app to within a fixed and limited minimal runtime with possibly emulating the processor? This would require OS function wrappers (Emulation or not).

The question isn't so much how to make this easier, so much as removing endianness as a possible issue ever.

As for the PPC/LE mode, not all of the PPC processors allow the use of the facility without motherboard support.
There are the reversed(LE in BE or BE in LE mode) instructions for reading and writing...but they come with their own twist. As you can't rely on read/write to memory to be the same instructions before and after the mode change.
(In the PowerPC programmers reference guide, It is the little green book in the set)
lwx for example, in BE mode reads and writes LE mode, but in LE mode, does BE.
Only the byte read/write instructions are unchanged, 16bit and 32bit read/write operations are the same.
They are "current mode reversed" read/write operations.

The LE option is documented as needing motherboard support (with nothing mentioned beyond this point) in the little green book I have with me.

The only possible solution I see is to complete a full container mechanism for any arbitrary application, and then proceed with total isolation within the container.

BE can be the "Host" default with container options of LE operations (everything within the container being proxied as a given anyway as an isolation requirement).
That way any application that is LE is run inside an LE environment which is also "Host safe" for the limited BE host access from container proxy functions when forwarding to the host is required, otherwise implemented locally within the container as part of isolation.

The easiest option is basically "petunia"/"WHDLoad" styled launching of a "container" where the container provides a BE"face" to the OS unchanged, while an Application (entirely LE compiled and runtime linked) uses LE wrappers provided by the same container.

That way the content of the container is 100% LE and all endianness issues are the container mechanisms (not the Application or OS)

There is no real alternative without getting into hand-wavium levels of magical thinking that I can see.

The PPC/LE mode has possible hardware requirements that make it potentially system-lethal at switching,
BE and LE are best kept separate or have as little cross-contamination code based on needs only.
Isolation and containment seem to be required essentials for any design/implementation here.

Have I missed a spot or messed up details? Please advise.
Post Reply