SDK:C/make very slow with OS4.1FE (SOLVED)

Have a question about our Software Developer Kit? Ask them here.
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: SDK:C/make very slow with OS4.1FE

Post by xenic »

tonyw wrote:That may be it, then. I'm only using OS4-native utilities, not abc-shell, for instance.

Python is only a distraction in my environment, although it may be used somehow in xenic's environment. There are differences between the 4.1.6 version and the FE version (using dirdiff).
Python had nothing to do with the problem. That was a mistake on my part; probably a result of testing late into the night. However, there is a big time difference between compiling after bootting from an OS4.1FE SFS2 partition and booting from an OS4.1FE FFS partition. The compile times were the same (fast) with OS4.1u6; regardless of the filesystem used on the boot partition. I can't explain it but I'm now booting from an FFS partion with OS 4.1FE and getting normal compile times. Unless someone has the make command sources and can profile the speed under various conditions (Update6 & FE from FFS & SFS2 partitions) the reason for any time differences will probably never be known.

Thanks anyway for testing and trying to help.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
broadblues
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 600
Joined: Sat Jun 18, 2011 2:40 am
Location: Portsmouth, UK
Contact:

Re: SDK:C/make very slow with OS4.1FE

Post by broadblues »

When you mentioned a lack of timestamps in Snoopy I decided to try adding "timer" in front of each "rm -f *.o" in my little test makefile and got really strange results. In both cases I was booting with an SFS2 partition. Without timer in front of each command the total execution time was 8 seconds. With timer running each command the total time was 0.2 seconds. It doesn't make sense that adding a command to launch and time each individual command would be faster than just running the commands.
This actually provides some evidence that it must be some odd scheduling quirk which is causing make to have to wait for longer for command completion than it should.

The is could be becuase of a bug in make revealed by FE or because of a bug in the scheduler revealed by make, but I'm out of ideas on how to procede at the moment.

The difference must somehow relate to the nested process command lauch model in the later make 53.1

Steven SSolie is the maintainer of make.
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: SDK:C/make very slow with OS4.1FE

Post by xenic »

@broadblues
The is could be becuase of a bug in make revealed by FE or because of a bug in the scheduler revealed by make, but I'm out of ideas on how to procede at the moment.
The other factor seems to be which boot filesystem I created an emergency boot CD and when I boot from the CD, make operates at normal speed.

When I boot from an SFS partition make is slow.
When I boot from an FFS partition make is fast.
When I boot from a CD make is fast.

Probably Solie is the only one who can determine exactly is going on.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
ssolie
Beta Tester
Beta Tester
Posts: 1010
Joined: Mon Dec 20, 2010 8:51 pm
Location: Canada
Contact:

Re: SDK:C/make very slow with OS4.1FE

Post by ssolie »

xenic wrote:Probably Solie is the only one who can determine exactly is going on.
Sorry, haven't had the time to look into this yet.
ExecSG Team Lead
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: SDK:C/make very slow with OS4.1FE

Post by xenic »

ssolie wrote:Sorry, haven't had the time to look into this yet.
Since my original report and trying various fixes like using an old make command or switching to an FFS boot partition, I've discoved a simple way to avoid the "make" slowdown. If I enter "path Workbench:C add head" in a shell (where Workbench is an FFS partition) then "make" is just as fast when booting from an SFS partition with OS4.1FE as it was with OS4.1u6 or an OS4.1FE FFS partition. It makes no sense to me but it works.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: SDK:C/make very slow with OS4.1FE

Post by tonyw »

And without that ruse, where is the "Make" being read from? What does "which make" say?

Then, a second question: What does your path look like? What does "path" say? Is there some directory within the path (and before SDK/C) that is very slow to search?
cheers
tony
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: SDK:C/make very slow with OS4.1FE

Post by xenic »

tonyw wrote:And without that ruse, where is the "Make" being read from? What does "which make" say?

Then, a second question: What does your path look like? What does "path" say? Is there some directory within the path (and before SDK/C) that is very slow to search?
Make is being read from "Work:SDK/C". My FFS alternate boot partition (Workbench:) is exactly the same as my normal SFS boot partition (System:). I just copied my SFS partition to the FFS partition so the paths are the same.

The "ruse" is just a method to work around this issue:
If I boot from my SFS partition (System:) the make command is slow.
If I boot from my FFS partition (Workbench:) the make command is fast.

The reason I discovered my "ruse" is that I tried rearranging the order of the paths and even eliminating all paths except the ones essential to the operation of "make". The only way you will see the problem is to test it for yourself. Here is a test makefile I wrote to demonstrate the problem:
Hidden Text - Click to Show :
#Test makefile

all: dodelete

.PHONY : dodelete
dodelete:
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
@echo testing >ram:test.xyz
@delete ram:test.#?
This forum strips the tabs so you need to copy the above test makefile and add tabs before each line beginning with an " @ " symbol. Boot from an SFS partition & save the above makefile listing to ram:makefile. Change directory to ram: and run make on it. The output lines will appear slowly. Repeat the procedure after booting from an FFS partition and the output lines will appear instantly. You can get the "timer" command from OS4Depot and use it with the "make" command (timer make) to get the elapsed time for both tests.

If you don't have an FFS boot partition & an SFS boot partition you can test with my "ruse" like this:

Create a temporary FFS partition and copy the System:C directory to the FFS partition.
Reboot from an SFS partition and create a makefile in ram: with the above code.
Change directory to ram: in a shell and enter "make".
Enter "path FFSpartition:C add head" (change FFSpartition to the name of your FFS partition) in the shell.
Run "make" again.

The time difference should be obvious. Andy already confirmed the problem so if you and others don't want to take the time to test for yourselves, that's O.K. I already have my "ruse" (as you call it) as a workaround for the problem and can compile quickly again. I should add that the difference in compile speeds between booting from an SFS partition and an FFS partition didn't occur with OS4.1u6.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
salass00
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 530
Joined: Sat Jun 18, 2011 3:12 pm
Location: Finland
Contact:

Re: SDK:C/make very slow with OS4.1FE

Post by salass00 »

Just did some tests on my beta OS setup and it seems that strangely just adding any FFS directory will make the makefile go faster.

Since I didn't have an FFS system partition and couldn't be bothered to create one just for this I used my kickstart partition which is formatted using DOS\3 and just doing "path KS0: ADD HEAD" before "make" makes the makefile finish in ~1.8 instead of ~8.8 seconds.

Seems to me that the new dos.library is the most likely suspect here, either that or SFS (don't remember if it got an update in 4.1 FE).
xenic
Posts: 1185
Joined: Sun Jun 19, 2011 12:06 am

Re: SDK:C/make very slow with OS4.1FE

Post by xenic »

salass00 wrote:Just did some tests on my beta OS setup and it seems that strangely just adding any FFS directory will make the makefile go faster.
I tried a similar test by mounting my X1000 bootloader partition (BDH0:) and doing "path BDH0: ADD HEAD" which produces the same speed increase. However, my speed increase is more dramatic than yours. After a fresh boot of my SFS partition the execution time for the makefile is 10.2088 sec and after changing the path it's 0.1368 sec. That's 75 times faster! (I also discovered a bug in the Mounter utility when I mounted my bootloader partition).
Seems to me that the new dos.library is the most likely suspect here, either that or SFS (don't remember if it got an update in 4.1 FE).
The Kickstart/Smartfilesystem file is the same size, version and compile time as the one in OS4.1u6. I copied the Kiskstart/dos.library.kmod file from my OS4.1u6 backup to my FE boot partition and it made no difference. As I stated in an earlier post, probably only ssolie can find the problem by doing a little debugging with the make sources. I'm just happy I found an easy workaround. Compiling huge sources like Dopus5 takes 3 times as long without the workaround.
AmigaOne X1000 with 2GB memory - OS4.1 FE
User avatar
tonyw
AmigaOS Core Developer
AmigaOS Core Developer
Posts: 1479
Joined: Wed Mar 09, 2011 1:36 pm
Location: Sydney, Australia

Re: SDK:C/make very slow with OS4.1FE

Post by tonyw »

That is bloody dramatic !!

For some weeks I have been wondering why my current project takes (sometimes) 4m 40s to compile, whereas a few months ago it took about 2m 30s.

I have already reverted to an old dos version with no improvement and I was scratching my head whilst saying to myself "Don't waste time on it, do the main job first".
I had even moved my system volume and the SDK volume to an SSD without much improvement.

Previous make (after a "make clean" took 3m 52s.

My original path before I tried this trick:
>path
Current_directory
RAM Disk:
SSD-System:C
SSD-System:S/Shell
SSD-System:S/ARexx
SSD-System:System/Python/Scripts
SSD-System:Utilities
SSD-System:Utilities/Commodities
SSD-System:System
SSD-System:Prefs
APPDIR:
Progs:Utilities/PGP
Progs:Samba/bin
SSD-SDK:C
SSD-SDK:gcc/bin
SSD-SDK:local/C
C:

>which make
SSD-SDK:C/make

>path add Boot: head (Boot: is the 2 MB FFS boot partition with amigaboot.of only)

>path
Current_directory
Boot:
RAM Disk:
SSD-System:C
SSD-System:S/Shell
SSD-System:S/ARexx
SSD-System:System/Python/Scripts
SSD-System:Utilities
SSD-System:Utilities/Commodities
SSD-System:System
SSD-System:Prefs
APPDIR:
Progs:Utilities/PGP
Progs:Samba/bin
SSD-SDK:C
SSD-SDK:gcc/bin
SSD-SDK:local/C
C:

Then "made clean" again and another make - 1m 50s !!
I was able to repeat the tests by Removing the FFS Boot: partition from the path and Making again, then reinserting the Boot: partition at the head of the path; the results were exactly repeatable.

I'll bring this thread to Colin's attention.

Bug #9268 submitted.
cheers
tony
Post Reply