aarch64: enhance main makefile to support building from host for aarch64 #1106
Comments
|
As a step towards enabling native builds, I'd like to get rid of the dependency on locally unpacked RPM packages. To scope out the problem, I updated the Makefile to drop references to the downloaded files; instead using the distro provided library and headers. See the work in progress changes in the linked commit for the actual changes. With this, I was able to build stage 1 ( Can you try using the linked commit in your environment? Does the approach make sense to you? |
|
I was able to test the above changes to compile natively for x86_64 on Ubuntu 18.04 and it seems like things still work. whew Thinking about build use cases, the following are commonly ones that would be good to support -
Thoughts? |
|
If the second use case is "Cross-compilation to arm64 on x86_64", then you are right that these are the 3 use cases we want to support. Now the commit in your close you were referring to above seems to indicate that you are removing support of the 2nd use case, am I wrong? Also, the difficulty with use case 2 is that cross-compiling to arm64 on Ubuntu, Fedora, and possibly other Linux distributions would differ because of different locations of aarch64 headers and libraries on the x64 host and have to be accounted for in the main makefile I think (but I might be wrong). Also, we would need to modify But yes ideally we want to support all 3 use cases. BTW were you able to compile OSv natively on RockPro64 and run it using your modified makefile? |
Indeed - that's what I meant.
The branch is still work-in-progress and needs more updates before it's done. More below.
I am not too familiar with Fedora but I had naively assumed that it would have something similar to Debian multi-arch support - where it's possible to install packages (including libraries and development headers) from multiple architecture (link) on the same filesystem. Once the appropriate compiler for the target architecture is chosen, it can be queried for the library locations. With the multiarch support, there's no difference between cross and native compiling - which is a big win in terms of build system complexity. Following this, I started making the changes to the Makefile in the linked branch. On further investigation, it seems that Fedora does not support cross install of userspace libraries; only the compiler is available. This is somewhat OK for Linux kernel (or bare metal) development but doesn't work when cross compiling applications that need userspace libraries without resorting to tricks like what's being done for OSv. On Fedora like systems, one way forward would be create a unified sysroot with all the required cross dependencies, vs one-per-package as being done right now.
Unfortunately the build is broken on the RockPro64 due to unrelated issue with gcc-10 and the need for __getauxval() when linking against libgcc. Cross-compiling on Ubuntu succeeds but doesn't show any output when run. What would be the simplest test for building / running the kernel? Need to see where things are stuck once I figure out how to look at this through gdb (or something). |
|
I think the easiest way to support option 2 would be to replace all conditionals and let native As far as cross-building On a similar note, I actually took a stab at building OSv on Ubuntu 20.04.1 Raspberry PI 4B by using your updated Makefile and slightly adjusting it and other scripts. Here is a complete diff: Please note these critical lines are different from your version of the makefile: Without Now when I compare output of But I do not understand why we have to explicitly add these 4 paths on Now with my Makefile and slightly adjusted Unfortunately, trying to run it on the same PI4 leads to this crash: After connecting with gdb I see this: And it always crashes the same way. Based on what I see in the host and the info in this link and the value of From what I thought I saw in this issue conversation you reported that were able to boot OSv image on RockPro64 cross-built in Fedora container on some x64 machine, right? PS. BTW I am not sure how well Debian is supported on |
|
Many thanks for trying out the patch and your input.
If you look more closely at the patch, on x86 (but also on arm64 natively), instead of hard-coding the library and include paths, they are being queried from the compiler. Why do we need to add the special casing?
I agree, for simplicity let's focus on enabling / fixing the native build for arm64 now. As outlined below, there's a good chance, we'll get the cross builds for free once we fix the native build issues. On Debian (and also Ubuntu), this is true also for the cross compile option (Option 2) as long as we query using the right compiler ( For Fedora, we maybe able to achieve something similar but we have to include
If the above works out, then support for cross compiling for x86 on arm64 should fall out automatically assuming that the distribution provides packages for the cross-headers / libraries. (I'll split the thread here for ease of discussion) |
|
I think I have found a culprit causing kernel built on Ubuntu to hang like (or similar to) what I described above - the Here is a snippet of the makefile diff that fixes it: I will be opening a proper issue for it but in short, the problem is caused by There are probably at least 2 ways of solving this:
Please try to make similar changes to your Makefile to see if you can build AArch64 OSv on your Ubuntu machine and see if it runs successfully. |
|
Actually after closer examination of the generated assembly: It looks like it is same or similar code (see The bottom line is the same. |
|
Hi @punitagrawal, With this commit addressing |
|
Hi @wkozaczuk, Apologies for the silence in the past few days - I've been distracted by a few other things. I had a quick peek at the patch but won't have a chance to test it and your other suggestion about hiding symbols until the weekend. Instead of pulling in ATM, I am not sure there's a strong usecase for supporting LSE atomics (typically useful on multi-processor workloads in systems with large number of cores). |
|
Based on my tests even if you compile with |
I applied the patch and am able to successfully run |
|
It seems to be failing in trying to initialize virtio over mmio devices. Do
you expect any? Are you trying to run it on QEMU with KVM? Can you execute
run.py with --dry-run and send it to me?
…On Mon, Nov 30, 2020 at 7:34 AM Punit Agrawal ***@***.***> wrote:
... this patch
<https://groups.google.com/g/osv-dev/c/-N-yJtjrQJc/m/YVAZFnaWBgAJ> I sent
to the mailing list to make the early console work without symbol
relocation, you should be able to build bootable AArch64 kernel on Ubuntu
with gcc 10.3. I tested it on my RPI 4.
I applied the patch and am able to successfully run native-example about
half the time. The other half it crashes with -
...
getauxval() stubbed
smp_launch ENTERED, lr=00000000400da784
page_fault ENTERED, lr=000000004020e828
faulting address ffff80000a000000
elr exception ra 00000000402eb080
page fault outside application, addr: 0xffff80000a000000
[registers]
PC: 0x00000000402eb080 <mmio_getl(void volatile*)+0>
X00: 0xffff80000a000000 X01: 0x008000000a000713 X02: 0xffff80000a001000
X03: 0x008000000a000713 X04: 0xffff80000a000000 X05: 0xffff80000a000fff
X06: 0xffff80000a000000 X07: 0x0000000000000000 X08: 0x0000000000000001
X09: 0xffff800040c87008 X10: 0x0000000000000713 X11: 0xfffffffffffff8a3
X12: 0x0080000000000003 X13: 0x0000200000100a50 X14: 0x0000000000000703
X15: 0x0000000000000000 X16: 0x0000000000000000 X17: 0x0000000000000000
X18: 0x0000000000000000 X19: 0xffffa00040c89180 X20: 0xffffa000407d71c0
X21: 0x0000000000000030 X22: 0x0000000000000200 X23: 0x00000000406d4e00
X24: 0xffffa000407d7280 X25: 0xffffa00040955d00 X26: 0x0000000040200eb0
X27: 0x000000000a000000 X28: 0x0000000040723e98 X29: 0x0000200000100ab0
X30: 0x0000000040200bd4 SP: 0x0000200000100ab0 ESR: 0x0000000096000007
PSTATE: 0x0000000080000345
Aborted
[backtrace]
0x00000000401dc6fc <mmu::vm_fault(unsigned long, exception_frame*)+748>
0x000000004020ea28 <page_fault+244>
0x000000004020e824 <???+1075898404>
0x0000000040200d44 <virtio::register_mmio_devices(hw::device_manager*)+180>
0x000000004020be2c <arch_init_drivers()+152>
0x00000000400db00c <do_main_thread(void*)+108>
0x0000000040349d50 <???+1077189968>
0x00000000402e8760 <thread_main_c+32>
0x00000000402e6060 <sched::cpu::reschedule_from_interrupt(bool, std::chrono::duration<long, std::ratio<1l, 1000000000l> >)+812>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1106 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABINEIOY66IQGMDXN7KAG7LSSOGOXANCNFSM4TG7JWVA>
.
|
I am indeed running it with Qemu on KVM. Are you normally running it straight from the boot loader? Here's the build and qemu command line (ignore the taskset, I need that due to big.LITTLE on RockPro64) - |
|
Can you comment out the line I wonder if we have some bug either in parsing DTB or memory-mapping the areas corresponding to where where these devices should be mapped. Can you also add some debugging statements around the logic that parses the TDB (dtb_parse_mmio_virtio_devices()). It is interesting I do not see this problem when running on RPI4. |
Currently, we only support cross-compiling
aarch64version of OSv kernel onx64Fedora builds machine/container. GIven these days there is cheaper yet powerful arm64 hardware available like RPI 4, it would be nice to support building natively from the host foraarch64architecture.The text was updated successfully, but these errors were encountered: