Titanfall 2 fails to start (Unimplemented Opcode) #210
Comments
|
Something which might be relevant for fixing this: https://patchwork.kernel.org/project/linux-arm-kernel/patch/1447195301-16757-3-git-send-email-yang.shi@linaro.org/ |
|
Also, note that this was with the Northstar dedicated server + wine64 + d3d11 and gfsdk stubs in the game dir (with d3d11 set to native in winecfg). |
|
Thanks @ptitSeb, we'll try running it again soon (I didn't notice your commit earlier). |
|
OK, another one: |
|
the opcode is a RET FAR. it maybe linked to a 64bits <-> 32bits call / ret, and need some more refactor. I'll probably start working on next dev. cycle (so in 0.1.9) |
|
Current release is 0.2.0, what's the current status on it? |
|
Current status of what? I don't have Titanfall 2 in my collection so I cannot tes myself. I haven't advanced on 64<->32 call as I have no program using that currently. |
|
If you tell me your email or a steam account I can buy Titanfall 2 for you as a steam gift, if you're still interested in working on this. It's currently on sale, so it's a good time to do so. |
|
If you just want to test the server, it requires few of the actual game files, there’s a script somewhere that downloads the correct files. You wouldn’t be able to open the game and join the server for full testing though, caveat there. |
|
My steam account is |
|
~70gb off the top of my head
|
I've sent you a friend request on steam. For some reason I can't send a gift using just a name or an email, at least according to the web interface. This is the project where this was tested: According to the readme it's possible to trim the game to 2GB-4GB if necessary, but the base game is fairly big. |
|
I'll probably need to remove Doom from my ARM dev. machine then, as it's about the same size. Or Dota2, it's probably big too. Both are working fine now, so it's not a problem to switch to something else. |
|
|
Ok, I'll start grabbing the optimized server (my dev. arm machine has 8 cores, and I have a wine 7.5 & wine 7.22, both 64bits, ready to use) |
|
@ptitSeb I sent you a steam gift with Titanfall 2, I hope it helps. |
|
also cc @GeckoEidechse |
|
Thank you @Kuratius |
|
And if you want to connect to the server once it runs, you'll need to forward tcp/8081 and udp/37015 (you can change these, though), and you'll find it in the Northstar server list (see the wiki for client setup instructions for Steam).
Yes, and even if you do plan to get the full client working, it's a bit easier to test the server. Oh, and if you do try the client and get an
Yes; I did a lot of the stuff for getting the server working on Linux (docker image, nswrap, file optimization, d3d stubs) |
|
Ok, So I downloaded the server. |
nswrap will make debugging easier; it helps normalize stuff.
I'd need to see the logs around it; I haven't tried running it on ARM since Feb. P.S. I might take a little longer to respond (if I respond) for the next two hours. |
It was just a quick test. I'll build nswrap and try it properly... |
|
Ugh, haven't seen that one before... I'll look into it later today if you can't get it to work. Might also be a good idea to try running it unemulated so you can compare the output. @BobTheBob9 might also be able to help. |
|
Here with nswrap (just the end) Yeah, I'll check unemulated to see what it does |
|
Mmm, so, the program is crashing in a portion of code that is obfuscated. I guess there is still a bug somewhere in the dynarec to find. I activated |
|
If the offset from Northstar log is correct, it does appear it crashes inside of standard |
|
Turned out my UDP socket doesn't work because Respawn engine uses AF_INET6 family and apparently the ARM device's kernel that I flashed has IPv6 support completely disabled for some reason (💀), and due to how kernel is built for that device I'd have to basically rebuild it to actually undo that, so I won't be able to test any further for now until I find a workaround or this issue is fixed: hexdump0815/imagebuilder#15 ;_; (forcing AF_INET makes the socket open but game ignores packets, probably because the C structures are not really exactly inter-compatible, such as one for address when trying to send response) At least we know it's not a box64 issue though... (which is a good message, because I guess it means it should (without having tested that) genrally work for tfods, and then only Northstar is to be figured out...) |
|
I come back with good news. After sorting out my kernel, Titanfall Online dedicated server actually fully works now! Below some thoughts on performance results from my quick test for anyone who's interested in running these servers. When it comes to performance, it's worth noting that Oracle's Ampere A1 should be quite a bit more powerful than my poor MediaTek MT8173C (2x Cortex-A72 + 2x Cortex-A53, 1.8-2.1 GHz), while Ampere A1 is allegedly consistent 3.0 GHz. I also didn't disable any CPU security vulnerability mitigations. And yet for some reason my idle CPU usage on server was more like 3.5 ms of frame time, instead of the 2.5 ms or even 1.65 ms I'd get before, idk why. So, potentially, consider these results the worst case scenario. So the server as it first booted up, spawned the biggest initial amount of AIs. And then the server usage went above 100% sadly, which meant that the server can't keep up with the amount of frames it has to generate. Yet it really didn't make the game feel unplayable, perhaps the server was able to process multiple ticks per frame without completely falling behind. Then when the AIs all killed each other, the server usage dropped back to 70-80% CPU, which made it quite reasonable again (but then again, with only me as single player jumping around). But there was another problem, overage frames. AI processing likes to cause frametime spikes even on x64 CPUs, but usually the percentage of average frames doesn't ever exceed 0.5%. In case of our emulated ARM it was 5.5%, which is definitely far above ideal (though still better than the initial >100% CPU usage for sure). Afterwards I ran Some quick conclusions then:
So overall these results are honestly still very promising, I don't remember exactly what kind of numbers was @pg9182 getting when trying other emulators, but IMO here this is something we can absolutely work with. In worst case scenario the servers just won't have 100% flawless performance or would run without AI, but far from unplayable. I could also give a snarky remark here that it still might run very decent compared to official Respawn servers kek. Oh also we could potentially still improve performance further later on by tinkering around some BOX64 env options. (Also Northstar still broken, but it's 3 am so not figuring that out today xd) Big big thanks to ptitSeb for all his work on this project and for all the fixes that eventually led to this moment. |
|
@p0358 Are you sure you were reading the cpu usage indicator correctly? 3.5ms frametime is 285 fps, I don't think a game server normally runs that high of a tickrate, it's usually around. 20-120ticks depending on the game. Also what is an overage frame? |
|
First things first. It needs to run fine before we look at possible optimisations. For optimisations: There are the box64 env. var. to play with. Also, you can use Also, you can try to run with |
Actually no, fair point, Source should use WinAPI to measure process CPU time since last check, so that means above 100% could mean other threads caused that, rather than going over the mark. Still, the frame time was approaching the frame budget at that time.
So the servers run by default (in Titanfall 2 and Online) at 60 Hz tickrate, which gives 16.666... ms (1/60) maximum time for a single tick processing (this value is called INTERVAL_PER_TICK). Frames normally run much for much shorter amount of time, and then the main loop would measure how much a frame took to generate, and sleep for the remaining time to INTERVAL_PER_TICK, telling the kernel to yield the processor to other programs, so that the average framerate is equal to tickrate. An overage frame is a frame that exceeded this maximum time, which means the server has to start computing the next frame as soon as it has finished the previous one. What happens then is that it might either try to simulate two ticks in one frame, or rewind the clients' simulation back in time. A small percentage of overage frames shouldn't be a tragedy, but it impacts the ideal network experience for sure. As the percentage rises or the average frame time exceeds the INTERVAL_PER_TICK, trouble arises, as the server can't keep up with real-time processing and simulating the game world, and that will mean it will have to keep rewinding the clients back in time all the time, completely ruining the experience for players as they can visibly keep moving backwards on their screen during their movement. So 3.5 ms frametime doesn't equal to 285 fps, because the dedi is supposed to sleep for the remaining of (16.77.. - 3.5 ms) time. On the note of performance, I've read some random comment on the Internet claiming Ampere A1 has 3x better performance than Raspberry Pi 4B. If my CPU could be said to be comparable to this RPi (technically same Cortex-A72 cores but clocked a bit higher), then it's an excellent message, as it could basically mean our performance concerns are solved as the performance then should already be good enough. I should probably soon look into registering there to actually try it out too... Also for weaker devices in worst case scenario, and to keep using AI, we could turn back the framerate from 60 to 30 like Titanfall 1 did. Not ideal, but perhaps that'd still result in an overall better experience than running with average 5% of overage frames (idk). Also I imagine the PvE gamemode of Frontier Defense could be ran at 30 tickrate there for smooth experience without any moral concerns about it then xd
Right, if these are disabled by default now, then I should probably definitely run that to ensure nothing was missed and ignored (though such occurence has low chances of not resulting in a crash, so hopefully we shouldn't encounter actually any missing opcodes anymore). |
|
So first performance thing in my case is ensuring proper core affinity so that I run on Cortex A72 rather than A53. It seems Then
Honestly hard to tell which options cause what effect as mid-testing I started getting inconsistent results, so I had to cross out my previous findings. All in all with affinity and all the options below randomly enabled, it seemed that the performance on simple attrition on Box with some grunts was better than in my very first test. (I made a far-fetched assumption BOX64_DYNAREC_FASTPAGE would be safe as MinHook calls the WinAPI func to clear processor instruction cache after doing its hooks, and that we don't call the patched/hooked code at all before first time modifying it) However, at least on my hardware, Frontier Defense (most CPU intensive gamemode probably) is certainly no go at 60 tickrate: At 30 tickrate it should generally be fine though, as it only hit something like 45 ms frametime just once. Though who knows how that'd look like if 20 titans dropped in some later waves. In any case, I still have the highest hope of Ampere CPUs performance now to salvate us, rather than my 15 € Chromebook xd (who knows if it didn't start thermal throttling during my testing or something lol) |
|
It seems Northstar server does actually boot up with |
That sounds like either the server has multiple threads running on a single core (not optimal) or it is not actually a dedicated server and is instead running multiple servers. 3.5 ms seems like a magic number to me that might be a leftover from when a single server was supposed to host multiple matches or do x matches per core or something like that. I know valorant servers run that way. The screenshot you posted has more sensible numbers, though I'm unsure how 33ms (30fps) average frametime gets you to 48fps. I assume it probably dropped ticks to get that result, assuming that's possible. I still think it's probably not problematic, tf clients are only supposed to run 20 tickrate anyway so it might not be an issue if the server can't make 60. |
|
With this model, a server running 40/20 for example is probably still fine, assuming that's configurable. 40Hz also has the interesting property that it's exactly in the middle between 60Hz and 30Hz in terms of smoothness (frametime of (1/60 +1/30)/2 is 1/40) , so it should still be better than Tf1. |
The server was lagging and behind its 60 fps target at which it's normally supposed to sit, that's why. Notice how there's 17% of overage frames, which is a pretty big number. It seemed that dropping an enemy titan on FD was nail in the coffin on that CPU and tickrate at least.
No. The server actual frames are supposed to take the least time possible, then sleep until it's time to generate the next frame, according to the tickrate goal. On x64 it takes like < 1 ms for frame generation on an idle server (0.3 ms on idle mp_lobby with no players to be precise). 1/60 so 16.66 ms is just tick interval, how much MAXIMUM amount of time a frame generation can take, until you're running behind schedule and then you're in trouble, and players start warping back or having noticeable delay in shooting etc. (though tbh for most of the gameplay it surprisingly still wasn't THAT bad somehow, definitely much less bad than when the servers would get DoS'd by any of the past methods and actors in the wild xD)
Maybe, Source is apparently somewhat flexible in its tickrate. But currently something prevents you from even touching the updaterate and changing it to anything but 10 without causing severe issues with stutters, probably caused by wrong interp calculations, something that even Nexon didn't fix. So Titanfall Online runs at 60 Hz tickrate, but yet 30/10 cmdrate/updaterate (cmdrate can be changed just fine though). Tickrate is configurable by a convar in Titanfall 2, while for TFO I have made a bunch of patches you can enable by using Also cmdrate changes are rather cheap for the server, since a usercmd is generated every client frame anyways. They're just throttled and queued up to be sent in the next packet, so the server processes the same amount of data (so yes technically you having more client fps slows down the server lol). And updaterate changes are more expensive, because every time at least one client needs an update, a snapshot of world changes must be taken. However it's still very much cheaper than simulating a tick, where physics and all positions and optionally scripts need to be updated. So cmdrate/updaterate imo don't really matter that much in terms of compute performance of a server, they mostly matter for network optimization (not counting one quirk of constant cmdrate in Respawn engine that I'm not going to explain here rn). Tickrate is what matters for CPU time. (on an unrelated note I really wish I could bump up updaterate in r1 up from default 10, but figuring out the inlined and hardcoded interp calculations made me only wish to commit sudoku and didn't bring any progress whatsoever xd) (I'm salty about the networking improvements Respawn made in r2) |
That's still avoiding the question. Without answering why a 3.5ms target is necessary it's still a magic number. I'm going to assume that's a leftover from trying to run multiple matches per core, else it doesn't make sense. |
|
3.5 ms is not a magic number, it was just an example, since you asked about 3.5 ms that was on my screenshot. It's not a magic number or a target, it was simply the average frame time from last second at the moment of taking that screenshot, nothing more. The only target goal is to run with as lowest average tick time as possible, to stay as far away from quota as possible. Which is much more problematic here than normally due to emulation overhead. Valorant blog post is good, but in our situation we can't go that deep in optimalization as easily as they did without access to source code and as such deep insight into inner workings of the engine. Their target was to utilize every core to the fullest possible extent, which saves them real money on scale. However problem with Titanfall servers is that they (at least in r1) like to have frametime spikes, in which case it's easy for one such server to affect all others a lot. Which is why originally Titanfall 1 servers had 2 dedicated vCores, and Titanfall 2, iirc, would have one main core per server plus some helper core for multiple servers at once (they do some operations in a threaded fashion on the thread pool such as bone processing and stuff like that here and there, also network packet sending/receiving). But yeah, the concept of frame's deadline (in their case 1/128) is kinda what applies here for us too. |
|
Thank you for explaining, it seems I misread your previous comment. |
|
This is probably also interesting, it's a comment on the tickrate by a Titanfall server engineer. https://old.reddit.com/r/titanfall/comments/4v36zj/tickrate_is_about_ai_and_not_about_player/ It seems that even running 20Hz tickrate would be acceptable (?). |
|
Good find, I was just thinking of scrolling through his reddit comments to find something related to that.
Uhhhh, that's not exactly true. I hope he oversimplified here, as frame rate is supposed to be limited on a dedi to the same amount as tickrate normally, and a lot of things assume that's the case in engine, but that doesn't mean it's a guarantee. In abnormal circumstances a single frame run can simulate two ticks at once, or none!
Very true. I kinda wish I was aware of these comments before I figured that part on my own kek. This is also why cmdrate is an artificial limitation/leftover from Valve's Source. It actually might cause issues, since every time a packet from client is scheduled to be sent (with multiple usercmds), it has single CLC_ClientTick attached with current tick and current simulation times. I kinda wonder if the latter aren't making the simulation of usercmds inaccurate. They allegedly fixed this in Apex Legends and made every usercmd actually send a packet, and it's a thing I added as experimental option to TFO, seems to improve things a bit. It only made sense to keep cmdrate in case someone's client connection couldn't handle say 60 or 144 frames per second to be sent instead of 30 (60 in r2), otherwise it practically only works against the player (at least compared to players without this limitation, who can get their packets faster to the server without it and thus gain better "ping advantage" kinda).
On that note, I noticed something weird during experimentation around updaterate. I think the function that evaluates whether it's time to send a world snapshot to a player is called several many times single frame. If I'm correct on that, it could mean server wastes some performance by generating multiple separate world snapshots for players, instead of reusing the same snapshot from the same tick for multiple players like original Source did. Would definitely try to optimize it if I didn't have the interp issue ;_;
That's an interesting perspective to look at it. Hmm... He does have a point, but at the same time I'm not sure if lower tickrate and thus simulating world in bigger increments still wouldn't have its side effects, especially in regards to timings on who kills whom at which time etc. AI also isn't really simulated every frame either methinks. If what he said was really true then it wouldn't make sense to have 30 tickrate instead of 10 in the first place, neither to rise it to 60 in Titanfall 2, would it now? At the same time the low updaterate kinda does work against the accuracy of the gameplay. Only every 3rd tick there's a snapshot of all changes generated and sent, and clients need to roll back their time and interpolate the movements between them, which isn't accurate. I wonder what originally led to this decision. Servers supposedly have a lot of bandwidth, clients should have on average must faster download then upload speeds. It kinda seems like this is the worst of both worlds, either lower the tickrate as he claimed here, or rise the updaterate, so that clients get real accurate updates for their local simulation. If tight client bandwidth optimization was their primary goal/paranoia back in 2013 (as it sounded from his talks), I think we could safely lift this updaterate constraint here now 10 years later... I really wish I could honestly talk to that guy and ask him some further questions about all of this 10 years later, especially in regards of his newer experience with Titanfall 2 and Apex, not sure if he'd respond to me nowadays now that he's not at EA anymore either... |
This is just speculation, but if making a gameworld snapshot is sufficiently expensive, and you make a different snapshot for every player to limit wallhacks/esp instead of sending everyone the same snapshot then that could have something to do with it. Even if the current game doesn't do that, it could have been a design consideration. |
IIRC from the Respawn talks I watched many years back, (We're getting quite out of topic from the original issue here btw :P) |
|
Hey there! The |
|
Someone in our Matrix server did a quick test and based on their claims we can at the very least make it all the way to the in-game lobby now. We weren't able to test further yet but this is already quite promising :D |
|
Yeah the server appears to work fine, but there are some ping spikes, which I am not sure if they are caused by my PC or the server. Loading into a game also works fine (tested with two players). Used Proton 8.0 for testing. |
|
Tested on a Raspberry Pi 4 (4GB Ram) using the full 64bit Raspberry Pi OS. The only real issues appear to be caused by the unfit hardware of the Pi, but it worked regardless.
|
|
I don't think Northstar has actually any frame time performance metrics shown anywhere unlike TFORevive, it'd be pretty useful for this to see how it fares |
|
These people seem to have tested box64 for the full Titanfall 2 client on a switch.
And the same on a smartphone I do wonder what's causing the audio issues. |
Maybe high frame times? |
I'd like to think that audio playback isnt directly related to frametime because it could work on a separate thread than the render thread, but I guess technically it's possible that some part of it is cpu related and it just doesn't have any headroom on any core whatsover. Or there's some audio driver library that's being emulated when a native library would do better. |
|
@r3muxd If this issue is not fixed, please reopen it. However, since you have not answered since your first post and other people claim the issue is fixed, I'm closing this for now. |
6443|0x180008561: Unimplemented Opcode (C6) F0 0F C0 01 0F B6 D8 48 8B C6 8B CB 48 D3 E0
on boot
in tier0.dll
tier0.zip
The text was updated successfully, but these errors were encountered: