Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Liftoff shuts down my USB connected TX16S (ETX2.9.1) within 3 minutes and boots into EM #4262

Open
1 task done
TTGG4400 opened this issue Oct 30, 2023 · 42 comments
Open
1 task done
Labels
bug 🪲 Something isn't working

Comments

@TTGG4400
Copy link

TTGG4400 commented Oct 30, 2023

Is there an existing issue for this problem?

  • I have searched the existing issues

What part of EdgeTX is the focus of this bug?

Transmitter firmware

Current Behavior

I start liftoff on my laptop - I get it ready to fly freeflight.
I start my Tx, and select my liftoff sim model.
I hook up the laptop USB to the Tx top USB C port, and I select joystick mode.
I then use the Tx to take off in Liftoff, fly a minute or so.
Then my Tx display goes dark, and liftoff says I've lost connection.
I toggle power to the Tx and it boots up in emergency mode.
I toggle Tx power again, and it's working normally.
Then I try again - and the same thing happens. 4x in a row now.
Haven't tried liftoff in over a year - it used to work though.
Everything else on my Tx works fine.
Running ETX2.9.1.

Expected Behavior

The Tx should not boot into EM mode while running Liftoff. I don't have problems using the USB C port for normal updates to the Tx, BTW.

Steps To Reproduce

I start liftoff on my laptop - I get it ready to fly freeflight.
I start my Tx, and select my liftoff sim model.
I hook up the laptop USB to the Tx top USB C port, and I select joystick mode.
I then use the Tx to take off in Liftoff, fly a minute or so.
Then my Tx display goes dark, and liftoff says I've lost connection.
I toggle power to the Tx and it boots up in emergency mode.
I toggle Tx power again, and it's working normally.

Version

2.9.1

Transmitter

Radiomaster TX16S / TX16SMK2

Operating System (OS)

Windows

OS Version

Windows 11

Anything else?

No response

@TTGG4400 TTGG4400 added bug 🪲 Something isn't working triage Bug report awaiting review / sorting labels Oct 30, 2023
@pfeerick
Copy link
Member

When

Then my Tx display goes dark, and liftoff says I've lost connection.

is the power LED on the radio still on? If not, what happens if you hold the horizontal trims inwards, and power on the radio - does it go right into the bootloader? As that would suggest that for some reason the radio is losing power (and we need to figure out why that is happen), and the EM prompt when it is being powered back on would be the correct behaviour in that scenario.

@TTGG4400
Copy link
Author

TTGG4400 commented Oct 30, 2023

Hi pfeerick. The power LED on the radio is off. But the 6 position rotary ch LED is still on. However, screen is black and voice enunciations don't work and none of the switches or buttons have any effect (except pressing the 6pos sw causes the LED to change to the switch I just pressed), including the power button - the power button won't work until I remove the USB connection to the laptop. Then when I try your suggestion, of holding the two horizontal trim tabs inwards while powering on - it comes up in normal bootloader mode (no evidence of emergency mode IOW). Otherwise, it behaves as previously noted, if I don't hold the two horizontal trim tabs inward while powering on.

@pfeerick
Copy link
Member

pfeerick commented Oct 30, 2023

Ok, so the really does seem like for some reason it just powered off. The 6POS buttons and leds are controlled by their own MCU, and are probably being powered by the top USB port in this instance, while the main MCU is off. The battery has plenty of charge left (maybe also double-check the voltage calibration for it)?

I wonder @rotorman... do you think the main MCU could crash, and instead of going into EM, could get hung in DFU mode (thereby turning its own power off since it's no longer holding the power on).

@philmoz
Copy link
Collaborator

philmoz commented Oct 30, 2023

I think it could be a crash / restart. I get the same behaviour forcing a crash when the USB joystick is connected.
The radio restarts; but because the USB cable is still plugged in it goes straight to DFU mode.
The main LED is off; but the 6POS switch stays on.

@pfeerick pfeerick removed the triage Bug report awaiting review / sorting label Oct 30, 2023
@rotorman
Copy link
Member

rotorman commented Oct 30, 2023

If the top USB-C of TX16S is connected when STM32F429BIT6 resets, it indeed goes into DFU, as BOOT0 is pulled (via voltage divider circuit) by top USB-C VBUS (the +5V rail) high.

Following might shed some more light into TX16S DFU behavior: https://www.rcgroups.com/forums/showthread.php?3845213-Blog-9-RadioMaster-TX16S-DFU-button-modification

@3djc
Copy link
Collaborator

3djc commented Oct 30, 2023

Yeah, i don't think EM reboot can happen with top USB connected, but the question remains why does it EM ?? Could we have a copy of MODELS and RADIO ? Can you also confirm the USB is well plugged in ? In case of a loose connection firing connect/disconnect in fast sequence, unexpected can be, well, expected

@TTGG4400
Copy link
Author

TTGG4400 commented Oct 30, 2023

Here are the zip files of my MODELS and RADIO folders. And I tried another USB cable, and tried not moving the Tx while hooked up to Liftoff - and it still happens. I was flying model25.yml - the Liftoff model.
RADIO.zip

MODELS.zip

@TTGG4400
Copy link
Author

TTGG4400 commented Oct 31, 2023

I found that Velocidrone does the same thing. And it didn't use to do it - almost 2 years ago - the last time I tried it (Feb 2022).

@3djc
Copy link
Collaborator

3djc commented Oct 31, 2023

It is not bidirectional, so any software will do it windows side, even no software I guess

@TTGG4400
Copy link
Author

So, do you guys think it's my Tx hardware?

@philmoz
Copy link
Collaborator

philmoz commented Oct 31, 2023

So, do you guys think it's my Tx hardware?

Hard to say. There have been a few similar reports; but nothing reproducible at this stage.

I would try a few things to help identify if the problem is in your TX or PC:

  • Try a different PC if you have access to one.
  • Try a different USB cable.
  • Try different USB ports on the PC.
  • Try connecting via a USB hub if you have one.

@TTGG4400
Copy link
Author

Hi Philmoz, I tried a diff USB port. I've tried with and without a USB hub. I tried two diff USB cables besides the original. Nothing changed with the above trials. I don't have another PC to try it on though. However, while using Liftoff, I tried wiggling the USB cable a little (nothing) and a lot - and that (wiggling it a lot) caused a typical USB disconnect, but not an EM. So in this case, I only had to reconnect the cable and select USB Joystick on the Tx, to continue using Liftoff. So, I don't think it's a loose cable causing it to go into EM. Thx for the help, BTW.

@philmoz
Copy link
Collaborator

philmoz commented Nov 1, 2023

Thanks. I noticed in your model file you are using some widgets, logical switches and special functions.

Can you try creating a new model with the minimum setup (no widgets, etc), and see if that still has the problem.
Also turn off the trainer port in the model settings.

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 1, 2023

Philmoz - thx. I removed all widgets, turned off the timers, turned off the trainer port, made sure USB Joystick was set to classic, reverted to Edge TX default theme, made sure no logical switches or special functions were in play, disabled all preflight checks and disabled all the features - AND - I was able to run liftoff for some 25 min or so. Had to go a long time to prove it wouldn't fail. At least I know it's not my hardware now. Thanks again.

@philmoz
Copy link
Collaborator

philmoz commented Nov 1, 2023

Ok, so looks like a problem in EdgeTX then.

It would be very helpful (if you have time) to try adding things back one at a time to see if the problem can be narrowed down.

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 1, 2023

Ok, so looks like a problem in EdgeTX then.

It would be very helpful (if you have time) to try adding things back one at a time to see if the problem can be narrowed down.

Philmoz ... I finished my tests. It appears that if I insert a widget onto the display, it happens. What I did ... First I enabled the trainer functionality. No EM. Then I tried showall widget on the 1x1 display - i got EM. Then I tried gauge rotary on 1x1 - and got EM. Then I put timers 1 and 2 on a 2x1 - and got EM. Then I removed the widgets from the display and turned everything else back on - and that worked just fine. I tried this with two different themes - Edge Tx Default and D. V. A Pink. These made no diff. It seems that having a widget on the screen is the culprit.

@3djc
Copy link
Collaborator

3djc commented Nov 1, 2023

So any widget even in simple does it ? I wonder if this could be related to what you display. Could you create a model with only one screen and a simple value widget displaying one stick ? (Beware that some widgets can have background activities when not showing, so really try on an empty model)

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 1, 2023

3djc, turning off everything, then only putting a value widget on the display with rudder stick as the source didn't EM. However, as I added items back into the mix, such as the showall widget, it still didn't EM. Not until I'd added a few of them, did it EM. All I know at this point is it's more likely to EM with a lot of stuff turned on. But i can't say for sure which items. Because I cannot reliably duplicate a failure mode. For now - I'm tired of siming. Tomorrow, i can give it another shot. Thx for working on this.

@pfeerick
Copy link
Member

pfeerick commented Nov 1, 2023

For the ShowItAll widget... any idea what version you're currently running, and if updating it makes any difference (if you open the main.lua file in the WIDGETS/SHOWALL folder, you should see a 'History' heading with dates and version numbers)? Just to rule out a bug in that widget that has been fixed.

https://rc-soar.com/opentx/lua/showitall/

If it happens with any widget other than Gauge, Model Info, Outputs, Text, Timer or Value (these are all built into the firmware), the issue could somehow be related to Lua widgets.

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 1, 2023

Was running this version of show all ...
2022-10-17 v0.9.10 Min pane height for full info decreased to 169px (for EdgeTX 2.8)
So I loaded the latest, and ran it with everything else also running - and it EM'd in 20s.
Things don't always EM this quickly - the last test, with the old showall and everything running went just over 9 mins, before EM. Normally the full setup lasts less than 3 mins. Makes testing arduous, the inconsistent behavior.
Yeah - I've started timing the trials.

@philmoz
Copy link
Collaborator

philmoz commented Nov 1, 2023

Thanks very much for doing this testing - it really helps.
Could I ask you to test just the showall widget (everything else turned off, logical switches, special and global functions, trainer, etc).

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 1, 2023

I just went 21 mins, with the value widget on the display sourced by throttle stick - and no EM. Everything else was enabled.

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 2, 2023

Here's the log from my last test ...
Wednesday, November 1, 2023 6:57 PM

  1. Started with everything enabled, and D.V.A Pink theme
    
  2. loaded a new Showall and removed main.luac.
    
  3. It sitll EM'd, twice - 1st time 10s, 2nd time 1:20min.
    
  4. switched back to Edge Tx Default theme ...
    
  5. And ... EM after 4 mins
    
  6. Switched to value widget with throttle source
    
  7. And ... no EM after 21 mins
    

@philmoz
Copy link
Collaborator

philmoz commented Nov 2, 2023

I wonder if the mixer scheduler change in #3759 might be a factor.

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 2, 2023

Philmoz - you still want me to run showall by itself. If so - I'll do it tomorrow. Good night.

@philmoz
Copy link
Collaborator

philmoz commented Nov 2, 2023

Philmoz - you still want me to run showall by itself. If so - I'll do it tomorrow. Good night.

Yes, that would be greatly appreciated.

@3djc
Copy link
Collaborator

3djc commented Nov 2, 2023

I noticed your internal module and trainer settings are not off for your liftoff model, can you ensure they are both off for your tests ?

@3djc
Copy link
Collaborator

3djc commented Nov 2, 2023

I wonder if the mixer scheduler change in #3759 might be a factor.

I'm wondering the same, but we run at 1000Hz for ELRS too, so it should not be an issue

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 2, 2023

Latest test run results - basically running the showall widget with absolutely everything disabled or turned off ...
Thursday, November 2, 2023 10:59 AM
turned off internal RF per 3djc request
turned off everything else too ... ADC filter, timers 1, 2 and 3 (was always off); Trainer and I disabled all features
Added showall widget to display
Set desktop clock to 20 min as countdown timer to time liftoff flights.
Started liftoff, plugged in USB to Tx, started flying
And ... EM occurred after 6 min

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 2, 2023

Ran the flying in liftoff with showall widget on main display and all else turned off or disabled test again - this time it EM'd after 2 min.

@3djc
Copy link
Collaborator

3djc commented Nov 2, 2023

Ok, that allows some narrow down, but not sure I like where it seems to be heading

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 2, 2023

Thursday, November 2, 2023 11:47 AM

Ran the gauge rotary widget sourced by rudder with everything else turned off or disabled - note that this widget was blinking "Disconnected..." on the display.
And ... EM after 7 mins.

@gagarinlg
Copy link
Member

Hmm, looks like the joystick code might have an issue. This has been partially redone in a PR that is not yet merged.

@TTGG4400
Copy link
Author

TTGG4400 commented Nov 2, 2023

Thursday, November 2, 2023 12:03 PM
ran the gauge widget sourced by throttle with all else disconnected or turned off - unlike the gauge rotary widget, the gauge widget was updating as the throttle was moved
And ... EM'd after 8:30min

@philmoz
Copy link
Collaborator

philmoz commented Nov 2, 2023

Hmm, looks like the joystick code might have an issue. This has been partially redone in a PR that is not yet merged.

Bu why only when a Lua widget is running and only on some systems?
I've tried this repeatedly on my TX16S with the showall widget and can't get it to crash.
(I've used RealFlight on Windows and Aerofly on Mac - don't have Liftoff)

@gagarinlg
Copy link
Member

This should not dependent on the software that is running on the host. USB joystick communication should be the same.

Maybe we should try with an identical sd card contents?

@pfeerick
Copy link
Member

pfeerick commented Nov 2, 2023

Just as a reminder (also for myself as the penny only just dropped)... the OP said he's running 2.9.1... so #3759 does not apply.

If he is willing, it would be interesting to backup the RADIO and MODELS folders (i.e. I usually just make a copy on the SD card and add the version number... i.e. RADIO29, MODELS29) and try with a nightly build (may need to re-setup POTS/6POS/SLIDERS and do calibration again) to see if the issue is still lurking or has been somehow resolved. Otherwise, need a zip of the entire SD card, just in case it some wonderfully weird and wacky combination of specific versions of theme + widgets + audio file + model image + config, etc is needed ... I do have Liftoff so would be closer to being able to repo this more completely, but as Malte said the software really shouldn't matter...

@pfeerick
Copy link
Member

Can you try the latest nightly build (backup your RADIO and MODELS folder first) and see if that makes any difference?

@Tactical-Badger
Copy link

Tactical-Badger commented Jan 25, 2024

I can confirm this problem is still occurring in v2.9.2

I just upgraded my MK I from a very old version a few days ago, I think the previous version was something like 1.6.x or something like that, Never had a problem when using the controller as a joystick. It now frequently crashes. I tried removing all the widgets, problem still occurs. I tried going into the model configuration screen, and it seemed to take longer to crash(7-8 minutes or so) but did still crash and upon reboot going into EM.

A interesting thing I did notice that may or may not be of interest, I was plugged in to my USB and in joystick mode. I deleted all my widgets in this configuration and the radio crashed. After rebooting from the EM, all my widgets were back like I hadn't deleted them or made any changes.

@pfeerick @gagarinlg

@Tactical-Badger
Copy link

May have jumped the gun slightly on my last post, just saw 2.9.3 has been posted, upgraded the firmware and have been running the radio for the past 30 minutes or so with zero issues. Looks like something was fixed.

Thanks Devs!!!

@pfeerick
Copy link
Member

AFAIK there was nothing specific in 2.9.3 that should effect this, so you could just be lucky or it may rear its ugly head again. There were changes to the USB drivers that are only present in the 2.10 nightlies that will hopefilly actually fix this.

Hence the request earlier to try with a nightly and/or get a copy of the SD card content to see if it was a combination of settings/widget versions etc that was responsible...

@Tactical-Badger
Copy link

Tactical-Badger commented Jan 27, 2024

Yea the issue is still there. 2.9.3 seemed to be working ok, but I'm still having a issue. I just tried the 2.10 nightly 2c01bee and the problem persists at this point. I can tell there was some sort of driver update as I had to recalibrate my controller on all the sims. It seems to be doing it less often, but is still totally random when it happens. We'll get there i'm sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🪲 Something isn't working
Projects
None yet
Development

No branches or pull requests

7 participants