Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add viewer watchdog process to notice crashes and report. #3498

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4666872
Add a comment with lldb magic to print callstack for crashing test.
nat-goodspeed Jan 29, 2025
2081147
Build `LLProcess` on `LLCallbackList` instead of on `LLEventPumps`.
nat-goodspeed Jan 29, 2025
3906882
Add LLProcess::tick() method to poll underlying machinery
nat-goodspeed Jan 30, 2025
000fba3
Don't unregister LLProcessListener in its destructor.
nat-goodspeed Jan 30, 2025
4ffd81f
Fix irksome Python syntax warning on yet another branch.
nat-goodspeed Jan 31, 2025
fe79add
Add initial watchdog.py, which depends on a local Python install.
nat-goodspeed Jan 31, 2025
6d09281
Avoid blinkin MS global macros stdin, stderr.
nat-goodspeed Jan 31, 2025
0e04871
Use len(Path.parents) instead of str.count('/') for path depth.
nat-goodspeed Jan 31, 2025
dde6b25
Find watchdog.py even running in a Windows dev area.
nat-goodspeed Jan 31, 2025
b5e9304
Ditch watchdog.py DOS box on Windows.
nat-goodspeed Jan 31, 2025
bac8567
Build watchdog.py to watchdog executable and run that.
nat-goodspeed Feb 1, 2025
50c426e
Introduce ViewerManifest.exe_sfx and use for watchdog executable.
nat-goodspeed Feb 1, 2025
8418fe4
Fix a couple comments in llappviewer.cpp.
nat-goodspeed Feb 1, 2025
5959c3e
Explicitly pass lists of files to sign for each platform.
nat-goodspeed Feb 3, 2025
307004c
Fix copy error bringing post-bugsplat-windows/action.yaml inline.
nat-goodspeed Feb 4, 2025
ba0b771
Fix transcription error inlining post-bugsplat-windows/action.yaml.
nat-goodspeed Feb 4, 2025
bac0d88
Get diagnostic output to solve failure to post Windows symbols
nat-goodspeed Feb 4, 2025
971890a
Count Mac watchdog executable with apps, not dylibs.
nat-goodspeed Feb 4, 2025
7f000ed
Work around bug in GitHub runner machinery.
nat-goodspeed Feb 4, 2025
c71b5c8
Add a "run ID" to LLAppViewer, add "RunID" to static_debug_info.log.
nat-goodspeed Feb 5, 2025
e0cf6b8
Assemble crash report for sending. Check previously unsent reports.
nat-goodspeed Feb 5, 2025
aae7243
Finish first pass at adapting Windows machine ID logic.
nat-goodspeed Feb 5, 2025
f38c941
Fix trailing whitespace.
nat-goodspeed Feb 5, 2025
fcf919d
Try still harder to fix post-windows-symbols.
nat-goodspeed Feb 5, 2025
7a218b5
Restore original post-windows-symbols logic.
nat-goodspeed Feb 5, 2025
983c52b
Don't overquote the powershell -Command argument.
nat-goodspeed Feb 5, 2025
68c0d2e
Always put static_debug_info.log in main logs directory.
nat-goodspeed Feb 5, 2025
c623068
Fix watchdog "Date" and "Duration" calculations.
nat-goodspeed Feb 5, 2025
d70268a
Change watchdog process to a rotating log handler.
nat-goodspeed Feb 5, 2025
3c2d1de
Send Duration as integer seconds.
nat-goodspeed Feb 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ jobs:
path: .master-message-template

- name: Install autobuild and python dependencies
run: pip3 install autobuild llsd
run: pip3 install autobuild llsd PyInstaller

- name: Cache autobuild packages
id: cache-installables
Expand Down Expand Up @@ -308,13 +308,18 @@ jobs:
steps:
- name: Sign and package Windows viewer
if: env.AZURE_KEY_VAULT_URI && env.AZURE_CERT_NAME && env.AZURE_CLIENT_ID && env.AZURE_CLIENT_SECRET && env.AZURE_TENANT_ID
uses: secondlife/viewer-build-util/sign-pkg-windows@v2
uses: secondlife/viewer-build-util/sign-pkg-windows@project/watchdog
with:
vault_uri: "${{ env.AZURE_KEY_VAULT_URI }}"
cert_name: "${{ env.AZURE_CERT_NAME }}"
client_id: "${{ env.AZURE_CLIENT_ID }}"
client_secret: "${{ env.AZURE_CLIENT_SECRET }}"
tenant_id: "${{ env.AZURE_TENANT_ID }}"
files: |
SecondLifeViewer.exe
SLVersionChecker.exe
watchdog.exe
llplugin/dullahan_host.exe

sign-and-package-mac:
env:
Expand Down Expand Up @@ -347,7 +352,7 @@ jobs:

- name: Sign and package Mac viewer
if: env.SIGNING_CERT_MACOS && env.SIGNING_CERT_MACOS_IDENTITY && env.SIGNING_CERT_MACOS_PASSWORD && steps.note-creds.outputs.note_user && steps.note-creds.outputs.note_pass && steps.note-creds.outputs.note_team
uses: secondlife/viewer-build-util/sign-pkg-mac@v2
uses: secondlife/viewer-build-util/sign-pkg-mac@project/watchdog
with:
channel: ${{ needs.build.outputs.viewer_channel }}
imagename: ${{ needs.build.outputs.imagename }}
Expand All @@ -357,6 +362,15 @@ jobs:
note_user: ${{ steps.note-creds.outputs.note_user }}
note_pass: ${{ steps.note-creds.outputs.note_pass }}
note_team: ${{ steps.note-creds.outputs.note_team }}
apps:
Contents/Resources/updater/SLVersionChecker
Contents/Resources/SLPlugin.app/Contents/MacOS/SLPlugin
Contents/Resources/SLVoice
Contents/Resources/watchdog
files:
Contents/Resources/*.dylib
Contents/Resources/llplugin/*.dylib
"Contents/Frameworks/Chromium Embedded Framework.framework/Libraries"/*.dylib

post-windows-symbols:
env:
Expand All @@ -376,11 +390,18 @@ jobs:
uses: actions/download-artifact@v4
with:
name: Windows-symbols
path: _artifacts
- name: Extract viewer pdb
if: env.BUGSPLAT_USER && env.BUGSPLAT_PASS
shell: bash
run: |
tar -xJf "${{ needs.build.outputs.viewer_channel }}.sym.tar.xz" -C _artifacts
cd _artifacts
# 2025-02-04: Bug in GitHub runner machinery? The viewer_channel
# written by the build job as 'Second Life Project Watchdog' is
# being read here as 'Second Life Project watchdog', and
# empirically, that's enough difference to make this tar command
# fail. Trust that there's only one *.sym.tar.xz in this directory.
tar -xJf *.sym.tar.xz
- name: Post Windows symbols
if: env.BUGSPLAT_USER && env.BUGSPLAT_PASS
uses: secondlife-3p/symbol-upload@v10
Expand Down
1 change: 1 addition & 0 deletions indra/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ endif (NOT USE_BUGSPLAT)
add_subdirectory(${LIBS_OPEN_PREFIX}llplugin)
add_subdirectory(${LIBS_OPEN_PREFIX}llui)
add_subdirectory(${LIBS_OPEN_PREFIX}viewer_components)
add_subdirectory(${VIEWER_PREFIX}watchdog)

if( LL_TESTS )
# Legacy C++ tests. Build always, run if LL_TESTS is true.
Expand Down
32 changes: 14 additions & 18 deletions indra/lib/python/indra/util/llmanifest.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import itertools
import operator
import os
from pathlib import Path
import re
import shlex
import shutil
Expand All @@ -57,23 +58,18 @@ def __init__(self, msg):
super(MissingError, self).__init__(self.msg)

def path_ancestors(path):
drive, path = os.path.splitdrive(os.path.normpath(path))
result = []
while len(path) > 0 and path != os.path.sep:
result.append(drive+path)
path, sub = os.path.split(path)
return result
return Path(path).parents

def proper_windows_path(path, current_platform = sys.platform):
""" This function takes an absolute Windows or Cygwin path and
returns a path appropriately formatted for the platform it's
running on (as determined by sys.platform)"""
path = path.strip()
path = str(path).strip()
drive_letter = None
rel = None
match = re.match("/cygdrive/([a-z])/(.*)", path)
if not match:
match = re.match('([a-zA-Z]):\\\(.*)', path)
match = re.match(r'([a-zA-Z]):\\\(.*)', path)
if not match:
return None # not an absolute path
drive_letter = match.group(1)
Expand Down Expand Up @@ -309,7 +305,7 @@ def main(extra=[]):
class LLManifestRegistry(type):
def __init__(cls, name, bases, dct):
super(LLManifestRegistry, cls).__init__(name, bases, dct)
match = re.match("(\w+)Manifest", name)
match = re.match(r"(\w+)Manifest", name)
if match:
cls.manifests[match.group(1).lower()] = cls

Expand Down Expand Up @@ -391,10 +387,10 @@ def prefix(self, src='', build='', dst='', src_dst=None):
if src_dst is not None:
src = src_dst
dst = src_dst
self.src_prefix.append(src)
self.artwork_prefix.append(src)
self.build_prefix.append(build)
self.dst_prefix.append(dst)
self.src_prefix.append(str(src))
self.artwork_prefix.append(str(src))
self.build_prefix.append(str(build))
self.dst_prefix.append(str(dst))

## self.display_stacks()

Expand Down Expand Up @@ -551,7 +547,7 @@ def created_path(self, path):
b) schedule it for cleanup"""
if not os.path.exists(path):
raise ManifestError("Should be something at path " + path)
self.created_paths.append(path)
self.created_paths.append(str(path))

def put_in_file(self, contents, dst, src=None):
# write contents as dst
Expand Down Expand Up @@ -583,7 +579,7 @@ def copy_action(self, src, dst):
if src and (os.path.exists(src) or os.path.islink(src)):
# ensure that destination path exists
self.cmakedirs(os.path.dirname(dst))
self.created_paths.append(dst)
self.created_paths.append(str(dst))
self.ccopymumble(src, dst)
else:
print("Doesn't exist:", src)
Expand Down Expand Up @@ -663,7 +659,7 @@ def process_file(self, src, dst):
method = getattr(self, methodname, None)
if method is not None:
method(src, dst)
self.file_list.append([src, dst])
self.file_list.append([str(src), str(dst)])
return 1
else:
sys.stdout.write(" (excluding %r, %r)" % (src, dst))
Expand Down Expand Up @@ -795,11 +791,11 @@ def contents_of_tar(self, src_tar, dst_dir):

def wildcard_regex(self, src_glob, dst_glob):
src_re = re.escape(src_glob)
src_re = src_re.replace('\*', '([-a-zA-Z0-9._ ]*)')
src_re = src_re.replace(r'\*', '([-a-zA-Z0-9._ ]*)')
dst_temp = dst_glob
i = 1
while dst_temp.count("*") > 0:
dst_temp = dst_temp.replace('*', '\g<' + str(i) + '>', 1)
dst_temp = dst_temp.replace('*', r'\g<' + str(i) + '>', 1)
i = i+1
return re.compile(src_re), dst_temp

Expand Down
2 changes: 1 addition & 1 deletion indra/llcommon/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ if (LL_TESTS)
LL_ADD_INTEGRATION_TEST(llframetimer "" "${test_libs}")
LL_ADD_INTEGRATION_TEST(llheteromap "" "${test_libs}")
LL_ADD_INTEGRATION_TEST(llinstancetracker "" "${test_libs}")
LL_ADD_INTEGRATION_TEST(llleap "" "${test_libs}")
LL_ADD_INTEGRATION_TEST(llleap "" "${test_libs}") ## lldb -b -o run -k bt)
LL_ADD_INTEGRATION_TEST(llmainthreadtask "" "${test_libs}")
LL_ADD_INTEGRATION_TEST(llpounceable "" "${test_libs}")
LL_ADD_INTEGRATION_TEST(llprocess "" "${test_libs}")
Expand Down
26 changes: 15 additions & 11 deletions indra/llcommon/llcallbacklist.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -68,14 +68,7 @@ bool LLCallbackList::containsFunction( callback_t func, void *data)
{
callback_pair_t t(func, data);
callback_list_t::iterator iter = find(func,data);
if (iter != mCallbackList.end())
{
return true;
}
else
{
return false;
}
return (iter != mCallbackList.end());
}


Expand All @@ -84,7 +77,10 @@ bool LLCallbackList::deleteFunction( callback_t func, void *data)
callback_list_t::iterator iter = find(func,data);
if (iter != mCallbackList.end())
{
mCallbackList.erase(iter);
// Don't (yet) erase the list entry because we might be indirectly
// called by a traversal of this same list. Defer removal until a
// subsequent traversal.
iter->first = nullptr;
return true;
}
else
Expand All @@ -111,8 +107,16 @@ void LLCallbackList::callFunctions()
{
for (callback_list_t::iterator iter = mCallbackList.begin(); iter != mCallbackList.end(); )
{
callback_list_t::iterator curiter = iter++;
curiter->first(curiter->second);
if (! iter->first)
{
// entry previously marked for deletion by deleteFunction()
iter = mCallbackList.erase(iter);
}
else
{
auto curiter = iter++;
curiter->first(curiter->second);
}
}
}

Expand Down
1 change: 1 addition & 0 deletions indra/llcommon/llcallbacklist.h
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ class LLCallbackList
bool deleteFunction( callback_t func, void *data = NULL ); // removes the first instance of this function/data pair from the list, false if not found
void callFunctions(); // calls all functions
void deleteAllFunctions();
bool empty() const { return mCallbackList.empty(); }

static void test();

Expand Down
Loading