google cloud structured logging with task context #22

simonhugosson · 2024-09-11T14:49:40Z

use same logger as fitapi which logs as json with tenant name etc attached to each message

file is copied from here https://github.com/Volumental/fit-api/blob/master/backend/shoes/cloud_logging.py with very minor simplification

looks something like this when running locally

2024-09-11 14:44:42,300 |MainThread|                       server.py:83   |   INFO| [] tenant:None| Got a task
2024-09-11 14:44:42,306 |MainThread|                       server.py:97   |   INFO| [] tenant:None| Spawning new pool: _import_catalog
 2024-09-11 14:44:42,315 |ThreadPoolExecutor-0_0|                      catalog.py:60   |   INFO| Import catalog job scheduled: {'task': 'queued', 'task_id': 129821}
{"remote_address": "127.0.0.1", "time": "[11/Sep/2024:14:44:42 +0000]", "method": "PUT", "uri": "/v3/shoes/catalog/import/footjoy/catalog.csv", "status": "202", "bytes_out": "34", "user_agent": "python-requests/2.32.2"}
2024-09-11 14:44:42,319 |MainThread|                       server.py:20   |   INFO| [] tenant:None| Worker Starts
2024-09-11 14:44:42,319 |MainThread|                       server.py:26   |   INFO| [] tenant:None| loading task...
2024-09-11 14:44:42,328 |MainThread|                       server.py:42   |   INFO| [LEEK TASK] tenant:footjoy| running task...
importing catalog
2024-09-11 14:44:42,331 |MainThread|               import_catalog.py:273  |   INFO| [IMPORT CATALOG] tenant:footjoy| Loading catalog from gcloud for tenant footjoy with normalization footjoy
{"remote_address": "127.0.0.1", "time": "[11/Sep/2024:14:44:42 +0000]", "method": "GET", "uri": "/v3/task/129821", "status": "200", "bytes_out": "20", "user_agent": "python-requests/2.32.2"}
/home/simon/git/fit-api/venv/lib/python3.10/site-packages/google/auth/_default.py:76: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. See the following page for troubleshooting: https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds. 
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
2024-09-11 14:44:43,098 |MainThread|                      catalog.py:306  |   INFO| [IMPORT CATALOG] tenant:footjoy| Normalizing catalog
2024-09-11 14:44:43,170 |MainThread|                       server.py:53   |  ERROR| [LEEK TASK] tenant:footjoy| ...task failed
Traceback (most recent call last):
  File "/home/simon/git/django-leek/django_leek/server.py", line 46, in target
    return_value = pickled_task()
  File "/home/simon/git/django-leek/django_leek/api.py", line 32, in __call__
    return self.task_callable(*self.args, **self.kwargs)
  File "/home/simon/git/fit-api/backend/shoes/views/catalog.py", line 74, in _import_catalog
    import_catalog(
  File "/home/simon/git/fit-api/backend/shoes/import_catalog.py", line 278, in import_catalog
    catalog = backend.shoes.data_normalization.catalog.load_catalog(
  File "/home/simon/git/fit-api/backend/shoes/data_normalization/catalog.py", line 307, in load_catalog
    df_catalog = load_normalized_catalog_from_raw(
  File "/home/simon/git/fit-api/backend/shoes/data_normalization/catalog.py", line 232, in load_normalized_catalog_from_raw
    df_catalog = CatalogDF(loader(f))
  File "/home/simon/git/fit-api/backend/shoes/data_normalization/footjoy.py", line 40, in load_catalog_from
    return normalize_catalog(pd.read_csv(catalog, dtype=str, low_memory=False, on_bad_lines="skip"))
  File "/home/simon/git/fit-api/backend/shoes/data_normalization/footjoy.py", line 36, in normalize_catalog
    return default.normalize_catalog(df_catalog)
  File "/home/simon/git/fit-api/backend/shoes/data_normalization/default.py", line 226, in normalize_catalog
    raise NotImplementedError("This function is not implemented yet")
NotImplementedError: This function is not implemented yet
{"remote_address": "127.0.0.1", "time": "[11/Sep/2024:14:44:47 +0000]", "method": "GET", "uri": "/v3/task/129821", "status": "200", "bytes_out": "67", "user_agent": "python-requests/2.32.2"}

and gcp:

emazurova

lgtm! 👍

emazurova · 2024-09-18T13:02:41Z

django_leek/cloud_logging.py

@@ -0,0 +1,161 @@
+import copy


It would be nice to have a shared repo for logging. This file is already in 3 repos

yeah that does hurt a bit

we have this of course, would it make sense to add there?
https://github.com/Volumental/virga-python-client

see discussion https://volumental.slack.com/archives/C039C8G2924/p1679065996263339 about repo contents and name if you missed or forgot

Yeah, I think, we could use this repo. To me, it feels like the logger fits well there

the naming is so confusing 🫠

emazurova · 2024-09-18T13:05:34Z

django_leek/cloud_logging.py

+
+
+@contextmanager
+def append_logger_context_namespace(logger: logging.Logger, namespace: str):


is it used?

no, not here (but in fitapi)
I couldn't really decide between just copying this, or removing as much as possible (like all of the namespace stuff)
what do you think?

Yeah, I could also argue for both. Let's keep it as is with the plan of moving logging to the repo you posted above

https://volumental.monday.com/boards/4012902379/pulses/7461493631

oskli

LGTM - looks nice 👌

oskli · 2024-09-18T13:07:20Z

django_leek/cloud_logging.py

+        super(ImportContextFilter, self).__init__()
+        self.import_logging_context_handler = import_logging_context_handler
+
+    def filter(self, record):


Most of the functions are typed. Would be nice to add it to the few ones missing it now as well.

simonhugosson added 2 commits September 11, 2024 16:47

google cloud structured logging with task context

0dde01b

tests

2be5941

simonhugosson requested review from vidstige and emazurova September 16, 2024 07:08

use task pool as namespace

bc0cdc9

emazurova approved these changes Sep 18, 2024

View reviewed changes

oskli approved these changes Sep 18, 2024

View reviewed changes

type hints

a3197bf

simonhugosson merged commit 5e0ed7b into master Sep 19, 2024
2 checks passed

simonhugosson deleted the tech/structured-logging branch September 19, 2024 08:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

google cloud structured logging with task context #22

google cloud structured logging with task context #22

simonhugosson commented Sep 11, 2024 •

edited

Loading

emazurova left a comment

emazurova Sep 18, 2024

simonhugosson Sep 18, 2024

emazurova Sep 18, 2024

emazurova Sep 18, 2024

simonhugosson Sep 18, 2024

emazurova Sep 19, 2024

simonhugosson Sep 19, 2024

oskli left a comment

oskli Sep 18, 2024



		@contextmanager
		def append_logger_context_namespace(logger: logging.Logger, namespace: str):

google cloud structured logging with task context #22

google cloud structured logging with task context #22

Conversation

simonhugosson commented Sep 11, 2024 • edited Loading

emazurova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oskli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonhugosson commented Sep 11, 2024 •

edited

Loading