Instrument acreate's for open-ai #935

hmstepanek · 2023-10-11T00:36:45Z

Overview

Instrument Embedding.acreate and ChatCompletion.acreate for open ai.

github-actions · 2023-10-11T00:38:05Z

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Fixed	Elapsed time
✅ PYTHON	bandit	1		6.78s
✅ PYTHON	black	4	0	1.06s
✅ PYTHON	flake8	4		0.63s
✅ PYTHON	isort	4	0	0.26s
✅ PYTHON	pylint	4		4.47s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by

codecov-commenter · 2023-10-11T01:03:34Z

Codecov Report

❗ No coverage uploaded for pull request base (develop-openai-instrumentation@1d86430). Click here to learn what that means.
The diff coverage is n/a.

@@                        Coverage Diff                        @@
##             develop-openai-instrumentation     #935   +/-   ##
=================================================================
  Coverage                                  ?   81.81%           
=================================================================
  Files                                     ?      191           
  Lines                                     ?    19740           
  Branches                                  ?     3426           
=================================================================
  Hits                                      ?    16150           
  Misses                                    ?     2596           
  Partials                                  ?      994

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

umaannamalai · 2023-10-17T16:57:28Z

tests/mlmodel_openai/conftest.py

@@ -37,7 +39,7 @@
    "debug.log_data_collector_payloads": True,
    "debug.record_transaction_failure": True,
    "machine_learning.enabled": True,


I believe I removed this in the sync PR but we probably don't need the machine_learning.enabled check here.

umaannamalai · 2023-10-17T16:58:43Z

tests/mlmodel_openai/test_embeddings.py



 def test_openai_embedding_sync():
    openai.Embedding.create(input="This is an embedding test.", model="text-embedding-ada-002")


-def test_openai_embedding_async(loop):
+enabled_ml_insights_events = {"ml_insights_events.enabled": True}


We probably only need the disabled check since "ml_insights_events.enabled" is True by default in the conftest default settings.

umaannamalai · 2023-10-17T17:01:00Z

tests/mlmodel_openai/test_chat_completion.py

@@ -41,94 +42,101 @@ def set_trace_info():
    {"role": "user", "content": "What is 212 degrees Fahrenheit converted to Celsius?"},
 )

+_test_openai_chat_completion_async_messages = (


This was changed in the sync PR but we realistically don't need all the messages in this list to be tested since we have multiple user and assistant response coming out of this. Given that there isn't a benefit to validating additional message events, this can match what is in the sync PR. That includes just the first two messages.

umaannamalai · 2023-10-17T17:25:23Z

newrelic/hooks/mlmodel_openai.py

 from newrelic.common.object_names import callable_name
+from newrelic.common.object_wrapper import wrap_function_wrapper
 from newrelic.core.attribute import MAX_LOG_MESSAGE_LENGTH


This should have been removed in the sync PR, but we probably don't need to do this truncation check now within the instrumentation. Truncation of attributes to 4096 will be handled in a subsequent PR.

umaannamalai · 2023-10-17T17:27:56Z

newrelic/hooks/mlmodel_openai.py

+        "response.headers.ratelimitRemainingRequests": check_rate_limit_header(
+            response_headers, "x-ratelimit-remaining-requests", True
+        ),
+        "vendor": "openAI",


Looks like vendor is in here twice :)

umaannamalai · 2023-10-17T17:35:57Z

Thanks for the cleanup and housekeeping in this PR!

umaannamalai · 2023-10-17T23:24:11Z

newrelic/hooks/mlmodel_openai.py

-    message_list = list(kwargs.get("messages", [])) + [response.choices[0].message]
+    message_list = list(messages)
+    if choices:
+        message_list.extend([choices[0].message])


Maybe we switch the dot operator here to a getattr check.

umaannamalai · 2023-10-17T23:24:19Z

newrelic/hooks/mlmodel_openai.py

        "response.usage.completion_tokens": response_usage.get("completion_tokens", "") if any(response_usage) else "",
        "response.usage.total_tokens": response_usage.get("total_tokens", "") if any(response_usage) else "",
        "response.usage.prompt_tokens": response_usage.get("prompt_tokens", "") if any(response_usage) else "",
        "request.temperature": kwargs.get("temperature", ""),
        "request.max_tokens": kwargs.get("max_tokens", ""),
-        "response.choices.finish_reason": response.choices[0].finish_reason,
-        "response.api_type": response.api_type,
+        "response.choices.finish_reason": choices[0].finish_reason if choices else "",


Maybe we switch the dot operator here to a getattr check.

I don't want this to block merging so will address this in a followup PR.

* Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Update OpenAI testing infra to match bedrock (#939) * Add OpenAI sync chat completion instrumentation (#934) * Add openai sync instrumentation. * Remove commented code. * Test cleanup. * Add request/ response IDs. * Fixups. * Add conversation ID to message events. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add OpenAI sync embedding instrumentation (#938) * Add sync instrumentation for OpenAI embeddings. * Remove comments. * Clean up embedding event dictionary. * Update response_time to duration. * Linting fixes. * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Instrument acreate's for open-ai (#935) * Instrument acreate's for open ai async * Remove duplicated vendor * Re-use expected & input payloads in tests * Attach ml_event to APM entity by default (#940) * Attach non InferenceEvents to APM entity * Validate both resource payloads * Add tests for non-inference events * Add OpenAI sync embedding instrumentation (#938) * Add sync instrumentation for OpenAI embeddings. * Remove comments. * Clean up embedding event dictionary. * Update response_time to duration. * Linting fixes. * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Fixup: test names --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add truncation for ML events. (#943) * Add 4096 char truncation for ML events. * Add max attr check. * Fixup. * Fix character length ml event test. * Ignore test_ml_events.py for Py2. * Cleanup custom event if checks. * Add import statement. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add framework metric for OpenAI. (#945) * Add framework metric for OpenAI. * [Mega-Linter] Apply linters fixes * Trigger tests * Fix missing version info. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add truncation support for ML events recorded outside txns. (#949) * Add ml tests for outside transaction. * Update validator. * Add ML flag to application code path for record_ml_event. * Bedrock Testing Infrastructure (#937) * Add AWS Bedrock testing infrastructure * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Remove OpenAI references --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Mock openai error responses (#950) * Add example tests and mock error responses * Set invalid api key in auth error test Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> * OpenAI ErrorTrace attributes (#941) * Add openai sync instrumentation. * Remove commented code. * Initial openai error commit * Add example tests and mock error responses * Changes to attribute collection * Change error tests to match mock server * [Mega-Linter] Apply linters fixes * Trigger tests * Add dt_enabled decorator to error tests * Add embedded and async error tests * [Mega-Linter] Apply linters fixes * Trigger tests * Add http.statusCode to span before notice_error call * Report number of messages in error trace even if 0 * Revert notice_error and add _nr_message attr * Remove enabled_ml_settings as not needed * Add stats engine _nr_message test * [Mega-Linter] Apply linters fixes * Trigger tests * Revert black formatting in unicode/byte messages --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: lrafeei <lrafeei@users.noreply.github.com> Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com> * Bedrock Sync Chat Completion Instrumentation (#953) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * TEMP * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Drop python 3.7 tests for Hypercorn (#954) * Apply suggestions from code review * Remove unused import --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Feature bedrock cohere instrumentation (#955) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Add cohere model * Remove openai instrumentation from this branch * Remove OpenAI from newrelic/config.py --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Tim Pansino <timpansino@gmail.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * AWS Bedrock Embedding Instrumentation (#957) * AWS Bedrock embedding instrumentation * Correct symbol name * Add support for bedrock claude (#960) Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> * Combine Botocore Tests (#959) * Initial file migration * Enable DT on all span tests * Add pytest skip for older botocore versions * Fixup: app name merge conflict --------- Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Pin openai tests to below 1.0 (#962) * Pin openai below 1.0 * Fixup * Add openai feedback support (#942) * Add get_ai_message_ids & message id capturing * Add tests * Remove generator * Add tests for conversation id unset * Add error code to mocked responses * Remove bedrock tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Add ingest source to openai events (#961) * Pin openai below 1.0 * Fixup * Add ingest_source to events * Remove duplicate test file * Handle 0.32.0.post1 version in tests (#963) --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Handle 0.32.0.post1 version in tests (#963) * Initial merge commit * Update moto * Test for Bedrock embeddings metrics * Add record_llm_feedback_event API (#964) * Implement record_ai_feedback API. * [Mega-Linter] Apply linters fixes * Change API name to record_ai_feedback_event. * Fix API naming. * Rename to record_llm_feedback_event and get_llm_message_ids. * [Mega-Linter] Apply linters fixes * Address review feedback. * Update test structure. * [Mega-Linter] Apply linters fixes * Bump tests. --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> * Bedrock Error Tracing (#966) * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Drop python 3.7 tests for Hypercorn (#954) * Fix pyenv installation for devcontainer (#936) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Remove duplicate kafka import hook (#956) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Initial bedrock error tracing commit * Handle 0.32.0.post1 version in tests (#963) * Add status code to mock bedrock server * Updating error response recording logic * Work on bedrock errror tracing * Chat completion error tracing * Adding embedding error tracing * Delete comment * Update moto --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Fix expected chat completion tests * Remove commented out code * Correct Bedrock metric name --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: lrafeei <lrafeei@users.noreply.github.com> Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Tim Pansino <timpansino@gmail.com>

* Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Update OpenAI testing infra to match bedrock (#939) * Add OpenAI sync chat completion instrumentation (#934) * Add openai sync instrumentation. * Remove commented code. * Test cleanup. * Add request/ response IDs. * Fixups. * Add conversation ID to message events. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add OpenAI sync embedding instrumentation (#938) * Add sync instrumentation for OpenAI embeddings. * Remove comments. * Clean up embedding event dictionary. * Update response_time to duration. * Linting fixes. * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Instrument acreate's for open-ai (#935) * Instrument acreate's for open ai async * Remove duplicated vendor * Re-use expected & input payloads in tests * Attach ml_event to APM entity by default (#940) * Attach non InferenceEvents to APM entity * Validate both resource payloads * Add tests for non-inference events * Add OpenAI sync embedding instrumentation (#938) * Add sync instrumentation for OpenAI embeddings. * Remove comments. * Clean up embedding event dictionary. * Update response_time to duration. * Linting fixes. * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Fixup: test names --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add truncation for ML events. (#943) * Add 4096 char truncation for ML events. * Add max attr check. * Fixup. * Fix character length ml event test. * Ignore test_ml_events.py for Py2. * Cleanup custom event if checks. * Add import statement. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add framework metric for OpenAI. (#945) * Add framework metric for OpenAI. * [Mega-Linter] Apply linters fixes * Trigger tests * Fix missing version info. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add truncation support for ML events recorded outside txns. (#949) * Add ml tests for outside transaction. * Update validator. * Add ML flag to application code path for record_ml_event. * Bedrock Testing Infrastructure (#937) * Add AWS Bedrock testing infrastructure * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Remove OpenAI references --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Mock openai error responses (#950) * Add example tests and mock error responses * Set invalid api key in auth error test Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> * OpenAI ErrorTrace attributes (#941) * Add openai sync instrumentation. * Remove commented code. * Initial openai error commit * Add example tests and mock error responses * Changes to attribute collection * Change error tests to match mock server * [Mega-Linter] Apply linters fixes * Trigger tests * Add dt_enabled decorator to error tests * Add embedded and async error tests * [Mega-Linter] Apply linters fixes * Trigger tests * Add http.statusCode to span before notice_error call * Report number of messages in error trace even if 0 * Revert notice_error and add _nr_message attr * Remove enabled_ml_settings as not needed * Add stats engine _nr_message test * [Mega-Linter] Apply linters fixes * Trigger tests * Revert black formatting in unicode/byte messages --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: lrafeei <lrafeei@users.noreply.github.com> Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com> * Bedrock Sync Chat Completion Instrumentation (#953) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * TEMP * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Drop python 3.7 tests for Hypercorn (#954) * Apply suggestions from code review * Remove unused import --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Feature bedrock cohere instrumentation (#955) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Add cohere model * Remove openai instrumentation from this branch * Remove OpenAI from newrelic/config.py --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Tim Pansino <timpansino@gmail.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * AWS Bedrock Embedding Instrumentation (#957) * AWS Bedrock embedding instrumentation * Correct symbol name * Add support for bedrock claude (#960) Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> * Combine Botocore Tests (#959) * Initial file migration * Enable DT on all span tests * Add pytest skip for older botocore versions * Fixup: app name merge conflict --------- Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Pin openai tests to below 1.0 (#962) * Pin openai below 1.0 * Fixup * Add openai feedback support (#942) * Add get_ai_message_ids & message id capturing * Add tests * Remove generator * Add tests for conversation id unset * Add error code to mocked responses * Remove bedrock tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Add ingest source to openai events (#961) * Pin openai below 1.0 * Fixup * Add ingest_source to events * Remove duplicate test file * Handle 0.32.0.post1 version in tests (#963) --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Handle 0.32.0.post1 version in tests (#963) * Initial merge commit * Update moto * Test for Bedrock embeddings metrics * Add record_llm_feedback_event API (#964) * Implement record_ai_feedback API. * [Mega-Linter] Apply linters fixes * Change API name to record_ai_feedback_event. * Fix API naming. * Rename to record_llm_feedback_event and get_llm_message_ids. * [Mega-Linter] Apply linters fixes * Address review feedback. * Update test structure. * [Mega-Linter] Apply linters fixes * Bump tests. --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> * Bedrock Error Tracing (#966) * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Drop python 3.7 tests for Hypercorn (#954) * Fix pyenv installation for devcontainer (#936) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Remove duplicate kafka import hook (#956) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Initial bedrock error tracing commit * Handle 0.32.0.post1 version in tests (#963) * Add status code to mock bedrock server * Updating error response recording logic * Work on bedrock errror tracing * Chat completion error tracing * Adding embedding error tracing * Delete comment * Update moto --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Fix expected chat completion tests * Remove commented out code * Switch openai to use custom events. * Cleanup. * Switch Bedrock instrumentation to custom events. * Fix record_feedback test. * Fix disabled settings. * Add attribute length setting to bedrock conftest. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: lrafeei <lrafeei@users.noreply.github.com> Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Tim Pansino <timpansino@gmail.com>

mergify bot added the tests-failing label Oct 11, 2023

hmstepanek force-pushed the add-openai-async branch from 5f0b6ca to 37efe8c Compare October 11, 2023 19:51

hmstepanek changed the base branch from main to develop-openai-instrumentation October 11, 2023 19:52

mergify bot removed the tests-failing label Oct 12, 2023

hmstepanek force-pushed the add-openai-async branch from cb4d869 to 78b35e3 Compare October 13, 2023 17:06

mergify bot added tests-failing merge-conflicts labels Oct 13, 2023

hmstepanek force-pushed the add-openai-async branch from c7b4d36 to 78b35e3 Compare October 13, 2023 21:30

mergify bot removed the merge-conflicts label Oct 13, 2023

hmstepanek changed the title ~~Instrument Embedding.acreate for open ai~~ Instrument acreate's for open-ai Oct 13, 2023

mergify bot added the merge-conflicts label Oct 16, 2023

hmstepanek force-pushed the add-openai-async branch from 5b3dfd7 to 188e7db Compare October 16, 2023 23:22

mergify bot removed the merge-conflicts label Oct 16, 2023

hmstepanek force-pushed the add-openai-async branch 3 times, most recently from 3c6d406 to 57585f4 Compare October 17, 2023 16:14

hmstepanek marked this pull request as ready for review October 17, 2023 16:15

hmstepanek requested a review from a team October 17, 2023 16:15

umaannamalai reviewed Oct 17, 2023

View reviewed changes

umaannamalai requested changes Oct 17, 2023

View reviewed changes

mergify bot added the merge-conflicts label Oct 17, 2023

hmstepanek force-pushed the add-openai-async branch from 2bb9c0c to 079a433 Compare October 17, 2023 20:56

mergify bot removed the merge-conflicts label Oct 17, 2023

hmstepanek added 3 commits October 17, 2023 14:22

Instrument acreate's for open ai async

d5783c5

Remove duplicated vendor

7dc8102

Re-use expected & input payloads in tests

fa61918

hmstepanek force-pushed the add-openai-async branch from 422b051 to fa61918 Compare October 17, 2023 21:22

hmstepanek requested a review from umaannamalai October 17, 2023 21:24

umaannamalai reviewed Oct 17, 2023

View reviewed changes

umaannamalai approved these changes Oct 18, 2023

View reviewed changes

umaannamalai merged commit 51e7362 into develop-openai-instrumentation Oct 18, 2023
46 of 47 checks passed

umaannamalai deleted the add-openai-async branch October 18, 2023 00:55

mergify bot removed the tests-failing label Oct 18, 2023

umaannamalai added this to the v9.8.0 milestone Mar 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Instrument acreate's for open-ai #935

Instrument acreate's for open-ai #935

hmstepanek commented Oct 11, 2023 •

edited

Loading

github-actions bot commented Oct 11, 2023 •

edited

Loading

codecov-commenter commented Oct 11, 2023 •

edited

Loading

umaannamalai Oct 17, 2023

umaannamalai Oct 17, 2023

umaannamalai Oct 17, 2023

umaannamalai Oct 17, 2023

umaannamalai Oct 17, 2023

umaannamalai commented Oct 17, 2023

umaannamalai Oct 17, 2023

umaannamalai Oct 17, 2023

umaannamalai Oct 18, 2023

Instrument acreate's for open-ai #935

Instrument acreate's for open-ai #935

Conversation

hmstepanek commented Oct 11, 2023 • edited Loading

Overview

github-actions bot commented Oct 11, 2023 • edited Loading

🦙 MegaLinter status: ✅ SUCCESS

codecov-commenter commented Oct 11, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

umaannamalai commented Oct 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hmstepanek commented Oct 11, 2023 •

edited

Loading

github-actions bot commented Oct 11, 2023 •

edited

Loading

codecov-commenter commented Oct 11, 2023 •

edited

Loading