Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ocp pipeline #396

Merged
merged 1 commit into from
Feb 2, 2024
Merged

ocp pipeline #396

merged 1 commit into from
Feb 2, 2024

Conversation

NeonJarbas
Copy link

@NeonJarbas NeonJarbas commented Jan 6, 2024 •
edited
Loading

docs OpenVoiceOS/ovos-technical-manual#14
training https://github.com/OpenVoiceOS/ovos-classifiers/tree/dev/scripts/training/ocp

Layer 1 - Unambiguous

Before regular intent stage, taking into account current OCP state (media ready to play / playing)

Only matches if user unambiguously wants to trigger OCP

uses padacioso for exact matches

  • play {query}
  • previous (media needs to be loaded)
  • next (media needs to be loaded)
  • pause (media needs to be loaded)
  • play / resume (media needs to be loaded)
  • stop (media needs to be loaded)
from ocp_nlp.intents import OCPPipelineMatcher

ocp = OCPPipelineMatcher()
print(ocp.match_high("play metallica", "en-us"))
# IntentMatch(intent_service='OCP_intents',
#   intent_type='ocp:play',
#   intent_data={'media_type': <MediaType.MUSIC: 2>, 'query': 'metallica',
#                'entities': {'album_name': 'Metallica', 'artist_name': 'Metallica'},
#                'conf': 0.96, 'lang': 'en-us'},
#   skill_id='ovos.common_play', utterance='play metallica')

Layer 2 - Semi-Ambiguous

uses a binary classifier to detect if a query is about media playback

from ocp_nlp.intents import OCPPipelineMatcher

ocp = OCPPipelineMatcher()

print(ocp.match_high("put on some metallica", "en-us"))
# None

print(ocp.match_medium("put on some metallica", "en-us"))
# IntentMatch(intent_service='OCP_media',
#   intent_type='ocp:play',
#   intent_data={'media_type': <MediaType.MUSIC: 2>,
#                'entities': {'album_name': 'Metallica', 'artist_name': 'Metallica', 'movie_name': 'Some'},
#                'query': 'put on some metallica',
#                'conf': 0.9578441098114333},
#   skill_id='ovos.common_play', utterance='put on some metallica')

Layer 3 - Ambiguous

Uses keyword matching and requires at least 1 keyword

OCP skills can provide these keywords at runtime, additional keywords for things such as media_genre were collected via SPARQL queries to wikidata

from ocp_nlp.intents import OCPPipelineMatcher

ocp = OCPPipelineMatcher()

print(ocp.match_medium("i wanna hear metallica", "en-us"))
# None

print(ocp.match_fallback("i wanna hear metallica", "en-us"))
#  IntentMatch(intent_service='OCP_fallback',
#    intent_type='ocp:play',
#    intent_data={'media_type': <MediaType.MUSIC: 2>,
#                 'entities': {'album_name': 'Metallica', 'artist_name': 'Metallica'},
#                 'query': 'i wanna hear metallica',
#                 'conf': 0.5027561091821287},
#    skill_id='ovos.common_play', utterance='i wanna hear metallica')

Config

companion PR OpenVoiceOS/ovos-config#96

{
  "OCP": {
     // ignore entries that report a score below this value
     // this is the way for the skills to say they are not confident
     // 0 - 100
     "min_score": 50,

    // minimum required confidence of classifier,
    // if below this the query uses MediaType.GENERIC
    // 0.0 - 1.0
    "classifier_threshold": 0.5,

    // ignore results that dont match the classifier's media type
    // does not apply to MediaType.GENERIC
    "filter_media": true

  }
}

LOGS

2024-01-09 23:46:23.880 - skills - ovos_core.transformers:transform:60 - DEBUG - ovos-utterance-normalizer: {'session': {'active_skills': [], 'utterance_states': {}, 'session_id': 'default', 'lang': 'en-us', 'context': {'timeout': 120, 'frame_stack': []}, 'site_id': 'unknown', 'pipeline': ['converse', 'ocp_high', 'padatious_high', 'adapt', 'common_qa', 'ocp_medium', 'fallback_high', 'padatious_medium', 'fallback_medium', 'ocp_fallback', 'fallback_low'], 'stt': {'plugin_id': 'ovos-stt-plugin-server', 'config': {}}, 'tts': {'plugin_id': 'ovos-tts-plugin-server', 'config': {}}}, 'lang': 'en-us'}
2024-01-09 23:46:23.992 - skills - padacioso:calc_intent:255 - DEBUG - {'entities': {'query': 'lovecraft'}, 'conf': 0.96, 'name': 'play'}
2024-01-09 23:46:27.150 - skills - ovos_core.intent_services.ocp_service:classify_media:552 - INFO - OVOSCommonPlay MediaType prediction: movie confidence: 0.3032108745919962
2024-01-09 23:46:27.150 - skills - ovos_core.intent_services.ocp_service:classify_media:553 - DEBUG -      utterance: lovecraft
2024-01-09 23:46:27.150 - skills - ovos_core.intent_services.ocp_service:classify_media:555 - INFO - ignoring MediaType classifier, low confidence prediction
2024-01-09 23:46:27.153 - skills - ovos_core.intent_services:handle_utterance:318 - DEBUG - intent matching took: 3.2681217193603516
2024-01-09 23:46:27.162 - skills - ovos_bus_client.session:update:546 - DEBUG - replacing default session with: {'active_skills': [['ovos.common_play', 1704843987.1537983]], 'utterance_states': {}, 'session_id': 'default', 'lang': 'en-us', 'context': {'timeout': 120, 'frame_stack': []}, 'site_id': 'unknown', 'pipeline': ['converse', 'ocp_high', 'padatious_high', 'adapt', 'common_qa', 'ocp_medium', 'fallback_high', 'padatious_medium', 'fallback_medium', 'ocp_fallback', 'fallback_low'], 'stt': {'plugin_id': 'ovos-stt-plugin-server', 'config': {}}, 'tts': {'plugin_id': 'ovos-tts-plugin-server', 'config': {}}}
2024-01-09 23:46:27.162 - skills - ovos_bus_client.client.client:on_default_session_update:161 - DEBUG - synced default_session
2024-01-09 23:46:27.162 - skills - ovos_bus_client.apis.ocp:__init__:828 - DEBUG - Created GENERIC query: lovecraft
2024-01-09 23:46:27.315 - skills - ovos_bus_client.apis.ocp:register_events:874 - DEBUG - Registering Search Bus Events
2024-01-09 23:46:27.330 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-lovecraft-comics.jarbasai is searching
2024-01-09 23:46:27.332 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-librivoxx.jarbasai is searching
2024-01-09 23:46:27.332 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-sovietwave.jarbasai is searching
2024-01-09 23:46:27.334 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-underrated-albums.jarbasai is searching
2024-01-09 23:46:27.335 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-horrorbabble.jarbasai is searching
2024-01-09 23:46:27.337 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-reddit-cartoons.jarbasai is searching
2024-01-09 23:46:27.338 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-smod.jarbasai is searching
2024-01-09 23:46:27.338 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-ccc.jarbasai
2024-01-09 23:46:27.339 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-WTvFDigitalGrindhouseDriveIn.jarbasai is searching
2024-01-09 23:46:27.339 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-ccc.jarbasai is searching
2024-01-09 23:46:27.341 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-lovecraft-comics.jarbasai
2024-01-09 23:46:27.341 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-wayne-june-lovecraft.jarbasai is searching
2024-01-09 23:46:27.342 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-ccc.jarbasai finished search
2024-01-09 23:46:27.342 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-lovecraft-comics.jarbasai finished search
2024-01-09 23:46:27.386 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-WTvFDigitalGrindhouseDriveIn.jarbasai
2024-01-09 23:46:27.386 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-sovietwave.jarbasai
2024-01-09 23:46:27.386 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-sovietwave.jarbasai finished search
2024-01-09 23:46:27.392 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.396 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 45 results from skill-horrorbabble.jarbasai
2024-01-09 23:46:27.397 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-WTvFDigitalGrindhouseDriveIn.jarbasai finished search
2024-01-09 23:46:27.397 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-underrated-albums.jarbasai
2024-01-09 23:46:27.398 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-horrorbabble.jarbasai finished search
2024-01-09 23:46:27.400 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 1 results from skill-smod.jarbasai
2024-01-09 23:46:27.400 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-underrated-albums.jarbasai finished search
2024-01-09 23:46:27.401 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 1 results from skill-smod.jarbasai
2024-01-09 23:46:27.402 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.405 - skills - ovos_workshop.resource_files:_locate:344 - ERROR - Could not find resource file librivox.voc
2024-01-09 23:46:27.407 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-reddit-cartoons.jarbasai
2024-01-09 23:46:27.407 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-smod.jarbasai finished search
2024-01-09 23:46:27.413 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-reddit-cartoons.jarbasai finished search
2024-01-09 23:46:27.414 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 1 results from skill-wayne-june-lovecraft.jarbasai
2024-01-09 23:46:27.414 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 10 results from skill-wayne-june-lovecraft.jarbasai
2024-01-09 23:46:27.414 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.415 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.415 - skills - ovos_workshop.resource_files:_locate:344 - ERROR - Could not find resource file librivox.voc
2024-01-09 23:46:27.452 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-wayne-june-lovecraft.jarbasai finished search
2024-01-09 23:46:28.480 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-librivoxx.jarbasai
2024-01-09 23:46:28.481 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-librivoxx.jarbasai finished search
2024-01-09 23:46:28.982 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:968 - INFO - Received search responses from all skills!
2024-01-09 23:46:29.021 - skills - ovos_bus_client.apis.ocp:remove_events:880 - DEBUG - Removing Search Bus Events
2024-01-09 23:46:29.022 - skills - ovos_core.intent_services.ocp_service:_execute_query:676 - DEBUG - Returning 5 search results
2024-01-09 23:46:29.022 - skills - ovos_core.intent_services.ocp_service:_search:650 - DEBUG - Got 58 results
2024-01-09 23:46:29.022 - skills - ovos_core.intent_services.ocp_service:filter_results:604 - DEBUG - filtered -2 low confidence results
2024-01-09 23:46:29.027 - skills - ovos_core.intent_services.ocp_service:_search:652 - DEBUG - Got 56 usable results
2024-01-09 23:46:29.027 - skills - ovos_core.intent_services.ocp_service:handle_play_intent:422 - DEBUG - Playing 56 results for: lovecraft
2024-01-09 23:46:29.028 - skills - ovos_core.intent_services.ocp_service:select_best:724 - INFO - OVOSCommonPlay selected: skill-horrorbabble.jarbasai - 60
2024-01-09 23:46:29.028 - skills - ovos_core.intent_services.ocp_service:select_best:725 - DEBUG - {'title': '"The Man of Stone" by H. P. Lovecraft / A HorrorBabble Production', 'author': 'Lovecraft', 'match_confidence': 60, 'media_type': 4, 'uri': 'youtube//https://youtube.com/watch?v=I5W3nk2Yrfs', 'playback': 2, 'skill_icon': 'https://github.com/OpenVoiceOS/ovos-ocp-audio-plugin/raw/master/ovos_plugin_common_play/ocp/res/ui/images/ocp.png', 'skill_id': 'skill-horrorbabble.jarbasai', 'image': 'https://i.ytimg.com/vi/I5W3nk2Yrfs/sddefault.jpg', 'bg_image': '/home/miro/.venvs/OCP/lib/python3.9/site-packages/skill_horrorbabble/res/bg.png'}

Issues

closes OpenVoiceOS/ovos-ocp-audio-plugin#73

@NeonJarbas NeonJarbas changed the title ocp ocp pipeline Jan 6, 2024
@codecov Codecov
Copy link

codecov bot commented Jan 6, 2024 •
edited
Loading

Codecov Report

Attention: 346 lines in your changes are missing coverage. Please review.

Comparison is base (972b319) 63.41% compared to head (6c9b852) 56.79%.
Report is 2 commits behind head on dev.

Files Patch % Lines
ovos_core/intent_services/ocp_service.py 27.84% 342 Missing ⚠️
ovos_core/intent_services/__init__.py 75.00% 4 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##              dev     #396      +/-   ##
==========================================
- Coverage   63.41%   56.79%   -6.62%     
==========================================
  Files          14       15       +1     
  Lines        2069     2553     +484     
==========================================
+ Hits         1312     1450     +138     
- Misses        757     1103     +346
Flag Coverage Δ
unittests 56.79% <29.67%> (-6.62%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@NeonJarbas NeonJarbas force-pushed the feat/ocp_pipeline branch 2 times, most recently from fd2dd8d to 1a4592d Compare January 6, 2024 01:56
@JarbasAl JarbasAl mentioned this pull request Jan 6, 2024
Merged
@NeonJarbas NeonJarbas mentioned this pull request Jan 6, 2024
Merged
@NeonJarbas NeonJarbas force-pushed the feat/ocp_pipeline branch 2 times, most recently from 268c2a1 to 926e498 Compare January 9, 2024 23:43
@NeonJarbas NeonJarbas force-pushed the feat/ocp_pipeline branch 2 times, most recently from 9ec2308 to 69e25e6 Compare January 29, 2024 06:35
@JarbasAl JarbasAl marked this pull request as ready for review February 1, 2024 22:54
@JarbasAl JarbasAl added the enhancement New feature or request label Feb 1, 2024
@JarbasAl
feat/ocp_pipeline
6c9b852 
feat/liked tracks intents

fix/MediaEntry_Playlist

SEI handling

typing

ocp_entities
no bumping utils to 0.1.0

:tada:

unify locale folders

rm unused files
@JarbasAl JarbasAl merged commit 330fcee into OpenVoiceOS:dev Feb 2, 2024
2 of 4 checks passed
JarbasAl added a commit that referenced this pull request Feb 2, 2024
@JarbasAl
fix/OCP missing languages
bacb661 
untranslated languages cause OCP pipeline to load , introduced in #396
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

"enable audio recordings" intent match
2 participants
@NeonJarbas @JarbasAl