feat: add tts to rai core #419
Conversation
src/rai_core/rai/runners/s2s.py
Outdated
| consumer_sampling_rate=vad.sampling_rate, | ||
| is_input=True, | ||
| ) | ||
| asr_agent = VoiceRecognitionAgent( |
There was a problem hiding this comment.
I would make it more explicit so that the user can guess that ros2 is enabled. Please use keyword arguments e.g. ros2_name="automatic_speech_recognition"
There was a problem hiding this comment.
We need to keep in mind that the default run method of the s2s should not use ROS2 to communicate.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
rachwalk
force-pushed
the
refactor/rai_tts
branch
from
3c2d44e to
18957ba
Compare
|
Unfortunately it's still lagging on my setup. The tts is lagging (mic on/mic off). On the first tts launch a loud buzzing could be heard. Even though the TTS lags, ASR and response stopping works very well in the oww commented out scenario. Well done |
There was a problem hiding this comment.
Well done,
lots of random comments, please clean up the code.
Did you test FasterWhisper and OpenAIWhisper?
Please add ElevenLabsTTS, you can use my commit 33e2a45
| ```bash | ||
| python -c 'import sounddevice as sd; print(sd.query_devices())' | ||
| The Agent utilises sounddevice module to access user's microphone, by default the `"default"` sound device is used. | ||
| To get information about available sounddeives use: |
| ``` | ||
|
|
||
| keep_speaker_busy: some speakers may go into low power mode, which may result in truncated speech beginnings. Set to true to play low frequency, low volume noise to prevent sleep mode. | ||
| The device can be identifed by name and passed to the configuration. |
There was a problem hiding this comment.
What do you mean? All of it
|
|
||
| ## TextToSpeechAgent | ||
|
|
||
| See `examples/s2s/wtts.py` for an example usage. |
There was a problem hiding this comment.
Applied in 3579e8e
| $ python ./examples/s2s/asr.py | ||
| $ python examples/s2s/tts.py | ||
| $ python examples/s2s/conversational.py |
There was a problem hiding this comment.
| $ python ./examples/s2s/asr.py | |
| $ python examples/s2s/tts.py | |
| $ python examples/s2s/conversational.py | |
| $ python examples/s2s/asr.py | |
| $ python examples/s2s/tts.py | |
| $ python examples/s2s/conversational.py |
| $ python ./examples/s2s/asr.py | |
| $ python examples/s2s/tts.py | |
| $ python examples/s2s/conversational.py | |
| $ python ./examples/s2s/asr.py | |
| $ python ./examples/s2s/tts.py | |
| $ python ./examples/s2s/conversational.py |
There was a problem hiding this comment.
Applied in 3579e8e
| vad = SileroVAD(args.vad_sampling_rate, args.vad_threshold) | ||
| oww = OpenWakeWord("hey jarvis", args.oww_threshold) | ||
| whisper = LocalWhisper("tiny", args.vad_sampling_rate) | ||
| # whisper = OpenAIWhisper("whisper-1", args.vad_sampling_rate, "en") |
There was a problem hiding this comment.
style
Did you test OpenAIWhisper?
There was a problem hiding this comment.
Added info in 6123d5c
| ros2_name = "rai_asr_agent" | ||
|
|
||
| agent = VoiceRecognitionAgent(microphone_configuration, ros2_name, whisper, vad) | ||
| # agent.add_detection_model(oww, pipeline="record") |
There was a problem hiding this comment.
Please add more information why is this commented out
There was a problem hiding this comment.
Added info in 6123d5c
| _targets = [ | ||
| target | ||
| if isinstance(target, tuple) | ||
| else (target, TopicConfig(is_subscriber=False)) | ||
| ( | ||
| target | ||
| if isinstance(target, tuple) | ||
| else (target, TopicConfig(is_subscriber=False)) | ||
| ) | ||
| for target in targets | ||
| ] | ||
| _sources = [ | ||
| source | ||
| if isinstance(source, tuple) | ||
| else (source, TopicConfig(is_subscriber=True)) | ||
| ( | ||
| source | ||
| if isinstance(source, tuple) | ||
| else (source, TopicConfig(is_subscriber=True)) | ||
| ) |
There was a problem hiding this comment.
Just curious, what's going on here?
There was a problem hiding this comment.
Pyright prefers this for some reason to simple multiline configuration. It's ruff compatible, and python valid syntax. Do you want this changed?
| # def run(self): | ||
| # self._executor.spin() | ||
|
|
There was a problem hiding this comment.
Applied in e69437e
Purpose
With the new design based on agents and connectors there was need to refactor rai tts, to reflect these changes.
Proposed Changes
Adds:
Issues
#399
#309
Testing
CI
Manual tests performed using soundevices:
"default"(ALSA - system defautl)"Jabra Speak2 40 MS: USB Audio"(conference speaker/microphone)"Sennheiser USB headset: Audio"(headset)To run and test this PR run:
docker run -it -p 5500:5500 synesthesiam/opentts:en --no-espeakand then in 3 separate terminal windows run:
python examples/s2s/asr.pypython examples/s2s/tts.pypython examples/s2s/conversational.pyIf you want to specify a particular device (example):
python ./examples/s2s/asr.py --device-name "Jabra Speak2 40 MS: USB Audio (hw:2,0)"python examples/s2s/tts.py --device-name "Jabra Speak2 40 MS: USB Audio (hw:2,0)"python examples/s2s/conversational.pyAlso note: Default values of agents configurations should work ok, but it's not guaranteed. Particular sound data profiles can have a significant effect on the accuracy of voice activity and wake word detections, so for best results it is reccomended to experiment with the values, until desired accuracy is achieved. A very common option to change is
VAD_DETECTION_THRESHOLDinexamples/s2s/asr.py, as the VAD performance with given device can differ a lot.