求大神指导一下unity编译安卓手机apk如何调用sheerpa-onnx

[newkitty]

我是新手小白，想自己练练手，所以想用unity直接编译个手机app看看，我按照一位前辈的issue里面的操作步骤做了，但是刚开始提示找不到sherpa-onnx，我放了sherpa-onnx.dll到plugin下面，其他的都和前辈发文截图里面的差不多一样，然后在文件拷贝完成后就闪退，请大神指导一下unity编译安卓手机apk如何调用sheerpa-onnx，感谢！ Good Morning,
i will be sharing the code needed to make sherpa-onnx tts work on unity specifically on android builds!
the problem is that unity merges the model files that we need for the tts in that jar file, making it not possible to pass the path directly to the sherpa-onnx plugin, and if you do the app will crash/quit!
there are work arounds for this, the simplest is using UnityWebRequest (similar to www of .net) to copy the files from the archive location in streamingAssets into a readable location and pass the new location to the sherpa-onnx plugin,
to do this, i created StreamingAssetsApi scripts which take care of such operations.
first on the unity editor side we have a script that records the hierarchy of the streamingAssets folder and writes it in a text file in the streaming assets (think of it as an index page of a book):

using UnityEngine;
using UnityEditor;
using System.IO;
using System.Collections.Generic;

public static class StreamingAssetsHierarchyBuilder
{
    // The output text file name:
    private const string OUTPUT_FILE_NAME = "StreamingAssetsHierarchy.txt";


    // Add a menu item to generate manually.
    // You could also run this from a build processor or any other Editor event.
    [MenuItem("BuildTools/Generate StreamingAssets Hierarchy")]
    public static void GenerateHierarchyFile()
    {
        // 1) The root folder in the Unity project for streaming assets
        string streamingAssetsRoot = Application.dataPath + "/StreamingAssets";
        if (!Directory.Exists(streamingAssetsRoot))
        {
            Debug.LogWarning("No StreamingAssets folder found. Creating one...");
            Directory.CreateDirectory(streamingAssetsRoot);
        }

        // 2) Collect all relative file paths inside streamingAssetsRoot
        List<string> allFiles = new List<string>();
        RecursivelyCollectFiles(streamingAssetsRoot, streamingAssetsRoot, allFiles);

        // 3) Write them to a text file
        // We’ll place it at the root of StreamingAssets for easy access
        string outputPath = Path.Combine(streamingAssetsRoot, OUTPUT_FILE_NAME);
        File.WriteAllLines(outputPath, allFiles);

        Debug.Log($"[StreamingAssetsHierarchyBuilder] Generated {allFiles.Count} entries in {OUTPUT_FILE_NAME}");
        // Refresh so Unity sees the new/updated text file
        AssetDatabase.Refresh();
    }

    /// <summary>
    /// Recursively scans 'currentPath' for files and subfolders,
    /// and appends their relative paths (relative to 'rootPath') to 'results'.
    /// </summary>
    private static void RecursivelyCollectFiles(string rootPath, string currentPath, List<string> results)
    {
        // Grab all files in this folder
        string[] files = Directory.GetFiles(currentPath);
        foreach (var file in files)
        {
            // Skip meta files
            if (file.EndsWith(".meta")) 
                continue;

            // Make a relative path from the root of StreamingAssets
            // e.g. root = c:\Project\Assets\StreamingAssets
            // file = c:\Project\Assets\StreamingAssets\Models\Foo\bar.txt
            // relative = Models/Foo/bar.txt
            string relativePath = file.Substring(rootPath.Length + 1)
                                    .Replace("\\", "/");
            results.Add(relativePath);
        }

        // Recurse subfolders
        string[] directories = Directory.GetDirectories(currentPath);
        foreach (var dir in directories)
        {
            // Skip .meta or system folders if any
            if (dir.EndsWith(".meta"))
                continue;

            RecursivelyCollectFiles(rootPath, dir, results);
        }
    }
}

with this we will have a unity function that shows in the editor like this:

so obviously to use it we click BuildsTools > Generate StreamingAssets Hierarchy.
this creates the following file that contains the streamingAssets Hierarchy:

and it contains something like this:

Optionally we can make unity call this function automatically each time we do a build:

using System.Collections;
using System.Collections.Generic;
using UnityEditor.Build;
using UnityEditor.Build.Reporting;
using UnityEngine;

public class PreBuildHierarchyUpdate : IPreprocessBuildWithReport
{
    public int callbackOrder => 0;

    public void OnPreprocessBuild(BuildReport report)
    {
        StreamingAssetsHierarchyBuilder.GenerateHierarchyFile();
    }
}

now on the android side, the StreamingAssetsApi script contains functions to copy a file or a directory recursively from the streaming assets to different directory, along with function that use the hierarchy file to get a list of files in a certain directory or sub directory.

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using UnityEngine;
using UnityEngine.Networking;

public static class StreamingAssetsAPI
{
    private const string HIERARCHY_FILE = "StreamingAssetsHierarchy.txt";


    /// <summary>
    /// Wraps an IEnumerator-based Unity coroutine in a Task,
    /// allowing you to 'await' it in an async method.
    /// </summary>
    public static Task RunCoroutine(this MonoBehaviour runner, IEnumerator coroutine)
    {
        var tcs = new TaskCompletionSource<bool>();
        runner.StartCoroutine(RunCoroutineInternal(coroutine, tcs));
        return tcs.Task;
    }

    private static IEnumerator RunCoroutineInternal(IEnumerator coroutine, TaskCompletionSource<bool> tcs)
    {
        yield return coroutine;
        tcs.SetResult(true);
    }


    /// <summary>
    /// Retrieves the hierarchy of files (relative paths) under a given subfolder in
    /// StreamingAssets, as defined in <see cref="HIERARCHY_FILE"/>. 
    /// 
    /// This method is NOT a coroutine itself. Instead, it starts an internal 
    /// coroutine to load and filter the hierarchy. Once loading finishes, 
    /// the provided <paramref name="onComplete"/> callback is invoked with 
    /// the resulting list of file paths. If there is an error, it passes <c>null</c>.
    /// 
    /// <para>Usage example:</para>
    /// <code>
    /// void Start()
    /// {
    ///     // Suppose 'this' is a MonoBehaviour
    ///     var api = new StreamingAssetsHierarchyAPI();
    ///     api.GetHierarchy(this, "Models", (files) =>
    ///     {
    ///         if (files == null)
    ///         {
    ///             Debug.LogError("Failed to retrieve files!");
    ///             return;
    ///         }
    ///         Debug.Log("Received " + files.Count + " files in 'Models'.");
    ///     });
    /// }
    /// </code>
    /// </summary>
    /// <param name="runner">A MonoBehaviour used to start the internal coroutine.</param>
    /// <param name="subfolder">
    /// The subfolder (relative to StreamingAssets root) to filter by. 
    /// If empty or null, it returns the entire hierarchy.
    /// </param>
    /// <param name="onComplete">
    /// Callback invoked once the list of files is ready. If an error occurs, <c>null</c> is passed.
    /// </param>
    public static void GetHierarchy( this MonoBehaviour runner, string subfolder, Action<List<string>> onComplete)
    {
        // Validate runner
        if (runner == null)
        {
            Debug.LogError("[StreamingAssetsHierarchyAPI] No MonoBehaviour provided to start coroutine!");
            onComplete?.Invoke(null);
            return;
        }

        // Start the internal coroutine
        runner.StartCoroutine(GetHierarchyCoroutine(subfolder, onComplete));
    }

    // The coroutine that actually fetches the hierarchy file
    public static IEnumerator GetHierarchyCoroutine(string subfolder, Action<List<string>> onComplete = null)
    {
        // 1) Load the entire hierarchy from the text file
        yield return GetHierarchyForSubfolder(subfolder, (list) =>
        {
            // This callback is invoked once the text is loaded & filtered
            onComplete?.Invoke(list);
        });
    }

    /// <summary>
    /// Reads the entire hierarchy from the generated file,
    /// filters for those starting with 'subfolder',
    /// and returns their relative paths through <paramref name="callback"/>.
    /// 
    /// Typically you won't call this method directly; instead, use <see cref="GetHierarchy"/>.
    /// 
    /// Example usage manually (if you wanted a coroutine):
    /// <code>
    /// yield return StartCoroutine(
    ///     StreamingAssetsHierarchyAPI.GetHierarchyForSubfolder("Models", (list) =&gt; { ... })
    /// );
    /// </code>
    /// </summary>
    /// <param name="subfolder">
    /// The subfolder (relative to StreamingAssets root) to filter by. 
    /// If empty, returns all paths.
    /// </param>
    /// <param name="callback">
    /// Invoked with a list of paths, or <c>null</c> if there's an error.
    /// </param>
    public static IEnumerator GetHierarchyForSubfolder(string subfolder, Action<List<string>> callback)
    {
        string path = Path.Combine(Application.streamingAssetsPath, HIERARCHY_FILE);

        using (UnityWebRequest www = UnityWebRequest.Get(path))
        {
            yield return www.SendWebRequest();

            if (www.result != UnityWebRequest.Result.Success)
            {
                Debug.LogError($"Failed to load {HIERARCHY_FILE}: {www.error}");
                callback?.Invoke(null);
                yield break;
            }

            // Parse lines
            string fileContent = www.downloadHandler.text;
            string[] allLines = fileContent.Split(
                new char[] { '\r', '\n' },
                StringSplitOptions.RemoveEmptyEntries
            );

            List<string> matched = new List<string>();

            if (string.IsNullOrEmpty(subfolder))
                subfolder = "";

            // We'll unify to forward slashes
            subfolder = subfolder.Replace("\\", "/").Trim().TrimEnd('/');

            foreach (var line in allLines)
            {
                // e.g. "Models/en_US-libritts_r-medium.onnx"
                // If subfolder is "Models", check if line.StartsWith("Models/")
                if (subfolder.Length == 0 || line.StartsWith(subfolder + "/"))
                {
                    matched.Add(line);
                }
            }

            callback?.Invoke(matched);
        }
    }


    /// <summary>
    /// Copies a single file from StreamingAssets (relative path) to a specified 
    /// local filesystem path. This uses UnityWebRequest to handle jar:file:// 
    /// URIs on Android.
    /// 
    /// <para>Example usage:</para>
    /// <code>
    /// yield return StreamingAssetsAPI.CopyOneFile(
    ///     "Models/data.json", 
    ///     "/storage/emulated/0/Android/data/com.example.myapp/files/data.json",
    ///     success => 
    ///     {
    ///         Debug.Log(success ? "File copied!" : "Copy failed.");
    ///     }
    /// );
    /// </code>
    /// </summary>
    /// <param name="relativeFilePath">Path within StreamingAssets. E.g. "Models/data.json".</param>
    /// <param name="destinationFullPath">Full local file path to write to.</param>
    /// <param name="onComplete">Invoked with true if copy succeeded, false on error.</param>
    public static IEnumerator CopyFile( string relativeFilePath, string destinationFullPath, Action<bool> onComplete = null)
    {
        // Build full path to the file in StreamingAssets
        string srcUrl = Path.Combine(Application.streamingAssetsPath, relativeFilePath);

        using (UnityWebRequest www = UnityWebRequest.Get(srcUrl))
        {
            yield return www.SendWebRequest();

            if (www.result != UnityWebRequest.Result.Success)
            {
                Debug.LogError($"[CopyOneFile] Failed to get {relativeFilePath}: {www.error}");
                onComplete?.Invoke(false);
                yield break;
            }

            // Ensure the directory of destinationFullPath exists
            string parentDir = Path.GetDirectoryName(destinationFullPath);
            if (!Directory.Exists(parentDir))
            {
                Directory.CreateDirectory(parentDir);
            }

            // Write the file
            byte[] data = www.downloadHandler.data;
            File.WriteAllBytes(destinationFullPath, data);

            Debug.Log($"[CopyOneFile] Copied {relativeFilePath} -> {destinationFullPath}");
            onComplete?.Invoke(true);
        }
    }

    /// <summary>
    /// Recursively copies *all files* from a given subfolder in StreamingAssets
    /// into the specified local directory (e.g., persistentDataPath).
    /// 
    /// It uses <see cref="GetHierarchyForSubfolder"/> to find all files,
    /// then calls <see cref="CopyOneFile"/> for each. 
    /// 
    /// Example usage:
    /// <code>
    /// yield return StreamingAssetsAPI.CopyDirectory(
    ///     runner: this,
    ///     subfolder: "Models",
    ///     localRoot: Path.Combine(Application.persistentDataPath, "Models"),
    ///     onComplete: () => { Debug.Log("Directory copied!"); }
    /// );
    /// </code>
    /// </summary>
    /// <param name="subfolder">Which subfolder in StreamingAssets to copy. E.g. "Models".</param>
    /// <param name="localRoot">
    /// The local directory path (e.g. persistentDataPath/Models) where files are written.
    /// </param>
    /// <param name="onComplete">Optional callback invoked when done.</param>
    public static IEnumerator CopyDirectory(string subfolder, string localRoot, Action onComplete = null )
    {
        // 1) Get the hierarchy for that subfolder
        bool done = false;
        List<string> fileList = null;

        yield return GetHierarchyForSubfolder(subfolder, list =>
        {
            fileList = list;
            done = true;
        });

        // Wait for callback
        while (!done) 
            yield return null;

        if (fileList == null)
        {
            Debug.LogError($"[CopyDirectory] Could not retrieve hierarchy for {subfolder}.");
            onComplete?.Invoke();
            yield break;
        }

        // e.g. fileList might contain ["Models/foo.txt", "Models/subdir/bar.json", ...]

        // 2) Copy each file
        for (int i = 0; i < fileList.Count; i++)
        {
            string relPath = fileList[i];
            // We want to remove "Models/" if subfolder = "Models"
            // so we only get the portion after that prefix
            string suffix = relPath;

            // unify slashes
            suffix = suffix.Replace("\\", "/");
            subfolder = subfolder.Replace("\\", "/").Trim().TrimEnd('/');

            if (subfolder.Length > 0 && suffix.StartsWith(subfolder + "/"))
            {
                // remove "Models/" prefix
                suffix = suffix.Substring(subfolder.Length + 1);
            }

            // Build destination path
            string dst = Path.Combine(localRoot, suffix);

            // yield return CopyOneFile:
            yield return CopyFile(relPath, dst);
        }

        Debug.Log($"[CopyDirectory] Copied {fileList.Count} files from '{subfolder}' to '{localRoot}'");
        onComplete?.Invoke();
    }
}

the StreamingAssetsApi uses coroutines to handle the copy operations, which you can call using the StartCoroutine function in a MonoBehaviour script, but also has an async await wrapper functions for this to run the coroutines using an awaitable RunCoroutine Function, for example:

await this.RunCoroutine( StreamingAssetsAPI.CopyDirectory( modelsDir, // subfolder in StreamingAssets BuildPath( modelsDir ), () => { Debug.Log( modelsDir+ ": Directory copied!"); } ) );

now that we have the streamingAssetsApi implemented, we can test the sherpa-onnx tts at runtime using the following script:

using UnityEngine;
using System;
using System.Runtime.InteropServices;
using SherpaOnnx;
using System.Text;
using UnityEngine.UI;
using System.IO;
using System.Collections;
using System.Collections.Concurrent;
using System.Linq;
using System.Threading;  // For Thread
using System.Text.RegularExpressions;

public class TTS : MonoBehaviour
{
    [Header("VITS Model Settings")]
    public string modelPath;      // e.g., "vits_generator.onnx"
    public string tokensPath;     // e.g., "tokens.txt"
    public string lexiconPath;    // e.g., "lexicon.txt" (if needed by your model)
    public string dictDirPath;    // e.g., "dict" folder (if needed)

    [Header("VITS Tuning Parameters")]
    [Range(0f, 1f)] public float noiseScale = 0.667f;
    [Range(0f, 1f)] public float noiseScaleW = 0.8f;
    [Range(0.5f, 2f)] public float lengthScale = 1.0f;

    [Header("Offline TTS Config")]
    public int numThreads = 1;
    public bool debugMode = false;
    public string provider = "cpu";  // could be "cpu", "cuda", etc.
    public int maxNumSentences = 1;

    [Header("UI for Testing")]
    public Button generateButton;
    public InputField inputField; // Or Text/TextMeshPro input
    public float speed = 1.0f;    // Speed factor for TTS
    public int speakerId = 0;     // If the model has multiple speakers

    // If you want to see "streaming" attempt
    [SerializeField] private bool streamAudio = false;

    // If true, we'll split text by sentences and generate each one on a background thread
    [SerializeField] private bool splitSentencesAsync = true;

    private OfflineTts offlineTts;

    // We'll reuse one AudioSource for sentence-by-sentence playback
    private AudioSource sentenceAudioSource;

    // For streaming approach
    private ConcurrentQueue<float> streamingBuffer = new ConcurrentQueue<float>();
    private int samplesRead = 0;
    private AudioSource streamingAudioSource;
    private AudioClip streamingClip;

    //put voice models directory path relative to the streaming assets folder
    [SerializeField] private string modelsDir;
    //put espeak-ng data directory path relative to the streaming assets folder
    [SerializeField] private string espeakDir;

    async void Start()
    {
        generateButton.gameObject.SetActive(false);
        //Log Cat log: 2025/01/07 09:56:38.283 6672 7831 Info Unity [CopyDirectory] Copied 355 files from 'espeak-ng-data' to '/storage/emulated/0/Android/data/com.DefaultCompany.TTSDemo/files/espeak-ng-data'
        if( Application.platform == RuntimePlatform.Android )
        {
            Debug.Log("running android copy process!");
            await this.RunCoroutine( StreamingAssetsAPI.CopyDirectory(
            modelsDir,      // subfolder in StreamingAssets
            BuildPath( modelsDir ),
            () => { Debug.Log( modelsDir+ ": Directory copied!"); } ) );

            await this.RunCoroutine( StreamingAssetsAPI.CopyDirectory(
            espeakDir,      // subfolder in StreamingAssets
            BuildPath( espeakDir ),
            () => { Debug.Log( espeakDir +": Directory copied!"); } ) );

        }


        // 1. Prepare the VITS model config
        var vitsConfig = new OfflineTtsVitsModelConfig
        {
            Model = BuildPath(modelPath),
            Lexicon = BuildPath(lexiconPath),
            Tokens = BuildPath(tokensPath),
            DataDir = BuildPath(espeakDir),
            DictDir = BuildPath(dictDirPath),

            NoiseScale = noiseScale,
            NoiseScaleW = noiseScaleW,
            LengthScale = lengthScale
        };

        // 2. Wrap it inside the ModelConfig
        var modelConfig = new OfflineTtsModelConfig
        {
            Vits = vitsConfig,
            NumThreads = numThreads,
            Debug = debugMode ? 1 : 0,
            Provider = provider
        };

        // 3. Create the top-level OfflineTtsConfig
        var ttsConfig = new OfflineTtsConfig
        {
            Model = modelConfig,
            RuleFsts = "",
            MaxNumSentences = maxNumSentences,
            RuleFars = ""
        };

        // 4. Instantiate the OfflineTts object
        Debug.Log("will create offline tts now!");
        offlineTts = new OfflineTts(ttsConfig);
        Debug.Log($"OfflineTts created! SampleRate: {offlineTts.SampleRate}, NumSpeakers: {offlineTts.NumSpeakers}");

        // Create a dedicated AudioSource for sentence-by-sentence playback
        sentenceAudioSource = gameObject.AddComponent<AudioSource>();
        sentenceAudioSource.playOnAwake = false;
        sentenceAudioSource.loop = false;

        // 5. Hook up a button to test TTS
        if (generateButton != null)
        {
            generateButton.gameObject.SetActive(true);

            generateButton.onClick.AddListener(() =>
            {
                if (inputField == null || string.IsNullOrWhiteSpace(inputField.text))
                {
                    Debug.LogWarning("No text to synthesize!");
                    return;
                }
                Speak();
            });
        }
    }

    public void Speak()
    {
        // If we want the sentence-by-sentence approach in a background thread:
        if (splitSentencesAsync)
        {
            StartCoroutine(CoPlayTextBySentenceAsync(inputField.text));
        }
        else
        {
            // The old single-shot approach or streaming approach
            if (streamAudio)
                PlayTextStreamed(inputField.text);
            else
                PlayText(inputField.text);
        }
    }

    /// <summary>
    /// 1) Splits the text into sentences using multiple delimiters,
    /// 2) For each sentence, spawns a background thread to generate TTS,
    /// 3) Waits for generation to finish (without freezing the main thread),
    /// 4) Plays the resulting clip in order.
    /// </summary>
    private IEnumerator CoPlayTextBySentenceAsync(string text)
    {
        // More delimiters: period, question mark, exclamation, semicolon, colon
        // We also handle multiple punctuation in a row, etc.
        // This uses Regex to split on punctuation [.!?;:]+ 
        // Then trim the results and remove empties.
        string[] sentences = Regex.Split(text, @"[\.!\?;:]+")
            .Select(s => s.Trim())
            .Where(s => s.Length > 0)
            .ToArray();

        if (sentences.Length == 0)
        {
            Debug.LogWarning("No valid sentences found in input text.");
            yield break;
        }

        foreach (string sentence in sentences)
        {
            Debug.Log($"[Background TTS] Generating: \"{sentence}\"");
            
            // Prepare a place to store the generated float[] 
            float[] generatedSamples = null;
            bool generationDone = false;

            // Run .Generate(...) on a background thread
            Thread t = new Thread(() =>
            {
                // Generate the audio for this sentence
                OfflineTtsGeneratedAudio generated = offlineTts.Generate(sentence, speed, speakerId);
                generatedSamples = generated.Samples;
                generationDone = true;
            });
            t.Start();

            // Wait until the thread signals it's done
            yield return new WaitUntil(() => generationDone);

            // Back on the main thread, we create the AudioClip and play it
            if (generatedSamples == null || generatedSamples.Length == 0)
            {
                Debug.LogWarning("Generated empty audio for a sentence. Skipping...");
                continue;
            }

            AudioClip clip = AudioClip.Create(
                "SherpaOnnxTTS-SentenceAsync",
                generatedSamples.Length,
                1,
                offlineTts.SampleRate,
                false
            );
            clip.SetData(generatedSamples, 0);

            sentenceAudioSource.clip = clip;
            sentenceAudioSource.Play();
            Debug.Log($"Playing sentence: \"{sentence}\"  length = {clip.length:F2}s");

            // Wait until playback finishes
            while (sentenceAudioSource.isPlaying)
                yield return null;
        }

        Debug.Log("All sentences have been generated (background) and played sequentially.");
    }

    /// <summary>
    /// Single-shot generation on the main thread (blocks Unity for large inputs).
    /// </summary>
    private void PlayText(string text)
    {
        Debug.Log($"Generating TTS for text: '{text}'");
        OfflineTtsGeneratedAudio generated = offlineTts.Generate(text, speed, speakerId);

        float[] pcmSamples = generated.Samples;
        if (pcmSamples == null || pcmSamples.Length == 0)
        {
            Debug.LogError("SherpaOnnx TTS returned empty PCM data.");
            return;
        }

        AudioClip clip = AudioClip.Create(
            "SherpaOnnxTTS",
            pcmSamples.Length,
            1,
            offlineTts.SampleRate,
            false
        );
        clip.SetData(pcmSamples, 0);

        var audioSource = gameObject.AddComponent<AudioSource>();
        audioSource.playOnAwake = false;
        audioSource.clip = clip;
        audioSource.Play();

        Debug.Log($"TTS clip of length {clip.length:F2}s is now playing.");
    }

    /// <summary>
    /// Attempted "streaming" approach. The callback is called only once in practice
    /// for the entire waveform, so it doesn't truly stream partial chunks.
    /// </summary>
    private void PlayTextStreamed(string text)
    {
        Debug.Log($"[Streaming] Generating TTS for text: '{text}'");

        int sampleRate = offlineTts.SampleRate;
        int maxAudioLengthInSamples = sampleRate * 300; // 5 min

        streamingClip = AudioClip.Create(
            "SherpaOnnxTTS-Streamed",
            maxAudioLengthInSamples,
            1,
            sampleRate,
            true,
            OnAudioRead,
            OnAudioSetPosition
        );

        if (streamingAudioSource == null)
            streamingAudioSource = gameObject.AddComponent<AudioSource>();

        streamingAudioSource.playOnAwake = false;
        streamingAudioSource.clip = streamingClip;
        streamingAudioSource.loop = false;

        streamingBuffer = new ConcurrentQueue<float>();
        samplesRead = 0;

        streamingAudioSource.Play();

        // This calls your callback, but typically only once for the entire wave
        offlineTts.GenerateWithCallback(text, speed, speakerId, MyTtsChunkCallback);

        Debug.Log("[Streaming] Playback started; awaiting streamed samples...");
    }

    private int MyTtsChunkCallback(System.IntPtr samplesPtr, int numSamples)
    {
        Debug.Log("chunk callback");
        if (numSamples <= 0)
            return 0;

        float[] chunk = new float[numSamples];
        System.Runtime.InteropServices.Marshal.Copy(samplesPtr, chunk, 0, numSamples);

        foreach (float sample in chunk)
            streamingBuffer.Enqueue(sample);

        return 0; 
    }

    private void OnAudioRead(float[] data)
    {
        for (int i = 0; i < data.Length; i++)
        {
            if (streamingBuffer.TryDequeue(out float sample))
            {
                data[i] = sample;
                samplesRead++;
            }
            else
            {
                data[i] = 0f; // fill silence
            }
        }
    }

    private void OnAudioSetPosition(int newPosition)
    {
        Debug.Log($"[Streaming] OnAudioSetPosition => {newPosition}");
    }

    /// <summary>
    /// Utility: Only call Path.Combine if 'relativePath' is not null/empty. Otherwise, return "".
    /// </summary>
    private string BuildPath(string relativePath)
    {
        if (string.IsNullOrEmpty(relativePath))
        {
            return "";
        }
        if( Application.platform == RuntimePlatform.Android )
        {
            return Path.Combine(Application.persistentDataPath, relativePath);
        }
        else
            return Path.Combine(Application.streamingAssetsPath, relativePath);
    }

    private void OnDestroy()
    {
        // Cleanup TTS resources
        if (offlineTts != null)
        {
            offlineTts.Dispose();
            offlineTts = null;
        }
    }
}

now you need to fill the inspector values:

the important ones are the paths, make sure to provide the paths relative to the streamingAssets Folder. and make sure that those files actually exist.
this way you have a function unity android build!
drawback:

sadly the first time you open the app on android you will have to wait for a few seconds for the streamingAssetsApi to do its work, it's possible to simply cross-reference the files (see if they exist in the destination location) the second time you open the app which allow us to not repeat the waiting process.

if you have a different work around for the streaming assets issue, i'd be happy to hear it :)
Enjoy!

Originally posted by @adem-rguez in #1635 (comment)

[csukuangfj]

送你一张图

---

你介不介意分享下闪退的 log?

(请不要只回答这个问题。请先看截图，按截图做)

[newkitty]

感谢大佬回复。
1.我想在unity 安卓里面使用sherpa-onnx的语音合成功能。（项目在编辑器模式下运行是正常的，可以语音合成和语音识别，于是我想编译到安卓手机上）
2.参考了@adem-rguez in #1635 (comment) 里面的做法。
3.首先，第一次遇到的问题是如果只是放安卓so文件到插件目录下面，编译会报错：
Assets\Code\AI\ASR\vitsAsr.cs(1,7): error CS0246: The type or namespace name 'SherpaOnnx' could not be found (are you missing a using directive or an assembly reference?)
然后我让安卓也加载sherpa-onnx.dll后就可以编译了。
第二次遇到的问题是在进入安卓应用后会闪退，log日志是：

02-19 00:12:38.224 19015 19047 I Unity : BuildPath(Model)=/storage/emulated/0/Android/data/com.test.chattest/files/models/vits-zh-hf-theresa/theresa.onnx
02-19 00:12:38.226 19015 19047 I Unity : BuildPath(Lexicon)=/storage/emulated/0/Android/data/com.test.chattest/files/models/vits-zh-hf-theresa/lexicon.txt
02-19 00:12:38.227 19015 19047 I Unity : BuildPath(Tokens)=/storage/emulated/0/Android/data/com.test.chattest/files/models/vits-zh-hf-theresa/tokens.txt
02-19 00:12:38.230 19015 19047 I Unity : BuildPath(DictDir)=/storage/emulated/0/Android/data/com.test.chattest/files/models/vits-zh-hf-theresa/dict
02-19 00:12:38.247 19015 19181 D Unity : Failed to load native plugin: Unable to load library '/data/app/~~i4zDJb9ylCzS819v4aIu5g==/com.test.chattest-w8bviQjXK4HY1IRpikl85g==/lib/arm64/libsherpa-onnx-c-api.so', error 'java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.ClassLoader java.lang.Class.getClassLoader()' on a null object reference'
02-19 00:12:38.263 19015 19181 E CRASH : pid: 19015, tid: 19181, name: Thread-8 >>> com.test.chattest <<<
02-19 00:12:38.263 19015 19181 E CRASH : #1 pc 0000000000161e80 /data/app/~~i4zDJb9ylCzS819v4aIu5g==/com.test.chattest-w8bviQjXK4HY1IRpikl85g==/lib/arm64/libsherpa-onnx-c-api.so (BuildId: eb9ea74e9636cbf59d49a9578db4ecc81e932b30)
02-19 00:12:38.491 19185 19185 F DEBUG : pid: 19015, tid: 19181, name: Thread-8 >>> com.test.chattest <<<
02-19 00:12:38.492 19185 19185 F DEBUG : #1 pc 0000000000161e80 /data/app/~~i4zDJb9ylCzS819v4aIu5g==/com.test.chattest-w8bviQjXK4HY1IRpikl85g==/lib/arm64/libsherpa-onnx-c-api.so (BuildId: eb9ea74e9636cbf59d49a9578db4ecc81e932b30)
02-19 00:12:38.884 1799 19188 W ActivityTaskManager: finishTopCrashedActivityLocked Force finishing activity com.test.chattest/com.unity3d.player.UnityPlayerActivity
02-19 00:12:38.886 1799 19188 I WindowManager: updateFocusedWindowLocked: changed =false displayId=0 topFocusedDisplayId=-1 shouldSleep=false newFocus=Window{7e424fb u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity} calls=com.android.server.wm.WindowManagerService.updateFocusedWindowLocked:6594 com.android.server.wm.DisplayContent.layoutAndAssignWindowLayersIfNeeded:4052 com.android.server.wm.TaskDisplayArea.positionStackAt:874 com.android.server.wm.TaskDisplayArea.positionStackAt:915 com.android.server.wm.TaskDisplayArea.positionStackAtTop:883 com.android.server.wm.ActivityStack.moveToFront:1221 com.android.server.wm.ActivityRecord.moveFocusableActivityToTop:3122 com.android.server.wm.TaskDisplayArea.moveHomeActivityToTop:1840
02-19 00:12:38.889 1799 19188 W HwActivityTaskManagerServiceEx: appSwitch from: com.test.chattest to: com.huawei.android.launcher
02-19 00:12:38.889 1799 19188 V HwActivityTaskManagerServiceEx: takeTaskSnapShot package com.test.chattest
02-19 00:12:38.892 2380 18762 I AssistantService-11.2.1.200: registerHwActivityNotifier call fromPackage:com.test.chattest, toPackage:com.huawei.android.launcher
02-19 00:12:38.895 2421 3655 I ScenarioService: app focus change from: com.test.chattest to:com.huawei.android.launcher
02-19 00:12:38.896 2380 7001 D AssistantService-11.2.1.200: handleMessage app switch fromPackage:com.test.chattest, toPackage:com.huawei.android.launcher
02-19 00:12:38.897 1799 1831 I SWAP_SCENE: Entering notifySceneData notifySceneData. EventType: APP_SWITCH_FROM, App: com.test.chattest, uid: 10345, pid: 0
02-19 00:12:38.907 1799 19188 I WindowManager: Changing focus from Window{7e424fb u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity} to null displayId=0
02-19 00:12:38.910 1799 19188 W WindowManager: Failed to take screenshot ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}} mHadTakenSnapShot true
02-19 00:12:38.921 2180 15437 D TaskStackListener: destory snapshot binder pid 0 myPid 2180 snapshot TaskSnapshot{ mId=1739895158890 mTopActivityComponent=com.test.chattest/com.unity3d.player.UnityPlayerActivity mSnapshot=android.graphics.GraphicBuffer@c658ba3 (2388x1080) mColorSpace=sRGB IEC61966-2.1 (id=0, model=RGB) mOrientation=2 mRotation=1 mTaskSize=Point(2388, 1080) mContentInsets=[0,0][0,0] mIsLowResolution=false mIsRealSnapshot=true mWindowingMode=1 mSystemUiVisibility=5888 mIsTranslucent=true
02-19 00:12:38.921 1799 1821 I HwWindowManagerServiceEx: com.test.chattest/com.unity3d.player.UnityPlayerActivity blur level: -1
02-19 00:12:38.921 1799 1821 I HwWindowManagerServiceEx: com.test.chattest blur level: -1
02-19 00:12:38.923 1799 1819 W ActivityManager: Skipping native crash dialog of ProcessRecord{ed63479 19015:com.test.chattest/u0a345}
02-19 00:12:38.998 2421 3833 I PG_ash : com.test.chattest become invisible
02-19 00:12:39.141 1799 1969 W InputDispatcher: channel '7e424fb com.test.chattest/com.unity3d.player.UnityPlayerActivity (server)' ~ Consumer closed input channel or an error occurred. events=0xd
02-19 00:12:39.141 1799 1969 E InputDispatcher: channel '7e424fb com.test.chattest/com.unity3d.player.UnityPlayerActivity (server)' ~ Channel is unrecoverably broken and will be disposed!
02-19 00:12:39.142 1799 1969 I WindowManager: WINDOW DIED Window{7e424fb u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity}
02-19 00:12:39.143 1799 1969 I WindowManager: removeIfPossible: Window{7e424fb u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity} callers=com.android.server.wm.WindowState.removeIfPossible:2946 com.android.server.wm.InputManagerCallback.notifyInputChannelBroken:110 com.android.server.input.InputManagerService.notifyInputChannelBroken:2329
02-19 00:12:39.143 1799 1969 W InputDispatcher: Attempted to unregister already unregistered input channel '7e424fb com.test.chattest/com.unity3d.player.UnityPlayerActivity (server)'
02-19 00:12:39.143 1799 1969 W WindowManager: Keyguard is occluded and there is no window in ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}}
02-19 00:12:39.145 1799 2403 I ActivityManager: Process com.test.chattest (pid 19015) has died: fg TOP
02-19 00:12:39.145 1799 2403 D ActivityManager: cleanUpApplicationRecord app: 19015:com.test.chattest/u0a345, bad: false, restarting: false, allowRestart: true
02-19 00:12:39.148 1799 1969 I WindowManager: postWindowRemoveCleanupLocked: Window{7e424fb u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity}
02-19 00:12:39.148 1799 1969 I WindowManager: Removing Window{7e424fb u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity} from ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}}
02-19 00:12:39.169 2380 3269 I UctpAppMonitor: handleAppDied: pkgName is com.test.chattest mIsNeedToOpen is false
02-19 00:12:39.174 1077 2706 D ITouchService: itouch currentPackageName = com.test.chattest
02-19 00:12:39.174 1077 2706 D ITouchService: itouch:app com.test.chattest is not game app
02-19 00:12:39.174 1077 2706 D ITouchService: itouch:app com.test.chattest is not in StylusPenList
02-19 00:12:39.187 1799 2403 I WindowManager: removeWindowToken: displayid:0 binder:Token{f7c9f35 ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}}} token:ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}}
02-19 00:12:39.187 1799 2403 V WindowManager: commitVisibility: ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}}: visible=false mVisibleRequested=false
02-19 00:12:39.188 2421 3655 I ScenarioService: app is visible from system: com.test.chattest
02-19 00:12:39.192 1799 2403 I WindowManager: removeWindowToken: displayid:0 binder:Token{f7c9f35 ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}}} token:null
02-19 00:12:39.193 1799 2403 I WindowManager: removeChild: child=ActivityRecord{8901fca u0 com.test.chattest/com.unity3d.player.UnityPlayerActivity t14947 f}} reason=removeChild
02-19 00:12:39.195 1799 2403 I ActivityTaskManager: removeStack: detaching Task{f49453b #14947 visible=false type=standard mode=fullscreen translucent=true A=10345:com.test.chattest U=0 StackId=14947 sz=0} from displayId=0
02-19 00:12:39.200 2421 4125 I PGServer: report state:10011 event type:2 pid:0 uid:0 pkg:com.test.chattest to pid: 2421
02-19 00:12:39.200 2421 3655 I AppsUsage: scnOff: false FgAPP: com.huawei.android.launcher BgAPP: com.test.chattest
02-19 00:12:39.200 2421 4125 I SceneReceiver: state type: 10011 eventType:2 pid:0 uid:0 pkg:com.test.chattest
02-19 00:12:39.200 2421 4125 D ApsPgPlug: APK: onStateChanged stateType: 10011, pkg:com.test.chattest
02-19 00:12:39.200 2421 4125 I PGServer: report state:10011 event type:2 pid:0 uid:0 pkg:com.test.chattest to pid: 2380
02-19 00:12:39.200 2421 4125 I PGServer: report state:10011 event type:2 pid:0 uid:0 pkg:com.test.chattest to pid: 1799
02-19 00:12:39.233 2380 2653 E DollieAdapterService: notifyActivityState pkg:com.test.chattest/com.unity3d.player.UnityPlayerActivity state:19 fg:false mUid:10345
02-19 00:12:39.233 2380 2653 I NetPredictEngine: Parent: active app change, old packageName: com.huawei.android.launcher, new packageName: com.test.chattest
02-19 00:12:39.235 2380 2653 I SmartDualCardConfig: getAppType,pkgName=com.test.chattest,appType=-1
02-19 00:12:39.236 2380 2653 I SmartDualCardConfig: getAppType,pkgName=com.test.chattest,appType=-1
02-19 00:12:39.236 2380 2653 I SmartDualCardSM: onUidStateUpdate,uid=10345,appName=com.test.chattest,state=BACKGROUND
02-19 00:12:39.237 2380 2653 I SmartDualCardConfig: getAppType,pkgName=com.test.chattest,appType=-1
02-19 00:12:39.238 2380 2653 I DollieChr: handleForegroundAppUpdateMsg,mAppState=2,scenes=1,mForegroundAppName=com.test.chattest
02-19 00:12:39.240 2380 2653 I SmartDualCardConfig: getAppType,pkgName=com.test.chattest,appType=-1
02-19 00:12:39.311 2380 2653 I NetPredictEngine: Parent: active app change, old packageName: com.test.chattest, new packageName: com.huawei.android.launcher

看着感觉像是加载插件libsherpa-onnx-c-api.so失败了，搜索了网上，然后以为是路径存放不对，又换了几种存放路径，还是加载失败。现在的插件路径存放是：

项目设置：

也添加了目录权限：

<?xml version="1.0" encoding="utf-8"?>
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools">
	<uses-permission android:name="android.permission.INTERNET"/>
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <application>
        <activity android:name="com.unity3d.player.UnityPlayerActivity"
                  android:theme="@style/UnityThemeSelector"
				   android:usesCleartextTraffic="true"
					android:requestLegacyExternalStorage="true"
				  >
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <meta-data android:name="unityplayer.UnityActivity" android:value="true" />
        </activity>
    </application>
</manifest>

麻烦帮我看看，请问还有什么地方需要我提供的？非常感谢！

[adem-rguez]

hello @newkitty i tried translating what you said (i don't speak chinese), i hope i understood the problem correctly.
could you share the code?

[newkitty]

Okay, thank you for your attention, respected experts.
I have uploaded my project here:https://github.com/newkitty/usherpa-onnx-tts-unityandroid.git
Please take a look and I would be very grateful. @adem-rguez @csukuangfj

[mtsangcl102]

@newkitty I did a few changes based on your project and its works on Android mobile

1. Find out that in the TTS.cs, that should be BuildPath(modelsDir), when calling CopyDirectory in Android case which as the same as adem-rguez comment

2. I copy all android api lib into plugins, currently like

3. I replaced StreamingAssetsAPI.cs as adem-rguez version

Thanks for your project, it help me a lot, hope above suggestion help you also

[mtsangcl102]

@newkitty I did a few changes based on your project and its works on Android mobile

1. Find out that in the TTS.cs, that should be BuildPath(modelsDir), when calling CopyDirectory in Android case which as the same as adem-rguez comment
2. I copy all android api lib into plugins, currently like

3. I replaced StreamingAssetsAPI.cs as adem-rguez version

Thanks for your project, it help me a lot, hope above suggestion help you also

Also I updated the packages ( .dll or others ) from nuget, since I am using MacOS, I have to install osx package

[newkitty]

@mtsangcl102
You guys are all amazing! thank you very much for your reply.

1. I am a bit confused about the difference between using 'Path.Combine(Application.persistentDataPath, relativePath)' inside 'CopyDirectory' and using 'BuildPath' inside 'this.RunCoroutine(StreamingAssetsAPI.CopyDirectory'.

Additionally, I would like to ask which version of sherpa-onnx you are using? I often experience crashes with the latest version.
3.
Can I invite you to be a collaborator on my project? To help update to the correct version and assist more people in need.
https://github.com/newkitty/usherpa-onnx-tts-unityandroid/invitations
Thank you again!

[mtsangcl102]

@mtsangcl102 You guys are all amazing! thank you very much for your reply.

1. I am a bit confused about the difference between using 'Path.Combine(Application.persistentDataPath, relativePath)' inside 'CopyDirectory' and using 'BuildPath' inside 'this.RunCoroutine(StreamingAssetsAPI.CopyDirectory'.

Additionally, I would like to ask which version of sherpa-onnx you are using? I often experience crashes with the latest version. 3. Can I invite you to be a collaborator on my project? To help update to the correct version and assist more people in need. https://github.com/newkitty/usherpa-onnx-tts-unityandroid/invitations Thank you again!

@newkitty that copy directory used for copy the tts models from streamAsset to a readable destination as I understand, in your code, it is doing copy from A folder to A folder

I do not sure if update to latest version is necessary, since I am using MacOS so I have to install lastest version to confirm editor works, I believe that you are not have to do that

Thanks for the invitation but I think I could not put much time into it, but i love to share my experience to help your project done

[newkitty]

Okey,@mtsangcl102 1.Take a look at the code here please, This is how I wrote it:

2.And what is the exact version of sherpa-onnx that you are using? I would like to try using your version.

3.Thank you for your help. Since I'm not very skilled in coding, I will try my best.

[mtsangcl102]

@newkitty
first of all, version on sherpa-onnx is 1.10.45 which I am using

back to the path, checked the code you uploaded, I think that is okay, but I got an error about read-only directory, I did not find out the reason, may be not about the code

so my suggestion is keep AndroidManifest and mainTemplate.gradle in Plugins/Android first,
then clean other files and copy android libraries I mentioned above