Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* Made truncation parameter automatically processed Signed-off-by: Yerzhaisang Taskali <tasqali1697@gmail.com> * Made max_length parameter dynamic Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Added unit test for checking truncation parameter Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Updated CHANGELOG.md Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Included the test of max_length parameter value Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Slightly modeified the test of max_length parameter value Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Modified CHANGELOG.md and removed the duplicate Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Enabled auto-truncation format also for ONNX Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Implemented reusable function Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Fixed the lint Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Change tokenizer.json only if truncation is null Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Removed function which had been accidentally added Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Renamed reusable function and added the description Signed-off-by: yerzhaisang <tasqali1697@gmail.com> * Fixed the lint Signed-off-by: yerzhaisang <tasqali1697@gmail.com> --------- Signed-off-by: Yerzhaisang Taskali <tasqali1697@gmail.com> Signed-off-by: yerzhaisang <tasqali1697@gmail.com> (cherry picked from commit e0d1750) Co-authored-by: Yerzhaisang <55043014+Yerzhaisang@users.noreply.github.com>
- Loading branch information