update (#32)

[chenzhehuai]

• [build] Allow configure script to handle package-based OpenBLAS (Update configure to handle rpm based openblas kaldi-asr/kaldi#2618)

• [egs] updating local/make_voxceleb1.pl so that it works with newer versions of VoxCeleb1 ([egs] egs/voxceleb1/ data preparation fix kaldi-asr/kaldi#2684)

• [egs,scripts] Remove unused --nj option from some scripts ([egs,scripts] Remove unused --nj option from some scripts kaldi-asr/kaldi#2679)

• [egs] Fix to tedlium v3 run.sh (rnnlm rescoring) (Fix to tedlium v3 run.sh (rnnlm rescoring) kaldi-asr/kaldi#2686)

• [scripts,egs] Tamil OCR with training data from yomdle and testing data from slam (Tamil ocr with training data for yomdle and testing data from slam kaldi-asr/kaldi#2621)

note: this data may not be publicly available at the moment. we'll work on that.

• [egs] mini_librispeech: allow relative pathnames in download_and_untar.sh (fixed path error in download_and_untar kaldi-asr/kaldi#2689)

• [egs] Updating SITW recipe to account for changes to VoxCeleb1 ([egs] Updating SITW recipe to account for changes to VoxCeleb1  kaldi-asr/kaldi#2690)

• [src] Fix nnet1 proj-lstm bug where gradient clipping not used; thx:@cbtpkzm ([src] Fix nnet1 proj-lstm bug where gradient clipping not used; thx:@cbtpkzm kaldi-asr/kaldi#2696)

• [egs] Update aishell2 recipe to allow online decoding (no pitch for ivector) (Update aishell2 ivector for online kaldi-asr/kaldi#2698)

• [src] Make cublas and cusparse use per-thread streams. ([src] Make cublas and cusparse use per-thread streams. kaldi-asr/kaldi#2692)

This will reduce synchronization overhead when we actually use multiple
cuda devices in one process go down drastically, since we no longer
synchronize on the legacy default stream.

More details here: https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html

• [src] improve handling of low-rank covariance in ivector-compute-lda ([src] improve error handling in ivector-compute-lda kaldi-asr/kaldi#2693)

• [egs] Changes to IAM handwriting-recognition recipe, including BPE encoding (Iam kaldi-asr/kaldi#2658)

• [scripts] Make sure pitch is not included in i-vector feats, in online decoding preparation (Add dim check for ivector in online decoding preparation kaldi-asr/kaldi#2699)

• [src] fix help message in post-to-smat ([src] fix help message in post-to-smat kaldi-asr/kaldi#2703)

• [scripts] Fix to steps/cleanup/debug_lexicon.sh ([scripts] Fix to steps/cleanup/debug_lexicon.sh kaldi-asr/kaldi#2704)

• [egs] Cosmetic and file-mode fixes in HKUST recipe ([egs] Cosmetic and file-mode fixes in HKUST recipe kaldi-asr/kaldi#2708)

• [scripts] nnet1: remove the log-print of args in 'make_nnet_proto.py', thx:mythilisharan@gmail.com (nnet1: remove the log-print of args in 'make_nnet_proto.py', kaldi-asr/kaldi#2706)

• [egs] update README in AISHELL-2 (update README in AISHELL-2 kaldi-asr/kaldi#2710)

• [src] Make constructor of CuDevice private ([src] Fix Singleton implementation of CuDevice. kaldi-asr/kaldi#2711)

• [egs] fix sorting issue in aishell v1 (fix sort issue in aishell v1 kaldi-asr/kaldi#2705)

• [egs] Add soft links for CNN+TDNN scripts ([egs] Add soft links for CNN+TDNN scripts kaldi-asr/kaldi#2715)

• [build] Add missing packages in extras/check_dependencies.sh (Add missing packages in extras/check_dependencies.sh kaldi-asr/kaldi#2719)

• [egs] madcat arabic: clean scripts, tuning, use 6-gram LM (madcat arabic scripts cleaning and adding higher order language model kaldi-asr/kaldi#2718)

• [egs] Update WSJ run.sh: comment out outdated things, add run_tdnn.sh. ([egs] Update WSJ run.sh, commenting out outdated things, adding run_tdnn.sh kaldi-asr/kaldi#2723)

• [scripts,src] Fix potential issue in scripts; minor fixes. ([scripts,src] Fix encoding issue in scripts; minor fixes. kaldi-asr/kaldi#2724)

The use of split() in latin-1 encoding (which might be used for other ASCII-compatible encoded data like utf-8) is not right because character 160 (expressed here in decimal) is a NBSP in latin-8 encoding and is also in the range UTF-8 uses for encoding. The same goes for strip(). Thanks @ChunChiehChang for finding the issue.

• [egs] add example script for RNNLM lattice rescoring for WSJ recipe (add example script for RNNLM lattice rescoring for WSJ recipe kaldi-asr/kaldi#2727)

• [egs] add rnnlm example on tedlium+lm1b; add rnnlm rescoring results (add rnnlm script on tedlium+lm1b; add rnnlm rescoring results kaldi-asr/kaldi#2248)

• [scripts] Small fix to utils/data/convert_data_dir_to_whole.sh (RE backups) (Backup in/utils/data/conver_data_dir_tw_whole.sh if files exist. kaldi-asr/kaldi#2735)

• [src] fix memory bug in kaldi::~LatticeFasterDecoderTpl(), (fixing memory bug in kaldi::~LatticeFasterDecoderTpl(), kaldi-asr/kaldi#2737)

• found it when running 'latgen-faster-mapped-parallel',
• core-dumps from the line: decoder/lattice-faster-decoder.cc:52
  -- the line is doing 'delete &(FST*)', i.e. deleting the pointer to FST, instead of deleting the FST itslef,
  -- bug was probably introduced by refactoring commit d0c68a6 from 2018-09-01,
  -- after the change the code runs fine... (the unit tests for src/decoder are missing)

• [egs] Remove per-utt option from nnet3/align scripts (Remove per-utt option from nnet3/align scripts kaldi-asr/kaldi#2717)

• [egs] Small Librispeech example fix, thanks: Yasasa Tennakoon. ([egs] Small Librispeech example fix, thanks: Yasasa Tennakoon. kaldi-asr/kaldi#2738)

• [egs] Aishell2 recipe: turn off jieba's new word discovery in word segmentation ([egs][aishell2] turn off new word discovery in word segmentation stage kaldi-asr/kaldi#2740)

• [egs] Add missing file local/join_suffix.py in TEDLIUM s5_r3; thx:anand@sayint.ai (fixed missing local/join_suffix.py in TEDLIUM v3 (s5_r3) kaldi-asr/kaldi#2741)

• [egs,scripts] Add Tunisian Arabic (MSA) recipe; cosmetic fixes to pbs.pl (resolved conflicts for #2722 kaldi-asr/kaldi#2725)

• [scripts] Fix missing import in utils/langs/grammar/augment_words_txt.py ([scripts] Missing import in utils/langs/grammar/augment_words_txt.py kaldi-asr/kaldi#2742)

• [scripts] Fix build_const_arpa_lm.sh w.r.t. where appears inside words (Minor change to handle cases where <s> is appears in text kaldi-asr/kaldi#2745)

• [scripts] Slight improvements to decode_score_fusion.sh usability (Assign frame_subsampling_factor automatically kaldi-asr/kaldi#2746)

• [build] update configure to support cuda 10 (update configure to support cuda 10 kaldi-asr/kaldi#2747)

• [scripts] Fix bug in utils/data/resample_data_dir.sh (Fix minor bug in utils/data/resample_data_dir.sh kaldi-asr/kaldi#2749)

• [scripts] Fix bug in cleanup after steps/cleanup/clean_and_segment_data*.sh (Bug fixed in egs/wsj/s5/steps/cleanup/clean_and_segment_data.sh kaldi-asr/kaldi#2750)

• [egs] several updates of the tunisian_msa recipe (several updates of the tunisian_msa recipe  kaldi-asr/kaldi#2752)

• [egs] Small fix to Tunisian MSA TDNN script (RE train_stage) ([egs] Reset train_stage in tdnn script Tunisian_MSA kaldi-asr/kaldi#2757)

• [src,scripts] Batched nnet3 computation ([src] Nnet batch computation kaldi-asr/kaldi#2726)

This PR adds the underlying utilities for much faster nnet3 inference on GPU, and a command-line binary (and script support) for nnet3 decoding and posterior computation. TBD: a binary for x-vector computation. This PR also contains unrelated decoder speedups (skipping range checks for transition ids... this may cause segfaults when graphs are mismatched).

• [build] Add python3 compatibility to install scripts (Use get_python_inc and print function in install scripts kaldi-asr/kaldi#2748)

• [scripts] tfrnnlm: Modify TensorFlow flag format for compatibility with recent versions (Made flag changes in some files kaldi-asr/kaldi#2760)

• [egs] fix old style perl regex in egs/chime1/s5/local/chime1_prepare_data.sh ([egs] fix old style perl regex in egs/chime1/s5/local/chime1_prepare_… kaldi-asr/kaldi#2762)

• [scripts] Fix bug in steps/cleanup/debug_lexicon.sh (fixed a bug in steps/cleanup/debug_lexicon.sh kaldi-asr/kaldi#2763)

• [egs] Add example for Yomdle Farsi OCR (Adding yomdle_fa egs kaldi-asr/kaldi#2702)

• [scripts] debug_lexicon.sh: Fix bug introduced in fixed a bug in steps/cleanup/debug_lexicon.sh kaldi-asr/kaldi#2763. ([scripts] debug_lexicon.sh: Fix bug introduced in #2763. kaldi-asr/kaldi#2764)

• [egs] add missing online cmvn config in aishell2 ([egs][aishell2] fix: add missing online cmvn config kaldi-asr/kaldi#2767)

• [egs] Add CNN-TDNN-F script for Librispeech (CNN-TDNN-F script for Librispeech kaldi-asr/kaldi#2744)

• [src] Some minor cleanup/fixes regarding CUDA memory allocation; other small fixes. ([src,egs] Some minor cleanup/fixes regarding CUDA memory allocation (etc.) kaldi-asr/kaldi#2768)

• [scripts] Update reverberate_data_dir.py so that it works with python3 (Update to reverberate_data_dir.py so that it works with python3 kaldi-asr/kaldi#2771)

• [egs] Chime5: fix total number of words for WER calculation (Total number of words for WER calculation was incorrect. kaldi-asr/kaldi#2772)

• [egs] RNNLMs on Tedlium w/ Google 1Bword: Increase epochs, update results (update rnnlm results on tedlium with one billion words dataset kaldi-asr/kaldi#2775)

• [scripts,egs] Added phonetisaurus-based g2p scripts (Added phonetisaurus-based g2p scripts kaldi-asr/kaldi#2730)

Phonetisaurus is much faster to train then sequitur.

• [egs] madcat arabic: clean scripts, tuning, rescoring, text localization (madcat arabic: clean scripts, tuning, rescoring, text localization kaldi-asr/kaldi#2716)

• [scripts] Enhancements & minor bugfix to segmentation postprocessing (Enhancements & minor bugfix to segmentation postprocessing kaldi-asr/kaldi#2776)

• [src] Update gmm-decode-simple to accept ConstFst (changed to read correct Fst type kaldi-asr/kaldi#2787)

• [scripts] Update documentation of train_raw_dnn.py (Add documentation for target matrices kaldi-asr/kaldi#2785)

• [src] nnet3: extend what descriptors can be parsed. ([src] nnet3: extend what descriptors can be parsed. kaldi-asr/kaldi#2780)

• [src] Small fix to 'fstrand' (make sure args are parsed) (added missing po.Read kaldi-asr/kaldi#2777)

• [src,scripts] Minor, mostly cosmetic updates ([src,scripts] Minor, mostly cosmetic updates kaldi-asr/kaldi#2788)

• [src,scripts] Add script to compare alignment directories. ([src,scripts] Add script to compare alignment directories. kaldi-asr/kaldi#2765)

• [scripts] Small fixes to script usage messages, etc. (cosmetic usage and warning in scripts kaldi-asr/kaldi#2789)

• [egs] Update ami_download.sh after changes on Edinburgh website. (Update ami_download.sh kaldi-asr/kaldi#2769)

• [scripts] Update compare_alignments.sh to allow different lang dirs. ([scripts] Update compare_alignments.sh to allow different lang dirs. kaldi-asr/kaldi#2792)

• [scripts] Change make_rttm.py so output is in determinstic order (deterministic results for make_rttm kaldi-asr/kaldi#2794)

• [egs] Fixes to yomdle_zh RE encoding direction, etc. (Small bugfix for yomdle_zh kaldi-asr/kaldi#2791)

• [src] Add support for context independent phones in gmm-init-biphone (for e2e) (Add support for context independent phones in gmm-init-biphone (for e2e) kaldi-asr/kaldi#2779)

• [egs] Simplifying multi-condition version of AMI recipe (Simplifying AMI recipe kaldi-asr/kaldi#2800)

• [build] Fix openblas build for aarch64 (Fix openblas build for aarch64 kaldi-asr/kaldi#2806)

• [build] Make CUDA_ARCH configurable at configure-script level (Add build configure option for cuda_arch kaldi-asr/kaldi#2807)

• [src] Print maximum memory stats in CUDA allocator (maximum memory stats kaldi-asr/kaldi#2799)

• [src,scripts] Various minor code cleanups (reformat some code kaldi-asr/kaldi#2809)

• [scripts] Fix handling of UTF-8 in filenames, in wer_per_spk_details.pl (Update wer_per_spk_details.pl kaldi-asr/kaldi#2811)

• [egs] Update AMI chain recipes (New AMI recipes kaldi-asr/kaldi#2817)

• [egs] Improvements to multi_en tdnn-opgru/lstm recipes ( Improvements to multi_en tdnn-opgru/lstm recipes kaldi-asr/kaldi#2824)

• [scripts] Fix initial prob of silence when lexicon has silprobs. Thx:@agurianov ([scripts] Fix initial-prob of sil when lexicon has silprobs. Thx:@agurianov kaldi-asr/kaldi#2823)

• [scripts,src] Fix to multitask nnet3 training (TypeError in wsj/s5/steps/libs/nnet3/train kaldi-asr/kaldi#2818); cosmetic code change. ([scripts,src] Fix to multitask nnet3 training (#2818); cosmetic code change. kaldi-asr/kaldi#2827)

• [scripts] Create shared versions of get_ctm_conf.sh, add get_ctm_conf_fast.sh ( move get_ctm_conf.sh from aspire to steps/conf/ kaldi-asr/kaldi#2828)

• [src] Use cuda streams in matrix library ([src] Use cuda streams in matrix library kaldi-asr/kaldi#2821)

• [egs] Add online-decoding recipe to aishell1 (Add online recipe to aishell1 kaldi-asr/kaldi#2829)

• [egs] Add DIHARD 2018 diarization recipe. ((WIP) Adding DIHARD 2018 recipe. kaldi-asr/kaldi#2822)

• [egs] add nnet3 online result for aishell1 (add nnet3 online result kaldi-asr/kaldi#2836)

• [scripts] RNNLM scripts: don't die when features.txt is not present ([scripts] prepare_rnn_lm.sh now treats features.txt as optional kaldi-asr/kaldi#2837)

• [src] Optimize cuda allocator for multi-threaded case ([src] Add optimization to cuda allocator kaldi-asr/kaldi#2820)

• [build] Add cub library for cuda projects ([build] Add cub library for cuda projects kaldi-asr/kaldi#2819)

not needed now but will be in future.

• [src] Make Cuda allocator statistics visible to program (allocator statistics kaldi-asr/kaldi#2835)

• [src] Fix bug affecting scale in GeneralDropoutComponent (non-continuous case) ([src] Fixing the scaling bug of the general-dropout component kaldi-asr/kaldi#2815)

• [build] FIX "configure --use-cuda=no" option is not respected kaldi-asr/kaldi#2842: properly check $use_cuda against false. (FIX #2842: properly check $use_cuda against false. kaldi-asr/kaldi#2843)

• [doc] Add note about OOVs to data-prep. ([doc] Add note about OOVs to data-prep. kaldi-asr/kaldi#2844)

• [scripts] Allow segmentation with nnet3 chain models (Allow segmentation with nnet3 chain models kaldi-asr/kaldi#2845)

• [build] Remove -lcuda from cuda makefiles which breaks operation when no driver present ([build] Remove -lcuda from cuda makefiles which breaks operation w/o driver kaldi-asr/kaldi#2851)

• [scripts] Fix error in analyze_lats.sh for long lattices (replace awk with perl) (Fix error while analyzing lattice kaldi-asr/kaldi#2854)

• [egs] add rnnlm recipe for librispeech (add rnnlm recipe for librispeech kaldi-asr/kaldi#2830)

• [build] change configure version from 9 to 10 (configure version should increment kaldi-asr/kaldi#2853) (config change kaldi-asr/kaldi#2855)

• [src] fixed compilation errors when built with --DOUBLE_PRECISION=1 (fixed errors when built with --DOUBLE_PRECISION=1 kaldi-asr/kaldi#2856)

• [build] Clarify instructions if cub is not found ([build] Clarify instructions if cub is not found kaldi-asr/kaldi#2858)

• [egs] Limit MFCC feature extraction job number in Dihard recipe (Limit MFCC feature extraction job number in Dihard recipe kaldi-asr/kaldi#2865)

• [egs] Added Bentham handwriting recognition recipe (Added Bentham handwriting recognition recipe kaldi-asr/kaldi#2846)

• [src] Share roots of different tones of phones aishell (WIP: Cluster nonsilence phones for aishell kaldi-asr/kaldi#2859)

• [egs] Fix path to sequitur in commonvoice egs (update sequitur path kaldi-asr/kaldi#2868)

• [egs] Update reverb recipe (Reverb kaldi-asr/kaldi#2753)

• [scripts] Fix error while analyzing lattice (parsing bugs) (Fix error while analyzing lattice (parsing bugs) kaldi-asr/kaldi#2873)

• [src] Fix memory leak in OnlineCacheFeature; thanks @Worldexe ([src] Fix memory leak in OnlineCacheFeature; thanks @Worldexe kaldi-asr/kaldi#2872)

• [egs] TIMIT: fix mac compatibility of sed command (fix for sed substitute i error in mac kaldi-asr/kaldi#2874)

• [egs] mini_librispeech: fixing some bugs and limiting repeated downloads (mini_librispeech: fixing some bugs and limiting the repeated downloads kaldi-asr/kaldi#2861)

• [src,scripts,egs] Speedups to GRU-based networks (special components) (Add GruNonlinearityComponent(by Dan) and OutputGruNonlinearityCompone… kaldi-asr/kaldi#2712)

• [src] Fix infinite recursion with -DDOUBLE_PRECISION=1. Thx: @hwiorn (nnet3-info occurs a segment fault. kaldi-asr/kaldi#2875) (error fixed for issue #2875. Thanks to @hwiorn kaldi-asr/kaldi#2876)

• Revert "[src] Fix infinite recursion with -DDOUBLE_PRECISION=1. Thx: @hwiorn (nnet3-info occurs a segment fault. kaldi-asr/kaldi#2875) (error fixed for issue #2875. Thanks to @hwiorn kaldi-asr/kaldi#2876)" (Revert "error fixed for issue #2875. Thanks to @hwiorn" kaldi-asr/kaldi#2877)

This reverts commit 84435ff.

• Revert "Revert "[src] Fix infinite recursion with -DDOUBLE_PRECISION=1. Thx: @hwiorn (nnet3-info occurs a segment fault. kaldi-asr/kaldi#2875) (error fixed for issue #2875. Thanks to @hwiorn kaldi-asr/kaldi#2876)" (Revert "error fixed for issue #2875. Thanks to @hwiorn" kaldi-asr/kaldi#2877)" (Revert "Revert "error fixed for issue #2875. Thanks to @hwiorn"" kaldi-asr/kaldi#2878)

This reverts commit b196b7f.

• Revert "[src] Fix memory leak in OnlineCacheFeature; thanks @Worldexe" (Revert "[src] Fix memory leak in OnlineCacheFeature; thanks @Worldexe" kaldi-asr/kaldi#2882)

the fix was buggy. apologies.

• [src] Remove unused code that caused Windows compile failure. Thx:@btiplitz ([src] Remove unused code that caused Windows compile failure. Thx:@btiplitz kaldi-asr/kaldi#2881)

• [src] Really fix memory leak in online decoding; thx:@Worldexe ([src] REALLY fix memory leak in online decoding; thx:@worldexe kaldi-asr/kaldi#2883)

• [src] Fix Windows cuda build failure (use C++11 standard include) (windows cuda build failure kaldi-asr/kaldi#2880)

• [src] Add #include that caused build failure on Windows (Windows stlfailure kaldi-asr/kaldi#2886)

• [scripts] Fix max duration check in sad_to_segments.py (Fix max duration check in sad_to_segments.py kaldi-asr/kaldi#2889)

• [scripts] Fix speech duration calculation in sad_to_segments.py (Fix speech duration calculation in sad_to_segments.py kaldi-asr/kaldi#2891)

• [src] Fix Windows build problem (timer.h) (fixed: In timer.cc:81, the destructor for Profiler fails as this does… kaldi-asr/kaldi#2888)

• [egs] add HUB4 spanish tdnn-f and cnn-tdnn script ([egs] Add HUB4 Spanish tdnn-f and cnn-tdnn-f scripts kaldi-asr/kaldi#2895)

• [egs] Fix Aishell2 dict prepare bug; should not affect results (Fix Aishell2 dict prepare bug kaldi-asr/kaldi#2890)

• [egs] Self-contained example for KWS for mini_librispeech (self-contained example for KWS for minilibrispeech kaldi-asr/kaldi#2887)

• [egs,scripts] Fix bugs in Dihard 2018 (WIP Fix bugs in Dihard 2018 kaldi-asr/kaldi#2897)

• [scripts] Check last character of files to match with newline (Check last character of files to match with newline kaldi-asr/kaldi#2898)

• [egs] Update Librispeech RNNLM results; use correct training data (update rescoring results on librispeech kaldi-asr/kaldi#2900)

• [scripts] RNNLM: old iteration model cleanup; save space ([RNNLM] old iteration model cleanup kaldi-asr/kaldi#2885)

• [scripts] Make prepare_lang.sh cleanup beforehand (prevents certain failures) ([scripts] Make prepare_lang.sh clean up beforehand (prevents certain failures) kaldi-asr/kaldi#2906)

• [scripts] Expose dim-range-node at xconfig level (Add dimension split for xconfig kaldi-asr/kaldi#2903)

• [scripts] Fix bug related to multi-task in train_raw_rnn.py ([scripts] Fix bug related to multi-task in train_raw_rnn.py. Thx:tessfu2001@gmail.com kaldi-asr/kaldi#2907)

[scripts] Fix bug related to multi-task in train_raw_rnn.py. Thx:tessfu2001@gmail.com

• [scripts] Cosmetic fix/clarification to utils/prepare_lang.sh ([scripts] Cosmetic fix/clarification to utils/prepare_lang.sh (relates to #2906) kaldi-asr/kaldi#2912)

• [scripts,egs] Added a new lexicon learning (adaptation) recipe for tedlium, in accordance with the IS17 paper. (Added a new lexicon learning (adaptation) recipe for tedlium, in accordance with the IS17 paper. kaldi-asr/kaldi#2774)

• [egs] TDNN+LSTM example scripts, with RNNLM, for Librispeech ([WIP, scripts] Recurrent neural network results for Librispeech kaldi-asr/kaldi#2857)

• [src] cosmetic fix in nnet1 code (fix a typo in struct NnetTrainOptions kaldi-asr/kaldi#2921)

• [src] Fix incorrect invocation of mutex in nnet-batch-compute code ([src] Fix incorrect invocation of mutex in nnet-batch-compute code kaldi-asr/kaldi#2932)

• [egs,minor] Fix typo in comment in voxceleb script (Fix typo kaldi-asr/kaldi#2926)

• [src,egs] Mostly cosmetic changes; add some missing includes (Yzmyyff indirect includes kaldi-asr/kaldi#2936)

• [egs] Fix path of rescoring binaries used in tfrnnlm scripts (rectified rescoring binaries called in scripts kaldi-asr/kaldi#2941)

• [src] Fix bug in nnet3-latgen-faster-batch for determinize=false ([src] Fix nnet3-latgen-faster-batch for determinize=false; thx:Maxim Korenevsky. kaldi-asr/kaldi#2945)

thx: Maxim Korenevsky.

• [egs] Add example for rimes handwriting database; Madcat arabic script cleanup (rimes handwriting database  kaldi-asr/kaldi#2935)

• [egs] Add scripts for yomdle korean (Yomdle Korean OCR setup kaldi-asr/kaldi#2942)

• [build] Refactor/cleanup build system, easier build on ubuntu 18.04. ([build] Refactor/cleanup build system, easier build on ubuntu 18.04. kaldi-asr/kaldi#2947)

note: if this breaks someone's build we'll have to debug it then.

• [scripts,egs] Changes for Python 2/3 compatibility (Python 2/3 compatibility kaldi-asr/kaldi#2925)

• [egs] Add more modern DNN recipe for fisher_callhome_spanish (Spanish dnn rnn recipe kaldi-asr/kaldi#2951)

• [scripts] switch from bc to perl to reduce dependencies (diarization scripts) (switched from bc to perl for overlap duration computation (issue #2954) kaldi-asr/kaldi#2956)

• [scripts] Further fix for Python 2/3 compatibility (Python 2/3 compatibility  kaldi-asr/kaldi#2957)

• [egs] Remove no-longer-existing option in tedlium_r3 recipe (fixed: Tedlium R3 failed in run_ivector_common.sh kaldi-asr/kaldi#2959)

• [build] Handle dependencies for .cu files in addition to .cc files (Handle dependencies for .cu files in addition to .cc files kaldi-asr/kaldi#2944)

• [src] remove duplicate test mode option from class GeneralDropoutComponent (Remove duplicate test mode option from class GeneralDropoutComponent kaldi-asr/kaldi#2960)

• [egs] Fix minor bugs in WSJ's flat-start/e2e recipe (Fix minor bugs in WSJ's flat-start chain recipe kaldi-asr/kaldi#2968)

• [egs] Fix to BSD compatibility of TIMIT data prep ([egs] Fix to BSD compatibility of TIMIT data prep kaldi-asr/kaldi#2966)

• [scripts] Fix RNNLM training script problem (chunk_length was ignored) (solve the rnnlm script problem where chunk_length is ignored for trai… kaldi-asr/kaldi#2969)

• [src] Fix bug in lattice-1best.cc RE removing insertion penalty (fix a bug in lattice-1best.cc about restoring insertion penalty from … kaldi-asr/kaldi#2970)

• [src] Compute a separate avg (start, end) interval for each sausage word ([src] Compute a separate avg (start, end) interval for each sausage word kaldi-asr/kaldi#2972)

• [build] Move nvcc verbose flag to proper location ([build] Fix nvcc verbose flag in cuda builds kaldi-asr/kaldi#2962)

• [egs] Fix mini_librispeech download_lm.sh crash; thx:chris.keith.johnson@gmail.com ([egs] Fix mini_librispeech download_lm.sh crash; thx:chris.keith.johnson@gmail.com kaldi-asr/kaldi#2974)

• [egs] minor fixes related to python2 vs python3 differences ([egs] Minor fixes in SRE recipes related to python2/3 compatibility  kaldi-asr/kaldi#2977)

• [src] Small fix in test code, avoid spurious failure ([src] Small fix in test code, avoid spurious failure kaldi-asr/kaldi#2978)

• [egs] Fix CSJ data-prep; minor path fix for USB version of data (Minor path fix for egs/csj/s5/local/csj_make_trans/csj_autorun.sh kaldi-asr/kaldi#2979)

• [egs] Add paper ref to README.txt in reverb example (Update README.txt in reverb kaldi-asr/kaldi#2982)

• [egs] Minor fixes to sitw recipe (fix problem introdueced in Python 2/3 compatibility kaldi-asr/kaldi#2925) ([egs] Minor fixes to sitw recipe kaldi-asr/kaldi#2985)

• [scripts] Fix bug introduced in Python 2/3 compatibility  kaldi-asr/kaldi#2957, RE integer division (Fixing bad initializer error (due to float num-filter variable) kaldi-asr/kaldi#2986)

• [egs] Update WSJ flat-start chain recipes to use TDNN-F not TDNN+LSTM (Update flat-start chain recipes to use TDNN-F kaldi-asr/kaldi#2988)

• [scripts] Fix typo introduced in Python 2/3 compatibility kaldi-asr/kaldi#2925 (Fix for typo kaldi-asr/kaldi#2989)

• [build] Modify Makefile and travis script to fix Travis failures (WIP: [travis] Try splitting two tasks to remove duplicated tasks. kaldi-asr/kaldi#2987)

• [src] Simplification and efficiency improvement in ivector-plda-scoring-dense (Small update to ivector-plda-scoring-dense kaldi-asr/kaldi#2991)

• [egs] Update madcat Arabic and Chinese egs, IAM (madcat Arabic (optional extra corpus text for LM), IAM (making word-based and BPE based setup similar), madcat Chinese (clean scripts, tuning) kaldi-asr/kaldi#2964)

• [src] Fix overflow bug in convolution code (Small fix to mitigate overflow in Convolution.cc kaldi-asr/kaldi#2992)

• [src] Fix nan issue in ctm times introduced in [src] Compute a separate avg (start, end) interval for each sausage word kaldi-asr/kaldi#2972, thx: @vesis84 (Fix nan issue in ctm times kaldi-asr/kaldi#2993)

• [src] Fix 'sausage-time' issue which occurs with disabled MBR decoding. (Fix 'sausage-time' issue which occurs with disabled MBR decoding. kaldi-asr/kaldi#2996)

• [egs] Add scripts for yomdle Russian (OCR task) (Adding scripts for yomdle Russian kaldi-asr/kaldi#2953)

• [egs] Simplify lexicon preparation in Fisher callhome Spanish (Spanish lexicon simplification kaldi-asr/kaldi#2999)

• [egs] Update GALE Arabic recipe (word-based gale arabic setup new tdnn recipe kaldi-asr/kaldi#2934)

• [egs] Remove outdated NN results from Gale Arabic recipe (Gale Arabic: removing old NN results  kaldi-asr/kaldi#3002)

• [egs] Add RESULTS file for the tedlium s5_r3 (release 3) setup (added tedlium s5_r3 RESULTS file kaldi-asr/kaldi#3003)

• [src] Fixes to grammar-fst code to handle LM-disambig symbols properly ([src] Fixes to grammar-fst code to handle LM-disambig symbols properly kaldi-asr/kaldi#3000)

thanks: armando.muscariello@gmail.com

• [src] Cosmetic change to mel computation (fix option string) (fix mel high-freq help text: missing "=" kaldi-asr/kaldi#3011)

• [src] Fix Visual Studio error due to alternate syntactic form of noreturn (Fix Visual Studio error due to alternate syntactic form of noreturn kaldi-asr/kaldi#3018)

• [egs] Fix location of sequitur installation (Fixed errors due to changes in Sequitur installation set kaldi-asr/kaldi#3017)

• [src] Fix w/ ifdef Visual Studio error from alternate syntactic form noreturn (Fix w/ ifdef Visual Studio error from alternate syntactic form noreturn kaldi-asr/kaldi#3020)

• [egs] Some fixes to getting data in heroico recipe ([egs] Some fixes to getting data in heroico recipe kaldi-asr/kaldi#3021)

• [egs] BABEL script fix: avoid make_L_align.sh generating invalid files (changing behavior of make_L_align.sh to avoid generating invalid files kaldi-asr/kaldi#3022)

• [src] Fix to older online decoding code in online/ (OnlineFeInput; was broken by commit cc2469e). (Fixes OnlineFeInput, which was broken by commit cc2469e8. kaldi-asr/kaldi#3025)

• [script] Fix unset bash variable in make_mfcc.sh (fix unset bash variable in make_mfcc.sh kaldi-asr/kaldi#3030)

• [scripts] Extend limit_num_gpus.sh to support --num-gpus 0. (able to exclude a job from running on gpus kaldi-asr/kaldi#3027)

• [scripts] fix bug in utils/add_lex_disambig.pl when sil-probs and pron-probs used (fix bug in utils/add_lex_disambig.pl kaldi-asr/kaldi#3033)

bug would likely have resulted in determinization failure (only when not using word-position-dependent phones).

• [egs] Fix path in Tedlium r3 rnnlm training script (Get text in tedlium r3 rnnlm training script kaldi-asr/kaldi#3039)

• [src] Thread-safety for GrammarFst (thx:armando.muscariello@gmail.com) ([src] Thread-safety for GrammarFst (thx:armando.muscariello@gmail.com) kaldi-asr/kaldi#3040)

• [scripts] Cosmetic fix to get_degs.sh (move echo command before exit kaldi-asr/kaldi#3045)

• [egs] Small bug fixes for IAM and UW3 recipes (Bug fixes for IAM and UW3 kaldi-asr/kaldi#3048)

• [scripts] Nnet3 segmentation: fix default params ([scripts] Nnet3 segmentation: fix default params kaldi-asr/kaldi#3051)

• [scripts] Allow perturb_data_dir_speed.sh to work with utt2lang (Added treatment for utt2lang file kaldi-asr/kaldi#3055)

• [scripts] Make beam in monophone training configurable (Make beam in monophone training configurable  kaldi-asr/kaldi#3057)

• [scripts] Allow reverberate_data_dir.py to support unicode filenames (data augmentation UnicodeDecodeError kaldi-asr/kaldi#3060)

• [scripts] Make some cleanup scripts work with python3 (Make some cleanup scripts work with python3 and python2 kaldi-asr/kaldi#3054)

• [scripts] bug fix to nnet2->3 conversion, fixes nnet2 to nnet3 model converter kaldi-asr/kaldi#886 (Read const_component_dim during conversion of nnet2 to nnet3 model kaldi-asr/kaldi#3071)

• [src] Make copies occur in per-thread default stream (for GPUs) (Remove stream0 copies.  kaldi-asr/kaldi#3068)

• [src] Add GPU version of MergeTaskOutput().. relates to batch decoding (Add MergeTaskOutputGPU which keeps GPU data on the device. kaldi-asr/kaldi#3067)

• [src] Add device options to enable tensor core math mode. (Add device options to enable tensor core math mode. kaldi-asr/kaldi#3066)

• [src] Log nnet3 computation to VLOG, not std::cout ([src] Log nnet3 computation to VLOG, not std::cout kaldi-asr/kaldi#3072)

• [src] Allow upsampling in compute-mfcc-feats, etc. ([src] Allow upsampling in compute-mfcc-feats, etc. kaldi-asr/kaldi#3014)

• [src] fix problem with rand_r being undefined on Android (fix rand_r for Android kaldi-asr/kaldi#3037)

• [egs] Update swbd1_map_words.pl, fix them_1's -> them's (Update swbd1_map_words.pl kaldi-asr/kaldi#3052)

• [src] Add const overload OnlineNnet2FeaturePipeline::IvectorFeature ([src] Add const overload OnlineNnet2FeaturePipeline::IvectorFeature kaldi-asr/kaldi#3073)

• [src] Fix syntax error in egs/bn_music_speech/v1/local/make_musan.py (removed extra parenthesis in local/musan.py kaldi-asr/kaldi#3074)

• [src] Memory optimization for online feature extraction of long recordings (Memory optimization for online feature extraction of long recordings kaldi-asr/kaldi#3038)

• [build] fixed a bug in linux_configure_redhat_fat when use_cuda=no (Fixed a bug in linux_configure_redhat_fat when use_cuda=no kaldi-asr/kaldi#3075)

• [scripts] Add missing '. ./path.sh' to get_utt2num_frames.sh (Minor fix: source path.sh in get_utt2num_frames.sh kaldi-asr/kaldi#3076)

• [src,scripts,egs] Add count-based biphone tree tying for flat-start chain training ( Add count-based biphone tree tying for flat-start chain training kaldi-asr/kaldi#3007)

• [scripts,egs] Remove sed from various scripts (avoid compatibility problems) (Sed compatibility  kaldi-asr/kaldi#2981)

• [src] Rework error logging for safety and cleanliness ([src] Rework error logging for safety and cleanliness kaldi-asr/kaldi#3064)

• [src] Change warp-synchronous to cub::BlockReduce (safer but slower) (Replaced warp-synchronous reductions with Block-Reduce kaldi-asr/kaldi#3080)

• [src] Fix && and || uses where & and | intended, and other weird errors ([src] Fix '&&' and '||' uses where '&' and '|' were intended kaldi-asr/kaldi#3087)

• [build] Some fixes to Makefiles ([makefiles] Fix warnings due to makefiles kaldi-asr/kaldi#3088)

clang is unhappy with '-rdynamic' in compile-only step, and the
switch is really unnecessary.

Also, the default location for MKL 64-bit libraries is intel64/.
The em64t/ was explained already obsolete by an Intel rep in 2010:
https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/285973

• [src] Fixed -Wreordered warnings in feat (Fixed -Wreordered warnings in feat kaldi-asr/kaldi#3090)

• [egs] Replace bc with perl -e ([egs] Replace bc with perl -e kaldi-asr/kaldi#3093)

• [scripts] Fix python3 compatibility issue in data-perturbing script (Python3 error: minor modifications to avoid error at Stage 20 of run.sh in Librispeech kaldi-asr/kaldi#3084)

• [doc] fix some typos in doc. ([doc] fix some typos in doc. kaldi-asr/kaldi#3097)

• [build] Make sure expf() speed probe times sensibly ([src] Make sure expf() speed probe times sensibly kaldi-asr/kaldi#3089)

• [scripts] Make sure merge_targets.py works in python3 (Removed import __future__ and fixed a bug in merge_targets.py kaldi-asr/kaldi#3094)

• [src] ifdef to fix compilation failure on CUDA 8 and earlier (ifdef to fix Cuda incompatibility issue kaldi-asr/kaldi#3103)

• [doc] fix typos and broken links in doc. ([doc] fix typos and broken links in doc. kaldi-asr/kaldi#3102)

• [scripts] Fix frame_shift bug in egs/swbd/s5c/local/score_sclite_conf.sh (fix frame_shift bug in egs/swbd/s5c/local/score_sclite_conf.sh kaldi-asr/kaldi#3104)

• [src] Fix wrong assertion failure in nnet3-am-compute (Failed assertion only if divide by priors is set to True. kaldi-asr/kaldi#3106)

• [src] Cosmetic changes to natural-gradient code ([src] Cosmetic changes to natural-gradient code kaldi-asr/kaldi#3108)

• [src,scripts] Python2 compatibility fixes and code cleanup for nnet1 (nnet1: we fix the python scripts (fixing bug for python2), kaldi-asr/kaldi#3113)

• [doc] Small documentation fixes; update on Kaldi history (update the docs: kaldi-asr/kaldi#3031)

• [src] Various mostly-cosmetic changes (copying from another branch) ([src] Various mostly-cosmetic changes (copying from another branch) kaldi-asr/kaldi#3109)

• [scripts] Simplify text encoding in RNNLM scripts (now only support utf-8) (Rnnlm scripts utf8 support kaldi-asr/kaldi#3065)

• [egs] Add "formosa_speech" recipe (Taiwanese Mandarin ASR) ("formosa_speech" recipe and database for Taiwanese Mandarin speech recognition kaldi-asr/kaldi#2474)

• [egs] python3 compatibility in csj example script (Update run_tdnn_1a.sh kaldi-asr/kaldi#3123)

• [egs] python3 compatibility in example scripts ([scripts] python3 compatibility in example scripts kaldi-asr/kaldi#3126)

• [scripts] Bug-fix for removing deleted words ([scripts] Bug-fix for removing deleted words kaldi-asr/kaldi#3116)

The type of --max-deleted-words-kept-when-merging in segment_ctm_edits.py
was a string, which prevented the mechanism from working altogether.

• [scripts] Add fix regarding num-jobs for segment_long_utterances*.sh(Adding fix for splitting text kaldi-asr/kaldi#3130)

• [src] Enable allow_{upsample,downsample} with online features (fixing the propagation of allow_{upsample,downsample} kaldi-asr/kaldi#3139)

• [src] Fix bad assert in fstmakecontextsyms ([src] Fix wrong assertion failure in fstmakecontextsym kaldi-asr/kaldi#3142)

• [src] Fix to "Fixes to grammar-fst & LM-disambig symbols" ([src] Fixes to grammar-fst code to handle LM-disambig symbols properly kaldi-asr/kaldi#3000) ([src] Fix to "Fixes to grammar-fst & LM-disambig symbols" (#3000) kaldi-asr/kaldi#3143)

• [build] Make sure PaUtils exported from portaudio (make sure you export PaUtils when compiling portaudio during make ext kaldi-asr/kaldi#3144)

• [src] cudamatrix: fixing a synchronization bug in 'normalize-per-row' (cudamatrix: fixing a synchronization bug in 'normalize-per-row' kaldi-asr/kaldi#3145)

was only apparent using large matrices

• [src] Fix typo in comment ([doc] fix an error in the comment for "LevenshteinEditDistance()" kaldi-asr/kaldi#3147)

• [src] Add binary that functions as a TCP server (Online2 NNet3 TCP server program kaldi-asr/kaldi#2938)

• [scripts] Fix bug in comment (Fixed a typo in parse_options.sh kaldi-asr/kaldi#3152)

• [scripts] Fix bug in steps/segmentation/ali_to_targets.sh (Debugged syntax bug in *steps/segmentation/ali_to_targets.sh* kaldi-asr/kaldi#3155)

• [scripts] Avoid holding out more data than the requested num-utts (due to utt2uniq) ([egs] Sample hold-out set fairly kaldi-asr/kaldi#3141)

• [src,scripts] Add support for two-pass agglomerative clustering. ([src,egs] Add support for two-pass agglomerative clustering. kaldi-asr/kaldi#3058)

• [src] Disable unget warning in PeekToken (and other small fix) ([src] Disable unget warning in PeekToken kaldi-asr/kaldi#3163)

• [build] Add new nvidia tools to windows build ([WIP] Windows cub building kaldi-asr/kaldi#3159)

• [doc] Fix documentation errors and add more docs for tcp-server decoder (Documentation errors kaldi-asr/kaldi#3164)