forked from julius-speech/julius
-
Notifications
You must be signed in to change notification settings - Fork 0
/
00readme-DNN.txt
156 lines (113 loc) · 6.33 KB
/
00readme-DNN.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
Julius for DNN-based speech recognition
(updated 2013/09/29)
A. How it works
================
This version of Julius can perform, Julius can perform DNN-HMM based
recognition by receiving pre-computed state probabilities. In that mode,
Julius does not read any feature parameter vectors and compute the state
output probability of an HMM state in it, but just read the "output
probabilities vectors" of the HMM states already computed in other tools,
via socket or file.
The "output probabilities input" is called "outprob vector" in Julius,
which contains a sequence of vectors, each of them consists of
pre-computed state probabilities a vector of state-num-of-HMM dimension.
The most important thing to know before using this scheme is that,
each dimension in the input outprob vector and each state in the HMM in
Julius should corresponds. In other words, the index of HMM states
and outprob vector should match. The details are described in the
following section.
To run, you need
1) a GMM-HMM AM for Julius, (GMM defs are ignored, only HMM structure is used)
2) a DNN state definition of DNN-HMM that corresponds to 1),
3) a program to compute outprob vector from audio input using 2),either
to file or to Julius socket.
The related Julius options are:
- "-input outprob" for file input of outprob vector,
- "-input vecnet" for vector input (feature/outprob auto-detected by header)
You can also see the demo samples in DNN dictation toolkit which is available on the Web.
B. State ID to make correspondence between outprob vector and states
=====================================================================
Julius should know the correspondence between the states in the HMM
definition and the dimension number of the given input vector. The
dimension index, beginning from zero, shoule be assigned for each
state in the HMM definition. The index is called "state ID" in this
document.
You can explicitly specify the state ID of each state within HMM
defnition by embedding extra tag "<SID> value" in the hmmdefs. When
the "<SID>" tag exist in the given HMM file, Julius uses them as
dimension to access the input outprob vector. Other tools that
generate the outprob vector using DNN should also refer to the values
to generate an outprob vector in the proper order that matches the hmm
definition file.
If "<SID>" tag does not exist in the hmmdefs, Julius assigns the state
ID of each state in the order of appearance in the ASCII hmmdefs. In
that case the input outprob vector should also have the values in the
same order.
- Detailed format definition:
The "<SID> value" should be inserted at the head of "state_info"
statement, as decribed in the section "HTK definition language" in the
HTKBook. Currently it is not an official extension, and an hmmdefs
with "<SID>" embedded can not be used in the current HTK. You can see
the example script of manually embedding the "<SID>" tag into hmmdefs
at the script "embed_sil.pl" in the archive.
C. Will the state ID (or the order) be kept in the binary HMM?
===============================================================
No at old versions, yes at the newer version.
The state ID will be kept in the binary HMM with mkbinhmm of this
version and later. "<SID>" will be kept in the binary HMM. If not,
the appearance order of the source will be saved.
Please note that the older version of mkbinhmm does not convern about
the order of appearance in the source hmmdefs. You CANNOT use the
binary HMM generated by the older version for DNN. When you want to
perform DNN-based recognition, please re-convert from ASCII hmmdefs
with the newest version of mkbinhmm.
D. Making outprob vector
==========================
D.1. Format of outprob vector file
===================================
To make an outprob vector file, just save the state outout
probabilities of each input frame in HTK parameter format with "USER"
parameter type. The length of parameter vector should match the
number of states in the HMM definition. If the source hmmdefs have
"<SID>" tag, the output vector should have the same dimension order.
If don't, you should store the values in the order of appearance of
state definitions in the source hmmdefs file.
Advice: HTK by default cannot handle a vector input longer than 5000
bytes (= 1250 dim.). To handle large vector, you may have to modify
the source code of HTK.
D.2. Testing generation of an outprob vector file with Julius
--------------------------------------------------------------
Julius has a test function to save the outprob vector computed while
recognition. Run recognition with "-outprobout filename" and process
an input file. Then the state probabilities of the whole given input
will be written to the given filename.
Note that currently this function does not support batch processing
using "-filelist". Only the last one will be saved.
D.3. Use the outprob vector for recognition
---------------------------------------------
Run Julius with "-input outprob", and give the outprob vector file as
an input. Julius will refer to the pre-computed state probabilities
and perform decoding.
Julius still needs the source GMM-HMM definition to represent search
space. You should specify the source GMM-HMM using "-h" as normal
recognition even if using "-input outprob", and the state-dimension
correspondence as described in the "B" section above should be kept.
The "-input outprob" also accepts bacth input by "-filelist".
D.3. Sending feature / outprob vector via network
--------------------------------------------------
This version of Julius can receive input feature vector or outprob
vector from tcp/ip network to perform on-line recognition. To use
this, start Julius with an option "-input outprobnet", and connect
from other program with port number 5531.
The sample tiny program to send feature vector or outprob vector is in
"dnntools/sendvec.c". It reads a HTK parameter file and send it as
either input vector or outprob vector toward Julius. To test:
Terminal 1:
(compile Julius)
% ./julius/julius -C ..... -input vecnet
Terminal 2:
% cd dnntools
(edit sendvec.c to choose that the paramfile is whether an output
vector file or a feature vector file)
% cc -o sendvec sendvec.c
% sendvec paramfile localhost