MappedFile functionality replaced with NULL

[yonatankahana]

Hi!
I succedded compile Kaldi with your port of OpenFST and I cant load HCLG model because you changed the method to return always null: https://github.com/kkm000/openfst/blob/win/1.6/src/lib/mapped-file.cc
there is a reason for that change? how can I make my Kaldi code work now?

[jtrmal]

Can you point us to the Kaldi code that is using MappedFile? y.

…

On Wed, Dec 6, 2017 at 10:50 AM, yonatankahana ***@***.***> wrote: Hi! I succedded compile Kaldi with your port of OpenFST and I cant load HCLG model because you changed the method to return always null: https://github.com/kkm000/openfst/blob/win/1.6/src/lib/mapped-file.cc there is a reason for that change? how can I make my Kaldi code work now? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKisX4eqiadEpRoZ2dR4KwSK7hHq2SCxks5s9rfYgaJpZM4Q4INx> .

[yonatankahana]

fst::internal::ConstFstImpl<fst::ArcTpl<fst::TropicalWeightTpl<float> >,unsigned int>::Read(std::basic_istream<char,std::char_traits<char> > & strm, const fst::FstReadOptions & opts) Line 213	C++	Symbols loaded.
 	my.dll!fst::ConstFst<fst::ArcTpl<fst::TropicalWeightTpl<float> >,unsigned int>::Read(std::basic_istream<char,std::char_traits<char> > & strm, const fst::FstReadOptions & opts) Line 274	C++	Symbols loaded.
>	my.dll!fst::ReadFstKaldiGeneric(std::basic_string<char,std::char_traits<char>,std::allocator<char> > rxfilename, bool throw_on_err) Line 76	C++	Symbols loaded.

basically the entry for this issue in this stacktrace comes from 'ReadFstKaldiGeneric'.

[jtrmal]

OK, but IIRC we have decided not to use the mem mapping in kaldi so I'm asking what kaldi binary source code does this or if it's your own code. y.

…

On Wed, Dec 6, 2017 at 11:34 AM, yonatankahana ***@***.***> wrote: fst::internal::ConstFstImpl<fst::ArcTpl<fst::TropicalWeightTpl<float> >,unsigned int>::Read(std::basic_istream<char,std::char_traits<char> > & strm, const fst::FstReadOptions & opts) Line 213 C++ Symbols loaded. my.dll!fst::ConstFst<fst::ArcTpl<fst::TropicalWeightTpl<float> >,unsigned int>::Read(std::basic_istream<char,std::char_traits<char> > & strm, const fst::FstReadOptions & opts) Line 274 C++ Symbols loaded. > my.dll!fst::ReadFstKaldiGeneric(std::basic_string<char,std::char_traits<char>,std::allocator<char> > rxfilename, bool throw_on_err) Line 76 C++ Symbols loaded. basically the entry for this issue in this stacktrace comes from 'ReadFstKaldiGeneric'. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKisXzUcKMOGiXrjxsmpsRVEvWq39gjwks5s9sIJgaJpZM4Q4INx> .

[jtrmal]

OK, I'm looking at the OpenFST code of ConstFst and I think I see the issue -- while it does not enforce to use MAP mode, it will always use the MappedFile.
@yonatankahana You can backport the code from older ports of OpenFST -- at this time I don't really have too much time nor easy access to win machine. Or maybe @kkm000 might have time but he's usually very busy as well.

[yonatankahana]

you can see in the file online-nnet2-decoding.h:
there is a class called "SingleUtteranceNnet2Decoder" which its constructor

  SingleUtteranceNnet2Decoder(const OnlineNnet2DecodingConfig &config,
                              const TransitionModel &tmodel,
                              const nnet2::AmNnet &model,
                              const fst::Fst<fst::StdArc> &fst,
                              OnlineFeatureInterface *feature_pipeline);

this method wants an fst::Fstfst::StdArc
which can be only created with ReadFstKaldiGeneric.

[kkm000]

Yeah, it has been my impression that ConstFst had to use mapped memory. It's a very natural candidate for it. I'll write a (nonmapping) replacement for it today or tomorrow. Use the vector fst as the workaround meanwhile.

…

-kkm On 6 Dec 2017 8:51 a.m., yonatankahana <notifications@github.com> wrote: you can see in the file online-nnet2-decoding.h: there is a class called "SingleUtteranceNnet2Decoder" which its constructor SingleUtteranceNnet2Decoder(const OnlineNnet2DecodingConfig &config, const TransitionModel &tmodel, const nnet2::AmNnet &model, const fst::Fst<fst::StdArc> &fst, OnlineFeatureInterface *feature_pipeline); this method wants an fst::Fstfst::StdArc which can be only created with ReadFstKaldiGeneric. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#8 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH2NhZCZKbiM96nm5GYwVpzN7IwigHHLks5s9sX0gaJpZM4Q4INx>.

[kkm000]

Please pull latest code from win/1.6. I have restored the original mapped-file.cc, but disabled code paths actually trying to map the file before allocating memory and reading data into it.

If that does not cover it (and it should, generally), I can implement actual mapping with Win32 functions. But MSVC does not have a compatible mmap(), this is a bit more work than I can dedicte myself to at this moment.

/cc @jtrmal. Change against the original file

[Excalib88]

I am sorry, but i have problem with openfst. Who can help me please. I cannot load HCLG model on Windows https://pastebin.com/NtaswnAA