text2image segmentation fault on macOS ( regression #195?) #736

atuyosi · 2017-02-25T20:43:51Z

There seems to be a regression bug with text2image of 3.05 release.
The text2image utility caused segfault, so I couldn' use tesstrain.sh on macOS (10.12.3).

$ text2image --text eng.training_text --outputbase=eng.TimesNewRomanBold.exp0 --font='Times New Roman, Bold' --fonts_dir=/Library/Fonts --tlog_level=3
query weight = 700 	 selected weight =700
query_desc: 'Times New Roman, Bold' Selected: 'Times New Roman, Bold'
Render string of size 6801
Starting page 0
max_width = 3400, max_height = 4600
len = 6801  buf_len = 6801
Found offset = 6421
Segmentation fault: 11

ref. #195

for detail:

$ lldb /usr/local/bin/text2image
(lldb) target create "/usr/local/bin/text2image"
Current executable set to '/usr/local/bin/text2image' (x86_64).
(lldb) run --text eng.training_text --outputbase=eng.TimesNewRomanBold.exp0 --font='Times New Roman, Bold' --fonts_dir=/Library/Fonts
Process 40801 launched: '/usr/local/bin/text2image' (x86_64)
Process 40801 stopped
* thread #1: tid = 0xffe117, 0x0000000100bdd5a7 libpangoft2-1.0.0.dylib`pango_fc_font_get_glyph + 25, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000100bdd5a7 libpangoft2-1.0.0.dylib`pango_fc_font_get_glyph + 25
libpangoft2-1.0.0.dylib`pango_fc_font_get_glyph:
->  0x100bdd5a7 <+25>: movq   (%rcx), %rdi
    0x100bdd5aa <+28>: testq  %rdi, %rdi
    0x100bdd5ad <+31>: je     0x100bdd5b8               ; <+42>
    0x100bdd5af <+33>: movq   %rax, %rsi
(lldb)

$ tesseract -v
tesseract 3.05.00
 leptonica-1.74.1
  libjpeg 8d : libpng 1.6.28 : libtiff 4.0.7 : zlib 1.2.8

The text was updated successfully, but these errors were encountered:

amitdo · 2017-02-26T07:25:53Z

Here is the source for the regression:
709935851061#diff-b37dca9f063c3727f62c496e514177a9L440

amitdo · 2017-02-26T07:40:39Z

It also affects 4.00.

amitdo · 2017-02-26T17:38:32Z

@atuyosi, please try my patch:
https://github.com/amitdo/tesseract/commits/3.05-fc-fix
amitdo@da7a7fa2f23f

atuyosi · 2017-02-27T09:48:48Z

Hi @amitdo

Thanks for your quick response.

I tried your patch, but same result.

$ wget https://github.com/amitdo/tesseract/commit/da7a7fa2f23f.patch
$ patch -p1 < da7a7fa2f23f.patch
patching file training/pango_font_info.cpp

try patched binary:

$ text2image --text eng.training.txt --outputbase=eng.TimesNewRomanBold.exp0 --font='Times New Roman, Bold' --fonts_dir=/Library/Fonts --tlog_level=3
query weight = 700 	 selected weight =700
query_desc: 'Times New Roman, Bold' Selected: 'Times New Roman, Bold'
Render string of size 84
Starting page 0
max_width = 3400, max_height = 4600
len = 84  buf_len = 84
Segmentation fault: 11

$ lldb /usr/local/bin/text2image
(lldb) target create "/usr/local/bin/text2image"
Current executable set to '/usr/local/bin/text2image' (x86_64).
(lldb) run --text eng.training.txt --outputbase=eng.TimesNewRomanBold.exp0 --font='Times New Roman, Bold' --fonts_dir=/Library/Fonts --tlog_level=3
Process 88534 launched: '/usr/local/bin/text2image' (x86_64)
query weight = 700 	 selected weight =700
query_desc: 'Times New Roman, Bold' Selected: 'Times New Roman, Bold'
Render string of size 84
Starting page 0
max_width = 3400, max_height = 4600
len = 84  buf_len = 84
Process 88534 stopped
* thread #1: tid = 0x122eeb9, 0x0000000100cfb5a7 libpangoft2-1.0.0.dylib`pango_fc_font_get_glyph + 25, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000100cfb5a7 libpangoft2-1.0.0.dylib`pango_fc_font_get_glyph + 25
libpangoft2-1.0.0.dylib`pango_fc_font_get_glyph:
->  0x100cfb5a7 <+25>: movq   (%rcx), %rdi
    0x100cfb5aa <+28>: testq  %rdi, %rdi
    0x100cfb5ad <+31>: je     0x100cfb5b8               ; <+42>
    0x100cfb5af <+33>: movq   %rax, %rsi
(lldb)

amitdo · 2017-02-27T10:41:06Z

Try to manually set the environment variable PANGOCAIRO_BACKEND to fc and then call text2image.

atuyosi · 2017-02-27T11:55:09Z

No error occurred. It looks work. It seems to be working.

$ PANGOCAIRO_BACKEND=fc text2image --text eng.training.txt --outputbase=eng.TimesNewRomanBold.exp0 --font='Times New Roman, Bold' --fonts_dir=/Library/Fonts --tlog_level=3  2&> detail_log_736_1.txt

detail_log_736_1.txt

But macOS's Preview.app can't open this tif file (although Gimp can open this file ).
I'll try to run tesstrain.sh later.

atuyosi · 2017-02-27T18:19:31Z

I tried to run tesstrain.sh with above patch and workaround . It works fine. Successfully generated jpn.traineddata.

$ TESSDATA_PREFIX=/usr/local/share/  PANGOCAIRO_BACKEND=fc ./training/tesstrain.sh --lang jpn --langdata_dir ../github_tesseract/langdata/  --training_text ../github_tesseract/langdata/jpn/jpn.training_text --fonts_dir /Library/Fonts/

Note: I replaced mktemp with gmktemp (coreutils ) on macOS

amitdo · 2017-02-27T19:42:04Z

It should also work without my patch.

The patch was supposed to programmatically set the said environment variable. Since you reported it did not work, I asked you to do it manually. The call to the setenv function in my patch needs to be moved to another place in the code.

heyidi · 2017-03-09T01:21:31Z

thank you amitdo. it fixed my problem too~~

FernandoGOT · 2018-04-07T07:51:29Z

Got the same problem on MAC OSX 10.13.3 using tessercat 4.0.0-beta.1, fixed it using the PANGOCAIRO_BACKEND=fc before executing the tesstrain.sh file

amitdo · 2018-04-15T16:24:08Z

@FernandoGOT,

Can you please check my new patch?
amitdo@ee2d6739cf0782

Don't set the environment variable PANGOCAIRO_BACKEND when you test this patch.

stweil · 2018-04-15T19:27:08Z

Instead of setting PANGOCAIRO_BACKEND=fc, it should also be possible to replace pango_cairo_font_map_get_default() by pango_cairo_font_map_new_for_font_type(CAIRO_FONT_TYPE_FT).

amitdo · 2018-04-15T20:49:50Z

@stweil, your suggested method looks simpler. Please make a PR.

zdenop · 2018-09-27T12:52:24Z

Is this issue still valid? If yes can we get this modification to 4.0 finale release?

amitdo · 2018-09-27T13:25:24Z

I don't have a Mac, but we didn't done anything to fix this issue.

@stweil, #736 (comment)

atuyosi · 2018-09-27T13:53:28Z

@zdenop It has not fixed (4.0.0-beta.4-164-g5dfce).

It is still necessary to set environment variables PANGOCAIRO_BACKEND=fc .

zdenop · 2018-09-28T09:08:50Z

Can you please check if 345e5ee solve the problem?

atuyosi · 2018-09-28T10:22:45Z

@zdenop ,

I tested 345e5ee. No error occurred.

It's OK.

$ text2image --fonts_dir /System/Library/Fonts/  --list_available_fonts
<skip>
$ echo $?
0

$ text2image --text jpn_test.txt  --outputbase=jpn.HiraginoMinchoProLicht.exp0 --font='Hiragino Mincho Pro Light' --fonts_dir=/System/Library/Fonts --tlog_level=3 --ptsize 32
<skip>
$ echo $?
0

Software Version:

$ text2image --version
4.0.0-beta.4-167-g345e

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14
BuildVersion:	18A391

zdenop · 2018-10-20T09:58:32Z

I have to revert this patch because of side effects. Other solution should be found or implemented...

zdenop · 2018-10-20T19:42:41Z

@atuyosi: can you please try get the latest code and post output of text2image -v

zdenop · 2018-10-22T11:30:02Z

I have ready something like this:

diff --git a/src/training/text2image.cpp b/src/training/text2image.cpp
index 1142bb68..3e2b8a23 100644
--- a/src/training/text2image.cpp
+++ b/src/training/text2image.cpp
@@ -50,6 +50,9 @@
 #include "tlog.h"
 #include "unicharset.h"
 #include "util.h"
+#ifdef __APPLE__
+#  include "TargetConditionals.h"
+#endif

 // A number with which to initialize the random number generator.
 const int kRandomSeed = 0x18273645;
@@ -689,5 +692,19 @@ int main(int argc, char** argv) {
     }
   }
   tesseract::ParseCommandLineFlags(argv[0], &argc, &argv, true);
+#ifdef TARGET_OS_MAC
+  // Respect enviroment variable. could be:
+  // fc (fontconfig), win32, and coretext
+  // If not set force fontconfig for Mac OS.
+  // See https://github.com/tesseract-ocr/tesseract/issues/736
+  char* backend;
+  backend = getenv("PANGOCAIRO_BACKEND");
+  if (backend == NULL) {
+    putenv("PANGOCAIRO_BACKEND=fc");
+  } else {
+    printf("Using '%s' as pango cairo backend based on enviroment "
+           "variable.\n", backend);
+  }
+#endif
   return Main();
 }

Maybe we can implement it as option --pangocairo_backend for text2image, so user will be aware about this possibility to change backend.

amitdo · 2018-10-22T14:31:14Z

    } else {
      printf("Using '%s' as pango cairo backend based on enviroment "
             "variable.\n", backend);
   }

text2image's code needs the 'fc' backend on linux and mac.
Without the 'fc' backend there will be a crash.

zdenop · 2018-10-22T15:48:56Z

@amitdo : code you quoted means that program will accept what user will set (intentionally). This will give user option to choose backend (in case of new backend or fix for actual)

I have possibility to test it (just sort text - not whole training) on linux and windows:

On linux (opensuse 15) pango choose CAIRO_FONT_TYPE_FT automatically and I did not recognized any crash. I see no reason to modify code for linux.
On windows (10) pango choose CAIRO_FONT_TYPE_WIN32 automatically and I did not recognized any crash. Setting it to CAIRO_FONT_TYPE_FT (with PANGOCAIRO_BACKEND=fc). Anyway I prefer to use native solution if it works.

zdenop · 2018-10-22T19:51:51Z

@tfmorris : Can you please test this ('text2image -v' and patch above?)?

amitdo · 2018-10-22T20:31:09Z

Linux always uses 'fc'.

Originally Windows had the same issue:
#380

This patch fixed the crash on windows & mac:
8d2d94e4ed32c

but Ray partly reverted it, so it is only compiled for win32 and not on linux or mac.
709935851061#diff-b37dca9f063c3727f62c496e514177a9L440

Although the workaround is still there for win32, it has this remark:

// Fixme! Leaks memory and breaks unittests.

zdenop · 2018-10-23T07:38:22Z

@amitdo: thanks for analysis. Thank I will not limit this change to Mac.

amitdo · 2018-10-23T11:58:21Z

~~The new commit will still crash on mac..~~.

zdenop · 2018-10-23T12:07:21Z

Why?

amitdo · 2018-10-23T13:14:48Z

Because the default bacend on mac is 'coretext', but text2image is not designed to work with it

You should force 'fc' as backend, at least for mac.

zdenop · 2018-10-23T13:37:16Z

... and the patch DOES NOT care about default backend... It cares about environment variable definition. So it should work.

zdenop · 2018-10-23T13:57:20Z

@amitdo: do you have Mac or did you test it on Mac? Because you are claiming that Mac has set PANGOCAIRO_BACKEND variable by default to some value...

zdenop · 2018-10-23T14:03:07Z

@atuyosi: can you please test current code?

zdenop · 2018-10-23T14:29:18Z

...and patch, that reflect above logic, will do:

if env var is set = do nothing (because this PANGOCAIRO_BACKEND must be set by user and he should know what he is doing)
if env var is not set = set it fc (everything should works).

So crash should happened only in case 1. when user set PANGOCAIRO_BACKEND to wrong backend.

zdenop · 2018-10-23T14:56:41Z

@amitdo : can you please read my comments? You miss the point. Or try the code ;-)

amitdo · 2018-10-23T14:57:48Z

I will reread the code, maybe I missed something.

amitdo · 2018-10-23T15:10:34Z

It looks okay :-)

Please pardon me.

You changed the code a few times in the last week, and I was reading the diffs, so I didn't see the whole picture. That was the reason for my confusion.

stweil · 2020-12-22T18:07:50Z

I have to revert this patch because of side effects. Other solution should be found or implemented...

@zdenop, do you remember which side effects your initial patch had? It fixes pango_font_info_test for MacOS (see discussion for PR #3189).

amitdo · 2020-12-22T18:39:44Z

See #1999 and #2009

zdenop · 2020-12-22T20:07:12Z

@stweil : I do not remember, but pango was always strange for me. ;-) After pango changed build system, I lost interest to to play with as I am mainly on windows at this time...

amitdo · 2020-12-22T20:15:07Z

After pango changed build system, I lost interest to play with as I am mainly on windows at this time...

Meson is great and it does work on windows.

stweil · 2020-12-22T20:31:02Z

Thank you for your answers. The main problem was obviously unreleased memory. I now have a different solution for #3189.

amitdo mentioned this issue Apr 25, 2017

text2image segfault #195

Closed

zdenop closed this as completed Sep 28, 2018

This was referenced Oct 20, 2018

Not able to get correct ouput for tables #724

Closed

text2image segfault when using --list_available_fonts #2009

Closed

zdenop reopened this Oct 20, 2018

zdenop added this to the 4.0.0 milestone Oct 20, 2018

zdenop added the bug label Oct 20, 2018

zdenop closed this as completed in e60318f Oct 23, 2018

zdenop mentioned this issue Oct 23, 2018

RFC: Tesseract 4.0.0 – open tasks #1423

Closed

amitdo added the training label Jan 16, 2021

amitdo added the text2image label Sep 25, 2022

text2image segmentation fault on macOS ( regression #195?) #736

text2image segmentation fault on macOS ( regression #195?) #736

Comments

atuyosi commented Feb 25, 2017

amitdo commented Feb 26, 2017

amitdo commented Feb 26, 2017

amitdo commented Feb 26, 2017 • edited Loading

atuyosi commented Feb 27, 2017

amitdo commented Feb 27, 2017

atuyosi commented Feb 27, 2017

atuyosi commented Feb 27, 2017

amitdo commented Feb 27, 2017 • edited Loading

heyidi commented Mar 9, 2017

FernandoGOT commented Apr 7, 2018

amitdo commented Apr 15, 2018

stweil commented Apr 15, 2018

amitdo commented Apr 15, 2018

zdenop commented Sep 27, 2018

amitdo commented Sep 27, 2018

atuyosi commented Sep 27, 2018

zdenop commented Sep 28, 2018

atuyosi commented Sep 28, 2018

zdenop commented Oct 20, 2018

zdenop commented Oct 20, 2018

zdenop commented Oct 22, 2018

amitdo commented Oct 22, 2018

zdenop commented Oct 22, 2018

zdenop commented Oct 22, 2018

amitdo commented Oct 22, 2018 • edited Loading

zdenop commented Oct 23, 2018

amitdo commented Oct 23, 2018 • edited Loading

zdenop commented Oct 23, 2018

amitdo commented Oct 23, 2018 • edited Loading

zdenop commented Oct 23, 2018

zdenop commented Oct 23, 2018 • edited Loading

zdenop commented Oct 23, 2018

zdenop commented Oct 23, 2018 • edited Loading

zdenop commented Oct 23, 2018

amitdo commented Oct 23, 2018

amitdo commented Oct 23, 2018 • edited Loading

stweil commented Dec 22, 2020

amitdo commented Dec 22, 2020

zdenop commented Dec 22, 2020

amitdo commented Dec 22, 2020

stweil commented Dec 22, 2020

amitdo commented Feb 26, 2017 •

edited

Loading

amitdo commented Feb 27, 2017 •

edited

Loading

amitdo commented Oct 22, 2018 •

edited

Loading

amitdo commented Oct 23, 2018 •

edited

Loading

amitdo commented Oct 23, 2018 •

edited

Loading

zdenop commented Oct 23, 2018 •

edited

Loading

zdenop commented Oct 23, 2018 •

edited

Loading

amitdo commented Oct 23, 2018 •

edited

Loading