[OS X] LuaTeX, synctex, and pdf-tools don't play nicely

[LordZorgoth]

I'm aware that OS X is not supported, but I want to make it known that this bug exists. I have only seen the error with the combination of OS X and LuaTeX. XeTeX and LaTeX seem to be fine, and LuaTeX seems to be fine on Linux, though I don't know if that is OS related or related to package versions. It also only appears frequently with large documents. It renders the use of pdf-tools and LuaTeX with very large documents (like my thesis) impractical on OS X.

The bug is as follows: The commands pdf-sync-forward-search, pdf-sync-backward-search, and pdf-backward-search-mouse work... sometimes. At other times, they spit out the message

pdf-info-query: 

or less frequently,

pdf-info-query: The epdfinfo server quit unexpectedly.

and do nothing. Synctex is in fact enabled, and in fact sometimes works, as noted above. Repeatedly executing the command on the same point in the same document sometimes works and sometimes doesn't. There is no pattern discernable to me regarding how frequently or under what circumstances the error occurs or doesn't occur, except that it seems to be most prevalent in large documents. It almost never happens in very small documents.

The error log of epdfinfo was set in Customize, but the log is empty. Looking at the source code, it seems that there must have been a previous synctex error that causes epdfinfo: Unable to create synctex scanner, did you run latex with --synctex=1?, but I can't tell what error it is because it doesn't seem to be reported anywhere I know how to access.

I installed emacs and pdf-tools using

brew tap railwaycat/emacsmacport
brew cask install emacs-mac
brew install poppler

from the terminal and

(package-install 'pdf-tools)

from emacs.

The first time I run emacs after installing pdf-tools, it tells me it needs to build the server and spits out the attached compiler log, and the second time I run it, it works.

The issue can be reproduced by installing emacs and pdf-tools on OS X as described, renaming the attached file verylargedocument.txt as verylargedocument.tex, and compiling it using

lualatex -synctex=1 verylargedocument.tex

You will find the issue quickly if you then open verylargedocument.pdf and try jumping around between the PDF and the tex file using pdf-sync-forward-search, pdf-sync-backward-search, and pdf-backward-search-mouse.

pdftoolscompilelog.txt

verylargedocument.txt

[politza]

I don't have luatex at hand and can't find a problem using pdflatex. Could you upload the compiled document and the synctex.gz file ?

I also attached a script which you could try. I runs forward/backward searches multiple times on random data. You possibly need to adapt the path in the header.

make-bug.txt

[LordZorgoth]

Here are the PDF and the synctex file:

verylargedocument.pdf
verylargedocument.synctex.gz

As I said, the bug is only reproduceable with the OS X/LuaTeX combination, to my knowledge, though I actually have not tried using the combination of compiling on OS X, then attempting to sync on Linux.

When I try to sync a document on OS X that was compiled on Linux, it creates a blank buffer, so my guess is that the same thing will happen in the other direction. It may be because the path is /home/jonathan/Dropbox/Research/verylargedocument.tex in Linux and /Users/jonathan/Dropbox/Research/verylargedocument.tex in OS X?

[LordZorgoth]

I worked that issue by using the deeply profound trick of replacing the text /Users/ with /home/ in the synctex.gz file. SyncTeX works fine on Linux but not OS X with the same files (both compiled on OS X), which means that lualatex is doing what it's supposed to, and probably that epdfinfo or one of its dependencies doesn't work properly on OS X.

I also tested the Linux compiled version on OS X using the reverse of the above replacement, and epdfinfo fails on OS X (but not Linux).

[politza]

Ok, then I don't have to try. Sounds to me like someone needs to debug this on an OSX system.

The synctex version I'm using is slightly behind upstream, so updating could potentially fix this as well.

[LordZorgoth]

I would be willing to do some debugging on OS X, though I don't have that much time. For the moment, I've switched to xelatex and I am hoping against hope that switching doesn't do any hard-to-find and wacky things to my typography that I'll have to track down.

I can't say I know how, however. What would I do to find the source of the problem?

[politza]

The best way to debug this is by directly running the executable in the elpa directory. If started, it reads commands from stdin, one per line. Each line starts with the commands name, followed by its arguments separated by colons.

The synctex forward command looks like this:

synctex-forward-search:/path/to/pdf.pdf:/path/to/source.tex:line:column

where line and column are the line- and column-number of the position in the source file.

And the backward one like this

synctex-backward-search:/path/to/pdf.pdf:page:x:y

where x and y are relative image coordinates, i.e. 0.0 <= x,y <= 1.0 .

You should create a file, let's say commands.txt, containing at least one command which is triggering this bug. If you set pdf-info-log to t, all commands will be logged to the buffer *pdf-info-log* and you could copy them from there.

Then in Emacs cd to the elpa directory of pdf-tools and start up gdb

M-x gud-gdb RET gdb --fullname ./epdfinfo RET

This should create a buffer in which the gdb process is running. Next set your commands as input to the process via the following gdb command.

set args < /path/to/commands.txt

Finally start the program:

run

If this triggers the error, try

bt

to see a stack trace.

That should do it.

[allmightty]

(gdb) set args < /home/archie/downloads/command.txt (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/archie/.emacs.d/elpa/pdf-tools-20170915.1441/epdfinfo < /home/archie/downloads/command.txt [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". OK /_private_.tex:260:-1 . *** Error in /home/archie/.emacs.d/elpa/pdf-tools-20170915.1441/epdfinfo': free(): invalid next size (fast): 0x0000555555811490 ***

Program received signal SIGABRT, Aborted.
0x00007ffff57dc8a0 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007ffff57dc8a0 in raise () at /usr/lib/libc.so.6
#1 0x00007ffff57ddf09 in abort () at /usr/lib/libc.so.6
#2 0x00007ffff581f517 in __libc_message () at /usr/lib/libc.so.6
#3 0x00007ffff5825c84 in malloc_printerr () at /usr/lib/libc.so.6
#4 0x00007ffff5827599 in _int_free () at /usr/lib/libc.so.6
#5 0x0000555555561da5 in _synctex_free_input (node=0x555555811500)
at synctex_parser.c:732
#6 0x0000555555561da5 in _synctex_free_input (node=0x555555811570)
at synctex_parser.c:732
#7 0x0000555555561da5 in _synctex_free_input (node=0x5555558115e0)
at synctex_parser.c:732
#8 0x0000555555561da5 in _synctex_free_input (node=0x555555811650)
at synctex_parser.c:732
#9 0x0000555555561da5 in _synctex_free_input (node=0x5555558116c0)
at synctex_parser.c:732
#10 0x0000555555561da5 in _synctex_free_input (node=0x555555811730)
at synctex_parser.c:732
#11 0x0000555555561da5 in _synctex_free_input (node=0x5555558117a0)
at synctex_parser.c:732
#12 0x0000555555561da5 in _synctex_free_input (node=0x555555811810)
at synctex_parser.c:732
#13 0x0000555555561da5 in _synctex_free_input (node=0x555555811880)
at synctex_parser.c:732
#14 0x0000555555561da5 in _synctex_free_input (node=0x5555558118f0)
at synctex_parser.c:732
#15 0x0000555555561da5 in _synctex_free_input (node=0x555555811960)
at synctex_parser.c:732
#16 0x0000555555561da5 in _synctex_free_input (node=0x5555558119d0)
at synctex_parser.c:732
#17 0x0000555555561da5 in _synctex_free_input (node=0x555555811a40)
at synctex_parser.c:732
#18 0x0000555555561da5 in _synctex_free_input (node=0x555555811ab0)
at synctex_parser.c:732
#19 0x0000555555561da5 in _synctex_free_input (node=0x555555811b20)
at synctex_parser.c:732
#20 0x0000555555561da5 in _synctex_free_input (node=0x555555811ba0)
at synctex_parser.c:732
#21 0x0000555555561da5 in _synctex_free_input (node=0x555555811c10)
at synctex_parser.c:732
#22 0x0000555555561da5 in _synctex_free_input (node=0x555555811c80)
at synctex_parser.c:732
#23 0x0000555555561da5 in _synctex_free_input (node=0x555555811cf0)
at synctex_parser.c:732
#24 0x0000555555561da5 in _synctex_free_input (node=0x555555811d70)
at synctex_parser.c:732
#25 0x0000555555561da5 in _synctex_free_input (node=0x555555811df0)
at synctex_parser.c:732
#26 0x0000555555561da5 in _synctex_free_input (node=0x555555811e60)
at synctex_parser.c:732
#27 0x0000555555561da5 in _synctex_free_input (node=0x555555811ed0)
at synctex_parser.c:732
#28 0x0000555555561da5 in _synctex_free_input (node=0x555555811f40)
at synctex_parser.c:732
#29 0x0000555555561da5 in _synctex_free_input (node=0x555555811fb0)
at synctex_parser.c:732
#30 0x0000555555561da5 in _synctex_free_input (node=0x555555812030)
at synctex_parser.c:732
#31 0x0000555555561da5 in _synctex_free_input (node=0x5555558120a0)
at synctex_parser.c:732
#32 0x0000555555561da5 in _synctex_free_input (node=0x555555812110)
at synctex_parser.c:732
#33 0x0000555555561da5 in _synctex_free_input (node=0x555555812190)
at synctex_parser.c:732
#34 0x0000555555561da5 in _synctex_free_input (node=0x555555812200)
at synctex_parser.c:732
#35 0x0000555555561da5 in _synctex_free_input (node=0x555555812270)
at synctex_parser.c:732
#36 0x0000555555561da5 in _synctex_free_input (node=0x5555558122e0)
at synctex_parser.c:732
#37 0x0000555555561da5 in _synctex_free_input (node=0x555555812350)
at synctex_parser.c:732
#38 0x0000555555561da5 in _synctex_free_input (node=0x5555558123c0)
at synctex_parser.c:732
#39 0x0000555555561da5 in _synctex_free_input (node=0x555555812430)
at synctex_parser.c:732
#40 0x0000555555561da5 in _synctex_free_input (node=0x5555558124a0)
at synctex_parser.c:732
#41 0x0000555555561da5 in _synctex_free_input (node=0x555555812510)
at synctex_parser.c:732
#42 0x0000555555561da5 in _synctex_free_input (node=0x555555812590)
at synctex_parser.c:732
#43 0x0000555555561da5 in _synctex_free_input (node=0x555555812610)
at synctex_parser.c:732
#44 0x0000555555561da5 in _synctex_free_input (node=0x555555812680)
at synctex_parser.c:732
#45 0x0000555555561da5 in _synctex_free_input (node=0x5555558126f0)
at synctex_parser.c:732
#46 0x0000555555561da5 in _synctex_free_input (node=0x555555812760)
at synctex_parser.c:732
#47 0x0000555555561da5 in _synctex_free_input (node=0x5555558127e0)
at synctex_parser.c:732
#48 0x0000555555561da5 in _synctex_free_input (node=0x555555812850)
at synctex_parser.c:732
#49 0x0000555555561da5 in _synctex_free_input (node=0x5555558128c0)
at synctex_parser.c:732
#50 0x0000555555561da5 in _synctex_free_input (node=0x555555812930)
at synctex_parser.c:732
#51 0x0000555555561da5 in _synctex_free_input (node=0x5555558129a0)
at synctex_parser.c:732
#52 0x0000555555561da5 in _synctex_free_input (node=0x555555812a10)
at synctex_parser.c:732
#53 0x0000555555561da5 in _synctex_free_input (node=0x555555812a90)
at synctex_parser.c:732
#54 0x0000555555561da5 in _synctex_free_input (node=0x555555812b00)
at synctex_parser.c:732
#55 0x0000555555561da5 in _synctex_free_input (node=0x555555812b80)
at synctex_parser.c:732
#56 0x0000555555561da5 in _synctex_free_input (node=0x555555812bf0)
at synctex_parser.c:732
#57 0x0000555555561da5 in _synctex_free_input (node=0x555555812c60)
at synctex_parser.c:732
#58 0x0000555555561da5 in _synctex_free_input (node=0x555555812cd0)
at synctex_parser.c:732
#59 0x0000555555561da5 in _synctex_free_input (node=0x555555812d50)
at synctex_parser.c:732
#60 0x0000555555561da5 in _synctex_free_input (node=0x555555812dd0)
at synctex_parser.c:732
#61 0x0000555555561da5 in _synctex_free_input (node=0x555555812e40)
at synctex_parser.c:732
#62 0x0000555555561da5 in _synctex_free_input (node=0x555555812ec0)
at synctex_parser.c:732
#63 0x0000555555561da5 in _synctex_free_input (node=0x555555812f50)
at synctex_parser.c:732
#64 0x0000555555561da5 in _synctex_free_input (node=0x555555812fd0)
at synctex_parser.c:732
#65 0x0000555555561da5 in _synctex_free_input (node=0x555555813050)
at synctex_parser.c:732
#66 0x0000555555561da5 in _synctex_free_input (node=0x5555558130c0)
at synctex_parser.c:732
#67 0x0000555555561da5 in _synctex_free_input (node=0x555555813140)
at synctex_parser.c:732
#68 0x0000555555561da5 in _synctex_free_input (node=0x5555558131b0)
at synctex_parser.c:732
#69 0x0000555555561da5 in _synctex_free_input (node=0x555555813220)
at synctex_parser.c:732
#70 0x0000555555561da5 in _synctex_free_input (node=0x555555813290)
at synctex_parser.c:732
#71 0x0000555555561da5 in _synctex_free_input (node=0x555555813300)
at synctex_parser.c:732
#72 0x0000555555561da5 in _synctex_free_input (node=0x555555813380)
at synctex_parser.c:732
#73 0x0000555555561da5 in _synctex_free_input (node=0x555555813410)
at synctex_parser.c:732
#74 0x0000555555561da5 in _synctex_free_input (node=0x55555581af00)
at synctex_parser.c:732
#75 0x0000555555561da5 in _synctex_free_input (node=0x555555efbca0)
at synctex_parser.c:732
#76 0x000055555556710f in synctex_scanner_free (scanner=0x5555557cd900)
at synctex_parser.c:2797
#77 0x000055555555d136 in cmd_synctex_backward_search (ctx=, args=) at epdfinfo.c:3065
#78 0x000055555555acf9 in main (argc=, argv=) at epdfinfo.c:3689
(gdb) `

I use archlinux + synctex + lua(la)tex + emacs and i have a similar symptom .
Above is the log during a debug step you mentioned

synctex-backward-search:/_private_.pdf:5:0.0:0.1

And this is the text in command.txt ( i don't get a mean of x and y so i just switch it to 0.0 0.1 resp)
(For private part, i am sure i put a rigth path)
Any suggestion?

[politza]

Thank you for debugging this. It very much looks like a bug in the synctex library and I updated it to the most recent version (310d88e). Please see if this makes it go away.

[allmightty]

Although i reinstall pdf-tools from Elpa ,i still got symptoms : pdf-info-query: The epdfinfo server quit unexpectedly. The epdfinfo server quit, restart it ? (y or n) y epdfinfo: Unable to create synctex scanner, did you run latex with --synctex=1' ? [2 times]
pdf-info-query: The epdfinfo server quit unexpectedly.`
sort of things
(sometimes it work, but mostly it error like above)

If you let me know what information you need (Since i am a newbie i don't know much so plz hope you let me know many details) , then i will give it to you as far as i can
(perhaps it's not a problem of pdftools. Is there any package i have to reinstall in arch system? like dependencies. Let me know then i will do to erase sophisticated situation)

and i wrote
\documentclass[12pt,a4paper]{article} \synctex=1
on the top of my document. so i didn't exatly run latex with --synctex=1 as epdfinfo wanted.
(but fine until upgrade version so i don't think it matters)

add:
When i change to default engine (pdflatex) then all the errors are gone.
So luatex is really a core of this problem. (IMHO)

[politza]

Things to check:

1. Are you sure that you have recompiled the epdfinfo program ?

2. Is there a pdfname.synctex.gz file beside the pdfname.pdf file ?

There was no guaranty that updating the synctex files would solve this problem. I think I'll have to get my hand on luatex and see for myself.

Edit:
luatex is indeed creating a synctex file which the synctex parser can't handle. It tries to read the file, but bails out soon after because it expects a different format (maybe due to a bug in the parser itself). Here is its output.

SyncTeX ERROR: Bad glue record (2).
SyncTeX ERROR: Bad sheet content.
SyncTeX ERROR: SyncTeX Error: Bad content

[politza]

There is a similar bug report for Okular and I tried to build with the synctex version they are using. But it won't build (maybe due to interface changes), so needs more work.

[allmightty]

There ispdfname.synctex.gzbeside the pdfname.tex , pdfname.pdf.
and i don't know about recompiling epdfinfo
Is there any easy way i can do?

And so you suspect luatex engine is problem? or synctex is problem? or pdf-tools is problem?
(cause i can't understand what you wrote)

[ytzemih]

I don't know how far you have gotten with the solution to that issue. Just wanted to refer to a very similar problem that I posted on SX some time ago.

Summary: I am having a very similar issue using Luatex/Syntex with an Evince/Zathura integration with Emacs on Ubuntu 18.04.2 LTS. It particularly often happens with long documents. I haven't been able to do anything about it (also not an expert in debugging foreign packages on Linux). I am editing short documents (<20pages) where the problem really rarely happens such that I can get away with it. But for long documents, the Emacs integration with those viewers gets impractical.

Thanks for looking into it again.

[weirdNox]

I am using Arch Linux and I was also affected by this problem.
After updating the synctex files to the ones in the 2019 branch, the problem went away!
There are some modifications needed, related to changed names of typedefs and functions (everything's described here: https://github.com/jlaurens/synctex/blob/2019/synctex_parser_readme.md).

I am now able to use this without any problems with LuaLaTeX :)

[politza]

I already did that here: #527 . Please try it and report if it works or not.

[weirdNox]

I downloaded your version and it seems to be working much better. I checked the modifications and, as you said, it has everything I asked you to do; that must be it.

At least, for me, it doesn't seem to crash anymore!

Thanks :)

[NightMachinery]

I get this pdf-info-query: epdfinfo: Unable to create synctex scanner, did you run latex with `--synctex=1' when I double-click on a word to select it. Is it related to this? Should I switch branches on pdf-tools to the PR #527?