Larger Vim plugin help file cannot be highlighted

[rhysd]

Describe the bug

I'm maintainer of vim-lsp. The documentation is not highlighted on GitHub although its mode line description at bottom of the file is correct.
To reproduce this, open https://github.com/prabirshrestha/vim-lsp/blob/master/doc/vim-lsp.txt in your browser and check if highlight is applied.

Expected behaviour

Highlight is applied like this

Related discussion

Today I investigated why the document is not highlighted on GitHub.

The regex for heuristic matched correctly to modeline at the bottom of the document file.

However, the document file is larger than 51200 bytes. And the heuristic detection has 51200 bytes limit on finding the regex pattern. I understood that moving the modeline to top of the document file will make the heuristic detection work, but it's not possible because the format of Vim help file does not allow it. The first line is treated as summary of the document by Vim so modeline cannot be put at the line. On the other hand, Vim's mode line must be put on the first line or the last line of the file (please see :help modeline for more details). After all, modeline must be put at the last line in Vim help file. It means that Vim help files larger than 51200 bytes cannot be highlighted currently.

Additional notes

I think this issue is important for Vim plugin users because larger document is harder to be read and searched. Highlight would help to do them, but currently highlight is not applied due to this issue for larger documents.

[obcat]

Vim's mode line must be put on the first line or the last line of the file

I don't think this is true.

The :set line can be put at, for example, line 3 to be recognized as a modeline unless the value of the modelines option (default: 5) is less than 3. Please see :help 'modelines'.

[rhysd]

Yes, that's correct. But it's up to user's configuration. For example, when setting the value to 1, what I said is true. Since Vim help file cannot assume some user setting is set to specific value, I can't say putting the modeline at line 3 is always ok.

And also, putting modeline at line 3~5 is not good in terms of Vim's help file format. Usually table of contents is put at top of the file. Putting mode line just before the table looks weird.

[obcat]

Yes, that's correct. But it's up to user's configuration. For example, when setting the value to 1, what I said is true. Since Vim help file cannot assume some user setting is set to specific value, I can't say putting the modeline at line 3 is always ok.

You are right. I should have said "I don't think this is not always true".

And also, putting modeline at line 3~5 is not good in terms of Vim's help file format. Usually table of contents is put at top of the file. Putting mode line just before the table looks weird.

Well yes, I think so too. Anyway, it would be nice to have modeline at bottom recognized on GitHub, regardless of file size.

[lildude]

🤔 This should work because we look for the modeline in the first and last five lines, regardless of file size:

https://github.com/github/linguist/blob/32ec19c013a7f81ffaeead25e6e8f9668c7ed574/lib/linguist/strategy/modeline.rb#L129-L149

If this is making it though to the heuristics, it means your modeline didn't match the Vim modeline-specific regex at https://github.com/github/linguist/blob/32ec19c013a7f81ffaeead25e6e8f9668c7ed574/lib/linguist/strategy/modeline.rb#L52-L125

The modeline regex is also quite complex compared to the heuristic. Maybe it's missing something or your modeline is relying on old incorrect behaviour that is still accepted by the heuristic regex. I know @Alhadis spent a great deal of time to improve the modeline regex, with a good overview of the shortcomings, recently in #5271, so that might be a good place to start.

[lildude]

Worked it out. Your modeline isn't matching on the modeline strategy because you're specifying the file type/language as help but Linguist doesn't have a language or alias called "Help".

Changing your modeline to "vimhelp" however does work because it now matches the language alias:

https://github.com/github/linguist/blob/32ec19c013a7f81ffaeead25e6e8f9668c7ed574/lib/linguist/languages.yml#L6099-L6104

... as can be seen in my fork at https://github.com/lildude/vim-lsp/blob/master/doc/vim-lsp.txt (I'll delete this fork at some point).

When you move the modeline up into the range of the heuristic limits, the heuristic matches because it explicitly looks for "help" and matches that to "Vim Help File":

https://github.com/github/linguist/blob/32ec19c013a7f81ffaeead25e6e8f9668c7ed574/lib/linguist/heuristics.yml#L573-L574

Scroll all the way to the right and you'll see (?:filetype|ft|syntax)[ \t]*=help(?=[ \t]|:|$)

The final solution here would probably be to add help as an alias to the "Vim Help File" language entry.

[Alhadis]

The final solution here would probably be to add help as an alias to the "Vim Help File" language entry.

This was a deliberate omission on my part. Given the specificity of Vim's help format, I was a little reluctant to add an alias as generic as help out of concerns about misclassification. In hindsight, I may have been overly cautious.

Also, I completely forgot about the Vim Help heuristic when preparing #5271. 🥵 That'll need to be D.R.Y-ed up.