Add support for new Mathematical Web Fonts

[fred-wang]

No description provided.

[pkra]

@fred-wang Where should I leave bug reports for your webfont branch?

[pkra]

@fred-wang here's a bug with jquery-mobile and your open-type branch creating too much whitespace after a math expression http://jsfiddle.net/8AmED/. This doesn't happen with v2.2.

[fred-wang]

On Chrome the TeX source is not parsed and on Firefox I see image fonts... is it a security restriction or something? Also, note that the branch is not based on the final version of v2.2 but on an earlier version of Davide's develop branch (before localization changes) so some fixes may not have been integrated yet.

[pkra]

Ok, didn't know that branch wasn't up-to-date. Will you merge v2.2 in anytime soon?

I didn't see any problem on Chrome (it as slow, but not too bad); Firefox will get stuck at missing CORS headers from github. I can zip it up or you can just download jquery-mobile ;).

[dpvc]

I've gone through the fontdata file for STIX-web pretty carefully, and here is what I've found:

• There are references in the DELIMITERS array to Size4 characters U+23AA, U+23B0, and U+23B1, but these are no in that font. U+23AA seems to be in Size5 (as U+E00D), but the other two are missing entirely (it looks like they may not be in the 5-font version of STIX -- at least I can't find them with a quick look). If these values come from the OpenType Math table, then perhaps the table is wrong?

• -STIX-variant should be -STIX-Web-variant (the variant name is constructed from the HTML-CSS fontInUse value).

• The variants are not being handled properly. This is because the origin STIX font had a STIXVariants font to hold thee variants, and so -STIX-variant simply switched to that font and the variants were obtained automatically. Now, however, many of the variants are in the STIX_NonUnicode font, and not at their assigned Unicode points. So the remap table would need to be enlarged to properly map the regular positions to the PUA positions. Unfortunately, most of the characters don't have CMAP entries, so can't be obtained from STIX_NonUnicode. For example, the large-sized primes (used in f' for example), and the h with a straight bar (used for \hbar) are not available.

• It looks like the extra 3 MathJax glyphs are not in the NonUnicode fonts, so they aren't being detected properly (MathJax loads them, but times out trying to detect them).

• There is no need for TTF versions in the fonts/HTML-CSS directories, as the other forms cover all the browsers (this is only used as an intermediate form for use with TTX and for obtaining the EOT files for IE). Technically, we probably don't need SVG any longer, either (other than as an intermediate for producing the data for the SVG output jax), as this was only needed for early versions of Chrome, and (if I remember correctly) early versions of iOS.

• The DELIMITER data for U+20D0, U+20D1, U+20D6, U+20E1, U+20EC to U+20EF, U+2140, U+220F to U+2211, U+221B, U+221C, U+222B to U+2233, U+22C0 to U+22C3, U+23AA, U+23AF, U+23D0, U+2772, U+2773, U+27F0, U+27F1, U+2983 to U+2986, U+2A00 to U+2AFF, U+2B45, and U+2B46 could go in fontdata-extra.js, as these are not commonly used.

• There are a couple of delimiters that could be made infinitely stretchy (but they have only fixed size data):

  • U+203E could be able to stretch infinitely (using itself as the extender)
  • U+222B could be made infinitely stretchy (top: U+2320, rep: U+23AE, bot: U+2321)
  • U+27E6 and U+27E7 could be infinitely stretchy using box drawing characters (U+2553, U+2551, U+2559, etc.)

• In the current setup, the script, fraktur, double-struck, and other mathvariants will only show the characters from those math alphabets; if any other character is requested, it will be an unknown character and use the default unknownFamily to obtain it. This is not how the current STIX (or TeX) fonts work. Because the original STIX fonts had these mathvariants in the same fonts as the rest of the font (e.g., Sans-Serif was in the STIXGeneral font), something like \mathsf{A+B} would properly take the plus sign from STIXGeneral-Regular, but in the STIX-Web fonts, it is taken from the unknownFamily, and the proper bounding box is not known.

  One way around this would be to add the normal fonts to the fonts list in these variants. Alternatively, I could modify the variant handler to look through the normal variant (or the italic, bold, or bold-italic variants, depending on the bold and italic properties of the current variant) if the character is not found in the current variant's font list. That might be the better solution (and would reduce the needed data in the font data file).

Those are the things that I see at the moment. I have not looked closely at the other web font data, but will do so.

[dpvc]

I found the problem with the centering of the large operators: the SVG code was using the wrong sign (but since the MathJax TeX fonts have these centered already, it didn't matter). There is also a change needed to munderover to handle the shifted position properly.

I also found the problem with the changes to the integral spacing: The SVG code doesn't use the rfix value, and the 5th item is actually the path data, so you were wiping out the path by setting it. Because the SVG output jax doesn't use the system's font code (it places the paths itself), you just have to modify the bounding box data (there is no need to compensate for the actual font size, since that is never used). The sizes you used also were not always large enough, so I've adjusted them.

Finally, the problem with the space in the monospace font is that the monospace fonts doesn't include the nonbreaking space (U+00A0), and so that was being taken from the Main font instead, which is of the wrong size. This could be fixed by adding the U+00A0 character to the monospaced font (probably a good idea), but for now, I've added a remapping of U+00A0 to U+0020 in the fontdata file.

The changes I made are in the open-type-fonts branch of my fork of MathJax. These are the differences.

[fred-wang]

OK thanks for looking into this. I think you can extract the changes to the SVG output jax and merge them into the develop branch (I'm not sure we really want the changes to avoid infinite loop when DELIMITERS is incorrect). The changes to the fontdata should be integrated in the fontdata-adjust.js files of the Python scripts that generate them. I'll look into this next Monday.

[dpvc]

Right. I only committed them to this branch in order for you to be able to see them. I didn't take time to figure out where the changes had to be made in your font-generation code.

I'll move the changes to the SVG output jax to the develop branch.

[fred-wang]

There are references in the DELIMITERS array to Size4 characters U+23AA, U+23B0, and U+23B1

These are not in the Open Type Math table but were in the old STIX fonts, so I've added them by hand:
https://github.com/fred-wang/MathJax-dev/blob/open-type-fonts/fonts/OpenTypeMath/STIX-Web/config.py#L72
(it seems to be the only extra stretchy ops whose pieces are non-unicode chars, so that may be a bug in the Python script)

-STIX-variant should be -STIX-Web-variant

Fixed.

There is no need for TTF versions in the fonts/HTML-CSS directories

OK, I've removed the SVG and TTF fonts.

The variants are not being handled properly.
looks like the extra 3 MathJax glyphs are not in the NonUnicode fonts

I didn't really take care of the NonUnicode fonts, currently they are basically just the "remaining" glyphs once we have moved all the others in smaller fonts. I'm wondering if the important characters that MathJax really uses should just be moved in existing/new fonts and the NonUnicode fonts be dropped. Or if some users are willing to use these characters anyway and so the NonUnicode should be provided with the appropriate CMAP...

The DELIMITER data for U+20D0, U+20D1,

I think these are from the Open Type Math table. Currently the script reads the table and moves everything in fontdata.js. Some config options could probably be added to tell whether some glyphs should go in extra or not.

In the current setup, the script, fraktur, double-struck, and other mathvariants

I think we should discuss again how the mathvariants/style are handled. I'd like to have something consistent between MathJax/browsers/MathML spec and between the various fonts ; that requires less workarounds and allows the Python script to handle the fonts uniformly (unfortunately, this ideal case is probably not possible but we can try and find the best situation)

There are a couple of delimiters that could be made infinitely stretchy

I think these are from the Open Type Math table and the script does not currently change/complete the values provided by the font designer. Again, a config option could be added to customized. Probably the changes should be reported to the STIX consortium.

Finally, the problem with the space in the monospace font

I've modified the script so that the monospace fonts have both the 0x20 and 0xA0 glyphs.

I also found the problem with the changes to the integral spacing

I've updated the fontdata-adjust.js files to import your changes.

[fred-wang]

There are references in the DELIMITERS array to Size4 characters U+23AA, U+23B0, and U+23B1, but these are no in that font.

OK, the glyphs were correctly copied in the Size5 of PUA but the data were not generated correctly (it used Size4 and didn't print the PUA code point). I've fixed that so that should work now. Note that U+23AA is no longer stretchy by default, though.

[dpvc]

I'm not sure why these three characters are being put in PUA when they have assigned Unicode values (U+23AA, U+23B0, and U+23B1). Why aren't they placed at those positions? And why is Size5 the place for them, when they are really at the Size1 (or Main) size? Why don't they go into one of those fonts?

[dpvc]

The variants are not being handled properly.
looks like the extra 3 MathJax glyphs are not in the NonUnicode fonts

I didn't really take care of the NonUnicode fonts, currently they are basically just the "remaining" glyphs once we have moved all the others in smaller fonts. I'm wondering if the important characters that MathJax really uses should just be moved in existing/new fonts and the NonUnicode fonts be dropped. Or if some users are willing to use these characters anyway and so the NonUnicode should be provided with the appropriate CMAP...

I'm fine with having the used characters moved to other fonts. But I wouldn't want to drop the others entirely. Someone is bound to say "Character XX is in the STIX fonts, but I can't find it in MathJax." If these not-often-used characters are in a separate font, then it should not hurt MathJax to have the data for them, as it will rarely be loaded, but will be there if someone needs it.

The DELIMITER data for U+20D0, U+20D1,

I think these are from the Open Type Math table. Currently the script reads the table and moves everything in fontdata.js. Some config options could probably be added to tell whether some glyphs should go in extra or not.

Don't you already have that here? It looks like they could just be added to this list, no?

There are a couple of delimiters that could be made infinitely stretchy

I think these are from the Open Type Math table and the script does not currently change/complete the values provided by the font designer. Again, a config option could be added to customized. Probably the changes should be reported to the STIX consortium.

The suggested changes are not critical.

[fred-wang]

I'm not sure why these three characters are being put in PUA when they have assigned Unicode values (U+23AA, U+23B0, and U+23B1). Why aren't they placed at those positions? And why is Size5 the place for them, when they are really at the Size1 (or Main) size? Why don't they go into one of those fonts?

Well, that's a very special case that is not handled "normally". The script copies the non-Unicode glyphs to the Size5 PUA, so I've kept the same procedure. These glyphs are also used as components so they will be copied to the Size5 PUA anyway. Again the solution would be to report that to the STIX consortium, so that they are directly specified in the Open Type Math table.

I'm fine with having the used characters moved to other fonts. But I wouldn't want to drop the others entirely. Someone is bound to say "Character XX is in the STIX fonts, but I can't find it in MathJax." If these not-often-used characters are in a separate font, then it should not hurt MathJax to have the data for them, as it will rarely be loaded, but will be there if someone needs it.

Currently they are in non-Unicode position, so not accessible to Web authors by normal means anyway. I agree with having those used by MathJax in specific fonts and the remaining not-often-used characters in a separate font.

Don't you already have that here? It looks like they could just be added to this list, no?

I think this is only for custom stretchy operators (If I remember correctly), I have to check if that works for those in the Open Type Math font or otherwise teach the script to handle that case.

[dpvc]

These glyphs are also used as components so they will be copied to the Size5 PUA anyway.

Even if they already exist at another location in another font? Is that required? For example, if the parenthesis pieces are moved to their correct locations in the Misc. Technical block, will you still be moving them to Size5 anyway?

Currently they are in non-Unicode position, so not accessible to Web authors by normal means anyway.

I guess you are referring to the Word version of the files, but the STIXGeneral versions had these in the PUA at U+E000 through U+E3CF, and these do have CMAP entries and are accessible through normal means. (Indeed, that is how MathJax can access them.)

[fred-wang]

Even if they already exist at another location in another font? Is that required? For example, if the parenthesis pieces are moved to their correct locations in the Misc. Technical block, will you still be moving them to Size5 anyway?

If the parenthesis pieces are moved to their correct Unicode locations, then they won't be in Size5 but directly accessed from the MISC font (for both HW and stretch) as it's currently the case, for example, for 0x27F0.

I guess you are referring to the Word version of the files

Yes, I was referring to STIX-Word.

[fred-wang]

Don't you already have that here? It looks like they could just be added to this list, no?

I stand corrected, the script is already able to handle that. I've updated the list.

[fred-wang]

I've update the MathJax-dev branch (not the MathJax branch, though)

It looks like the extra 3 MathJax glyphs are not in the NonUnicode fonts, so they aren't being detected properly (MathJax loads them, but times out trying to detect them).

I fixed that, but we'll need to decide how to split this font.

There are a couple of delimiters that could be made infinitely stretchy (but they have only fixed size data):

I've added them.

[fred-wang]

The variants are not being handled properly

Davide, do you have a list of characters that require a variant? I see they are in STIXVariant and STIXNonUnicode from the STIXGeneral set, but I don't know which are precisely used by MathJax.

[fred-wang]

(and additionally, that would help to know how I can check the variants are correctly picked)

[dpvc]

Here is the list of variants produced by the TeX input jax, together with the tex macros that produce them.

U+2032  \prime
U+2033  x''
U+2034  x'''
U+2035  \backprime
U+2036
U+2037
U+2057  x''''
U+210F  \hbar
U+21CC  \rightleftharpoons
U+2205  \varnothing
U+2216  \backslash
U+2216  \setminus
U+2223  \shortmid
U+2224  \nshortmid
U+2225  \shortparallel
U+2226  \nshortparallel
U+223C  \thicksim
U+2248  \thickapprox
U+2268  \lvertneqq
U+2269  \gvertneqq
U+2270  \nleqq
U+2271  \ngeqq
U+2288  \nsubseteqq
U+2289  \nsupseteqq
U+228A  \varsubsetneq
U+228B  \varsupsetneq
U+22E0  \npreceq
U+22E1  \nsucceq
U+2322  \smallfrown
U+2323  \smallsmile
U+25B3  \vartriangle
U+25BD  \triangledown
U+2A87  \nleqslant
U+2A88  \ngeqslant
U+2ACB  \varsubsetneqq
U+2ACC  \varsupsetneqq

One way to test them would be to see if they produce different symbols from the plain character (as produced by \unicode{xNNNN} in TeX, or by a regular entity reference in MathML).

[fred-wang]

OK, thanks. I've done a quick check and it seems that this does not correspond exactly to those available in STIX (some are not available, others are not used in MathJax). I guess I'll just move all the *.var glyphs into a separate Variant font as you suggested. I'll try to do the same that for the other fonts too.

FYI and if we want to consider that in the future, here is however a testcase to access them via font-feature-settings. That seems to work for me in Firefox and Chrome:

     <p style="-moz-font-feature-settings: 'ss03'; -webkit-font-feature-settings: 'ss03'; font-family: STIX;">w&#x007C;&#x019B;&#x0264;&#x2032;&#x2033;&#x2034;&#x2035;&#x2036;&#x2037;&#x2057;&#x210F;&#x2140;&#x2190;&#x2191;&#x2192;&#x2193;&#x21D1;&#x21D3;&#x21E0;&#x21E2;&#x2205;&#x2208;&#x2209;&#x220B;&#x220C;&#x220F;&#x2210;&#x2211;&#x2216;&#x221A;&#x221B;&#x221C;&#x221D;&#x2223;&#x2224;&#x2225;&#x2226;&#x2229;&#x222A;&#x223C;&#x223E;&#x223F;&#x2248;&#x224C;&#x2272;&#x2273;&#x2293;&#x2294;&#x2295;&#x2297;&#x229C;&#x22DA;&#x22DB;&#x2322;&#x2323;&#x2423;&#x25A9;&#x2A3C;&#x2A3D;&#x2A9D;&#x2A9E;&#x2AAC;&#x2AAD;&#x2ACB;&#x2ACC;&#x2AEE;</p>

[fred-wang]

I've updated my MathJax-dev branch. I found the variants in STIX-Word for most of these characters, except U+21CC \rightleftharpoons. The difference is not obvious for me with the local STIX fonts. I don't really see the difference for \thicksim and \thickapprox either (with the MathJax TeX fonts I have to zoom in to see that they are thicker).

[dpvc]

It may be that the STIX fonts don't have those variants (some of the variants were only in the TeX fonts, and I think \thicksim was one, so \thickapprox probably is as well).

[fred-wang]

I think STIX-Word has the \thickapprox and \thicksim variants, it's just that the visual difference is not obvious. The only one I could not found was \rightleftharpoons (but it was in STIX-General I think)

[dpvc]

It turns out that the \rightleftharpoons variant is in the MathJax TeX fonts, not the STIX fonts, so that is why it appeared in the list above. I don't see the difference with the \thicksim and \thickapprox either (though I do see that the variant character is available in STIXVariants, but it is not different from the original glyph). Again, these two are really there for the MathJax fonts, which does have a different glyph.

[fred-wang]

OK I'm pushing an updated version to the branch.

[fred-wang]

So I just had a look at the variants for the other fonts. It seems that the most important are prime symbols, since the default size is too small, except for Asana. The others (Gyre-*, Latin Modern, Neo Euler) have larger variants for the prime symbols, so I've added them. It seems that there are other variants but that does not really correspond to MathJax's need.

[fred-wang]

I've create a clean branch issue489. The only difference with the previous open-type-fonts should be the addition of a few variants for the non-STIX fonts.

I'll send a pull request.

[dpvc]

OK, I've merged this, and am closing this issue. Start new tracker issues for any new problems that are discovered.

=> Merged.

[fred-wang]

I'm not sure if that happened before and is now only visible now that the testing framework accept arbitrary fonts. But if you pass an invalid font, it seems that MathJax won't fallback to what was introduced in the #363 and *.FONTDATA may be null. See:

http://devel.mathjax.org/testing/testsuite/API/Hub/setRenderer-1.html?&mathJaxPath=http://devel.mathjax.org/mathjax/MathJax-develop/unpacked/&font=STIX&outputJax=HTML-CSS (will raise an error in the SVG that does not support STIX local)
http://devel.mathjax.org/testing/testsuite/API/Hub/setRenderer-1.html?&mathJaxPath=http://devel.mathjax.org/mathjax/MathJax-develop/unpacked/&font=xxx&outputJax=HTML-CSS (will raise an error in the HTML-CSS)

[fred-wang]

OK, this is not a regression since it happened with MathJax v2.2 too:
http://devel.mathjax.org/testing/testsuite/API/Hub/setRenderer-1.html?&mathJaxPath=http://cdn.mathjax.org/mathjax/latest/unpacked/&font=STIX&outputJax=HTML-CSS