Character mangling in APIS XML

[ryanfb]

Many APIS XML files have � (U+FFFD REPLACEMENT CHARACTER) in the XML (often in the translation div, but also in other fields as well).

e.g.:

• https://github.com/papyri/idp.data/blob/c73ba9b762eea832636b605f6ee2783fd481619e/APIS/berkeley/xml/berkeley.apis.9382.xml = https://github.com/papyri/idp.data/blob/c73ba9b762eea832636b605f6ee2783fd481619e/APIS/berkeley/intake_files/berkeley.apis.9382.if (http://papyri.info/apis/berkeley.apis.9382)
• https://github.com/papyri/idp.data/blob/c73ba9b762eea832636b605f6ee2783fd481619e/APIS/michigan/xml/michigan.apis.2445.xml = https://github.com/papyri/idp.data/blob/c73ba9b762eea832636b605f6ee2783fd481619e/APIS/michigan/intake_files/michigan.apis.2445.if (http://papyri.info/ddbdp/p.mich;8;467)

These correspond to other characters in the intake files. In some cases, there appears to already have been some mangling in the intake files - in the second example above, the characters are already nonsense.

Ideally we should fix this in the crosswalk and then re-run to restore all characters mangled by the crosswalk into the XML. There may still be some cleanup to do after this, as in the second example. It may also be that certain codepoints in the intake would be better off removed by the crosswalk or replaced with a different character.

A list of all files with U+FFFD in the APIS directory: apis_mangling.txt