Exception on exporting files with accented characters in names

[dehall]

Observed the following exception on a CentOS system with a NAS (I don't know the full tech specs of the file system)

java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: Mar?a_Elena653_Duran646_482f31eb-b9f2-4d32-93c3-28e021c29160.json
        at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
        at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
        at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
        at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
        at org.mitre.synthea.export.Exporter.exportRecord(Exporter.java:113)
        at org.mitre.synthea.export.Exporter.export(Exporter.java:52)
        at org.mitre.synthea.engine.Generator.generatePerson(Generator.java:406)
        at org.mitre.synthea.engine.Generator.lambda$3(Generator.java:242)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Many of the "Spanish" names in Synthea use accented characters (à, ò, etc) and while I've never seen this exception on Mac or Windows, apparently some file systems do not support them. We should consider sanitizing the file names to be pure ASCII to be safe.

[shabiel]

My personal vote is against that as all file systems should support UTF-8 or UTF-16 nowadays. There is probably a setting to enable this. If you "sanatize" the output, you will have problems in the future supporting Synthea with non-English speakers. If you must, you can add a flag to do that, but I am still kind of surprised that there would be file systems which only support ASCII.

…

--Sam

On Thu, Sep 12, 2019 at 7:21 AM Dylan Hall ***@***.***> wrote: Observed the following exception on a CentOS system with a NAS (I don't know the full tech specs of the file system) java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: Mar?a_Elena653_Duran646_482f31eb-b9f2-4d32-93c3-28e021c29160.json at sun.nio.fs.UnixPath.encode(UnixPath.java:147) at sun.nio.fs.UnixPath.<init>(UnixPath.java:71) at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281) at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53) at org.mitre.synthea.export.Exporter.exportRecord(Exporter.java:113) at org.mitre.synthea.export.Exporter.export(Exporter.java:52) at org.mitre.synthea.engine.Generator.generatePerson(Generator.java:406) at org.mitre.synthea.engine.Generator.lambda$3(Generator.java:242) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Many of the "Spanish" names in Synthea use accented characters (à, ò, etc) and while I've never seen this exception on Mac or Windows, apparently some file systems do not support them. We should consider sanitizing the file names to be pure ASCII to be safe. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jawalonoski]

I am in agreement with @shabiel on this one.

If it is really an issue for certain systems, you can use UUID filenames by changing synthea.properties:

exporter.use_uuid_filenames = true