Thanks @IanMayo. I've been looking through some of the Java code for this format, and I think it will be helpful when implementing it in Python - particularly if we can translate some of the unit tests.

Quick question: it seems like the Debrief code supports imports from .doc, .docx and .pdf files. Do we need to support all of those formats? Or as .doc is very old now, can we ignore that one? (I suspect the answer is that we still need to support it, but I thought I'd ask!)

Sure. Yes - it would be acceptable to not support .doc. But, I know some platforms convert to .pdf before submitting their data.

@IanMayo Do you know how the hidden text is managed in the PDF versions of these documents? I've just done a simple export of the example document to PDF (from within Word), and we just get the normally visible text:

This means that we don't have any information on the date for each line - and no information seems to be given about the date anywhere else either. Even if I change the Word settings to display hidden text, it doesn't seem to be exported into the PDF.

Any thoughts?

Also, if you're able to get an example document containing the FCS ('contact') entries then that would be great. I've found various example test entries in the Java unit tests, but it'd be good to have a proper document.

@IanMayo Do you know how the hidden text is managed in the PDF versions of these documents? I've just done a simple export of the example document to PDF (from within Word), and we just get the normally visible text:
This means that we don't have any information on the date for each line - and no information seems to be given about the date anywhere else either. Even if I change the Word settings to display hidden text, it doesn't seem to be exported into the PDF.

I've just looked at my sample data. I do have a sample PDF, and see that it is missing the hidden data.
FCS_narrative_no_metadata.pdf

Here's a MS-Word document with FCS (Fire Control Solution) data:
FCS_extra_narrativetypes.doc.zip

@IanMayo How do you suggest we handle PDFs that don't have the hidden fields in them? Without the hidden fields we don't have dates to use for the timestamps.

While the unicorn example doesn't include it, I have seen month/year markers.

I'll check with the clients if these markers were present in the unicorn example that got sanitised.

Hi @IanMayo - a few more questions about this importer:

1. Some of the example Word documents I found don't seem to have a header. Given that, do we need to be able to handle Word docs without a header? That would mean no information on privacy, or vessel name - so we'd need to prompt for them.

2. I found some code in the Java implementation that handles the month/year markers that you mentioned in your previous comment. I'll make sure I support those.

3. The Java code deals with situations where the final comma after the Message Type field is missed out, and therefore the Message Type field runs straight into the actual text of the message. However, it does this by looking for the first space and splitting there (see https://github.com/debrief/debrief/blob/develop/org.mwc.debrief.legacy/src/Debrief/ReaderWriter/Word/ImportNarrativeDocument.java#L485). The example file we have has message types with spaces in them (CO Comment etc), so this won't work properly. In my code I look for a tab instead (as that's what we seem to have splitting the two fields) - but this might need updating in the Java code.

4. The messages that include the State information (eg. 58 34N 001 36W 089@12) - would this information always be the full content of a message, or could it be included in a longer message (eg. Blah blah we did something 58 34N 001 36W 089@12)? I can't seem to see any Java code that processes this, and I think I'm going to need to play around with regexes to try and match it. Unfortunately there doesn't seem to be a specific Message Type that contains this information - it would be really useful if there was, but oh well.

1. Header. I'll ask the question. There's a chance the sample files are missing the header just because they're mock data.

2. Yes. I had confirmation last night that for each file the analyst looked in, the content we're looking for started with the day/month/year header. I saw documents a couple of years ago where that marker didn't appear until after a hundred pages of other content. But, since the documents are "tidied" by a human before being sent - we have to remain wary that the marker may be missing, and that we have to ask the user for the month/year.

3. Tab separator. Thanks for the tip.

4. I am surprised to see that location in the General Comment. I don't think I've seen it before. Knowing location is really valuable, but normally the ships rely on the (multiple) other systems to capture it. So. Yes, let's grab this State element when it appears in a General Comment. But, we need to be quite relaxed in the formatting, since it's free-typed into MS-Word. (Many other inputs are entered via popup dialogs, which gives some consistency in formatting). So, we need to allow for the presence of a degree symbol, the presence of seconds. The last two elements are course (degs) and speed (kts). I guess we need to allow for Kts or kts being present too. Actually - I'll ask them to look at a wider sample, and see how often location is supplied in this way.

Thanks @IanMayo.

For item 4: yes, the more examples we can get of how location is specified, the better. I can see this being a bit tricky to parse reliably, so lots of examples will help tune the regex.

1. Header. All of the narrative documents found had the matching header expressed as a table. But, there were also some "composite" documents, where the narrative is included as an appendix, missing the narrative header.

There may be merit in combining this work with any client-code that parses/handles narrative files.

In that way we can combine the Pepys import logic from this work with the special case handling of the client-code.