UTF8 encoding is lost

[orinem]

Describe the bug 🪲

When using an API that takes a character string, for example creating an alarm, the API will fail.

To Reproduce 🪜

Try to create an alarm named FooÀ.

Expected behavior

The alarm should be created and displayed correctly

AMT Device (please complete the following information): 🖥️

• OS: N/A
• AMT Version: 16
• AMT Configuration Mode: ACM
• Network Configuration: Dynamic

Service Deployment (please complete the following information): ⛈️

• Deployment Type: Docker
• Node Version: 16
• Component & Version: MPS 2.6.0

Additional context

Although the string sent to the device is UTF8 encoded into a Buffer in HttpHandler.wrapIt(), it implicitly gets decoded when appended to the message string since the default parameter to Buffer.toString() is 'utf8'.

In addition, the length of the javascript string is used, rather than the length of the UTF8 encoded string.

Although inefficient, the easiest workaround is to explicitly convert back from Buffer using Buffer.toString('binary'). When eventually converted to a buffer, the string returned from wrapIt() is treated as 'binary' so no further changes are necessary.

Making the above changes allows the alarm to be created, but it displays incorrectly. This is because the xml parser ignores the encoding. A very inefficient fix is: xmlToParse = Buffer.from(xml, 'binary').toString('utf8') in HttpHandler.parseXML().

diff --git a/src/amt/HttpHandler.ts b/src/amt/HttpHandler.ts
index 60ce395..1ad2453 100644
--- a/src/amt/HttpHandler.ts
+++ b/src/amt/HttpHandler.ts
@@ -37,7 +37,7 @@ export class HttpHandler {
     try {
       const url = '/wsman'
       const action = 'POST'
-      let message = `${action} ${url} HTTP/1.1\r\n`
+      let message: any = `${action} ${url} HTTP/1.1\r\n`
       if (data == null) {
         return null
       }
@@ -62,6 +62,7 @@ export class HttpHandler {
         })
         message += `Authorization: ${authorizationRequestHeader}\r\n`
       }
+      /*
       // Use Chunked-Encoding
       message += Buffer.from([
         `Host: ${connectionParams.guid}:${connectionParams.port}`,
@@ -72,7 +73,16 @@ export class HttpHandler {
         0,
         '\r\n'
       ].join('\r\n'), 'utf8')
+      console.log(message)
       return message
+      */
+      const dataBuffer = Buffer.from(data, 'utf8')
+      message += `Host: ${connectionParams.guid}:${connectionParams.port}\r\nContent-Length: ${dataBuffer.length}\r\n\r\n`
+      message = Buffer.concat([Buffer.from(message, 'utf8'), dataBuffer])
+      if (dataBuffer.length !== data.length) {
+        logger.silly(`wrapIt data length mismatch: Buffer.length = ${dataBuffer.length}, string.length = ${data.length}`)
+      }
+      return message.toString('binary')
     } catch (err) {
       logger.error(`${messages.CREATE_HASH_STRING_FAILED}:`, err.message)
       return null
@@ -108,7 +118,8 @@ export class HttpHandler {
 
   parseXML (xmlBody: string): any {
     let wsmanResponse: string
-    this.parser.parseString(xmlBody, (err, result) => {
+    const xmlDecoded: string = Buffer.from(xmlBody, 'binary').toString('utf8')
+    this.parser.parseString(xmlDecoded, (err, result) => {
       if (err) {
         logger.error(`${messages.XML_PARSE_FAILED}:`, err)
         wsmanResponse = null

Ideally, once UTF8 encoded, only a Buffer object should be used, but that requires more extensive changes. The switch to using "Content-Length" rather than chunked encoding makes the fix easier (saves 19 bytes on the wire, but breaks the CIRAChannel test - fix below).

diff --git a/src/amt/CIRAChannel.test.ts b/src/amt/CIRAChannel.test.ts
index 945a3fe..83f54f9 100644
--- a/src/amt/CIRAChannel.test.ts
+++ b/src/amt/CIRAChannel.test.ts
@@ -85,7 +85,9 @@ describe('CIRA Channel', () => {
   })
   it('should resolve if data does not contain messageId', async () => {
     ciraChannel.state = 2
-    ciraChannel.sendcredits = 116
+    // use sendCredits = 116 if using chunked encoding;
+    // extra characters are: "ingChunked\r\n\r\n0\r\n\r\n"
+    ciraChannel.sendcredits = 97
     const data = 'KVMR'
     const params: connectionParams = {
       guid: '4c4c4544-004b-4210-8033-b6c04f504633',