[IMPROVED] Reduce allocations in writeMsgRecord

[neilalexander]

The following changes are all to reduce GC pressure:

• The hdr array looked like it should be stack-allocated but was actually always escaping to the heap because of the highwayhash writer, so instead of allocating a ton of those, just preallocate space on the underlying cache buffer instead (since that has already escaped to the heap);
• Reuse the memory of the last checksum instead of allocating unnecessarily there too, and use stringToBytes for the subject, avoiding a further copy.

Signed-off-by: Neil Twigg neil@nats.io

[derekcollison]

LGTM

[neilalexander]

Looks like TestJetStreamClusterConsumerLeak is unhappy with this change, although the amount seems to vary quite a bit (appears to vary with how much RAM the system has — the result is much worse on my 128GB RAM machine than it is in the 32GB CI worker, but fails either way).

[neilalexander]

I've backed out one of the changes as I think it may change baseline memory usage and I want to spend more time on that one before committing to it.

[derekcollison]

ok let's keep watching and testing. The pools work but can cause memory growth quite quickly in this situation vs what was there before.

[neilalexander]

Indeed, the other remaining changes in this PR will still reduce allocations and GC pressure in the meantime.

[wallyqs]

LGTM

[derekcollison]

LGTM