Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.

HashCode based on xxHash32 #14863

Merged
merged 10 commits into from
Nov 16, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/mscorlib/Resources/Strings.resx
Original file line number Diff line number Diff line change
Expand Up @@ -3691,4 +3691,7 @@
<data name="Memory_OutstandingReferences" xml:space="preserve">
<value>Release all references before disposing this instance.</value>
</data>
<data name="HashCode_EqualityNotSupported" xml:space="preserve">
<value>HashCode is a mutable struct and should not be compared with other HashCodes.</value>
</data>
</root>
132 changes: 66 additions & 66 deletions src/mscorlib/shared/System/HashCode.cs
Original file line number Diff line number Diff line change
Expand Up @@ -67,18 +67,18 @@ public struct HashCode
private static unsafe uint GenerateGlobalSeed()
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benaadams In the future, maybe you would be interested in making sure the JIT makes this a constant?

{
uint result;
Interop.GetRandomBytes((byte*)&result, sizeof(int));
Interop.GetRandomBytes((byte*)&result, sizeof(uint));
return result;
}

public static int Combine<T1>(T1 value1)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @terrajobst. I thought the only work this method should do was

=> value1?.GetHashCode() ?? 0;

Copy link
Author

@jcdickinson jcdickinson Nov 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the proposal:

we might also want public static int Combine(T1 value);. I know it looks a little funny, but it would provide a way of diffusing bits from something with a limited input hash space. For example, many enums only have a few possible hashes, only using the bottom few bits of the code. Some collections are built on the assumption that hashes are spread over a larger space, so diffusing the bits may help the collection work more efficiently.

Added a comment to explain the intent.

{
// Provide a way of diffusing bits from something with a
// limited input hash space. For example, many enums only
// have a few possible hashes, only using the bottom few bits
// of the code. Some collections are built on the assumption that
// hashes are spread over a larger space, so diffusing the bits
// may help the collection work more efficiently.
// Provide a way of diffusing bits from something with a limited
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: double space a limited

// input hash space. For example, many enums only have a few
// possible hashes, only using the bottom few bits of the code. Some
// collections are built on the assumption that hashes are spread
// over a larger space, so diffusing the bits may help the
// collection work more efficiently.

var hc1 = (uint)(value1?.GetHashCode() ?? 0);

Expand Down Expand Up @@ -323,33 +323,30 @@ public void Add<T>(T value, IEqualityComparer<T> comparer)
[MethodImpl(MethodImplOptions.AggressiveInlining)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty big method - in particular with all the force-inlining on the smaller methods that it calls. ForceInlining is very likely going to hurt performance of real workloads.

private void Add(int value)
{
// xxHash works as follows:
// 0. Initialize immediately. We can't do this in a struct (no default
// ctor).
// The original xxHash works as follows:
// 0. Initialize immediately. We can't do this in a struct (no
// default ctor).
// 1. Accumulate blocks of length 16 (4 uints) into 4 accumulators.
// 2. Accumulate remaining blocks of length 4 (1 uint) into the hash.
// 2. Accumulate remaining blocks of length 4 (1 uint) into the
// hash.
// 3. Accumulate remaining blocks of length 1 into the hash.

// There is no need for *3 as this type only accepts ints. _queue1,
// _queue2 and _queue3 are basically a buffer so that when ToHashCode is
// called we can execute *2 correctly. That's what first three case
// statements do.
// There is no need for #3 as this type only accepts ints. _queue1,
// _queue2 and _queue3 are basically a buffer so that when
// ToHashCode is called we can execute #2 correctly.

// We need to initialize the xxHash32 state (_v1 -> _v4) lazily (see *0)
// and the last place that can be done if you look at the original code
// is just before the first block of 16 bytes is mixed in. The xxHash32
// state is never used for streams containing fewer than 16 bytes.
// We need to initialize the xxHash32 state (_v1 to _v4) lazily (see
// #0) nd the last place that can be done if you look at the
// original code is just before the first block of 16 bytes is mixed
// in. The xxHash32 state is never used for streams containing fewer
// than 16 bytes.

// A bloom filter is used to determine whether the default case statement
// has even been executed. To do that we check if the length is smaller
// than 4 (_length ^ position will be non-zero if this is the case). The
// case statement is for values larger than 2, so the check only succeeds
// on exactly 3.

// To see what's really going on here, have a look at the Combine methods.
// To see what's really going on here, have a look at the Combine
// methods.

var val = (uint)value;
uint position = _length % 4;
uint previousLength = _length++;
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storing this locally shaves off many bytes in the final code.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's important, then please add a comment about it.

uint position = previousLength % 4;

// Switch can't be inlined.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is quite a shame. @AndyAyersMS is this a problem with the JIT or something that's expected?

If this is in fact to be expected, then you could consider avoiding a branch for cases 2 and 3 below by doing a binary search-type thing:

if (position % 2 == 0)
{
    if (position == 0) { ... }
    else { Debug.Assert(position == 2); ... }
}
else
{
    if (position == 1) { ... }
    else { Debug.Assert(position == 3); ... }
}

This would actually penalize for the position == 0 case since it would add an additional branch check. However, I believe the number of branches should decrease on average.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But try my suggestion on goto below first

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually reduced performance. I think we are at the bottom of the pit of optimization and won't get anything more out of this. Further improvements will probably come from RyuJIT.


Expand All @@ -359,75 +356,78 @@ private void Add(int value)
_queue2 = val;
else if (position == 2)
_queue3 = val;
else // == 3
else // position == 3
{
// length smaller than 4?
if ((_length ^ position) == 0)
if (previousLength == 3)
Initialize(out _v1, out _v2, out _v3, out _v4);

_v1 = Round(_v1, _queue1);
_v2 = Round(_v2, _queue2);
_v3 = Round(_v3, _queue3);
_v4 = Round(_v4, val);
}

// Throw for more than uint.MaxValue fields.
_length = checked(_length + 1);
}

public int ToHashCode()
{
uint position = _length % 4;
uint length = _length;
uint position = length % 4;

// If the length is less than 3, _v1 -> _v4 don't contain
// anything yet. xxHash32 treats this differently.
// If the length is less than 4, _v1 to _v4 don't contain anything
// yet. xxHash32 treats this differently.

uint hash = (_length ^ position) == 0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: uint hash = length < 4 ? MixEmptyState() : MixState(_v1, _v2, _v3, _v4)

? MixEmptyState()
: MixState(_v1, _v2, _v3, _v4);
uint hash;
if (length < 4)
hash = MixEmptyState();
else
hash = MixState(_v1, _v2, _v3, _v4);

// Multiply by 4 because we've been counting in ints, not
// bytes.
// Multiply by 4 because we've been counting in bytes, not ints.

hash += _length * 4;
hash += length * 4;

// Mix what remains in the queue
// Switch can't be inlined right now, so use as few
// branches as possible instead.

if (position > 0)
{
hash = QueueRound(hash, _queue1);
if (position > 1)
{
hash = QueueRound(hash, _queue2);
if (position > 2)
hash = QueueRound(hash, _queue3);
}
}
// Switch can't be inlined right now, so emulate case statement
// fallthrough using goto.

// position refers to the *next* queue position in this method, so
// position == 1 means that _queue1 is populated; _queue2 would have
// been populated on the next call to Add.

if (position == 0) goto mixFinal;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has the same effect as the nested-ifs you had earlier, actually. I had something different in mind for goto, but now I don't think it will let us get the same perf as if we could use switch. You can revert this part to how it was before.

Have you tried marking ToHashCode with AggressiveInlining and using switch? Does switch generate more code than ifs?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to my tests, the JIT will never inline a method with switch, regardless of AggressiveInlining.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is a limitation in the current inliner.

hash = QueueRound(hash, _queue1);
if (position == 1) goto mixFinal;
hash = QueueRound(hash, _queue2);
if (position == 2) goto mixFinal;
hash = QueueRound(hash, _queue3);

mixFinal:
hash = MixFinal(hash);
return (int)hash;
}

# pragma warning disable 0809
#pragma warning disable 0809
// Obsolete member 'memberA' overrides non-obsolete member 'memberB'.
// Disallowing GetHashCode is by design
// Disallowing GetHashCode and Equals is by design

// * We decided to not override GetHashCode() to produce the hash code
// as this would be weird, both naming-wise as well as from a behavioral
// standpoint (GetHashCode() should return the object's hash code, not
// the one being computed).
// as this would be weird, both naming-wise as well as from a
// behavioral standpoint (GetHashCode() should return the object's
// hash code, not the one being computed).

// * Even though ToHashCode() can be called safely multiple times on this
// implementation, it is not part of the contract. If the implementation
// has to change in the future we don't want to worry about people who
// might have incorrectly used this type.
// * Even though ToHashCode() can be called safely multiple times on
// this implementation, it is not part of the contract. If the
// implementation has to change in the future we don't want to worry
// about people who might have incorrectly used this type.

[Obsolete("Use ToHashCode to retrieve the computed hash code.", error: true)]
[Obsolete("HashCode is a mutable struct and should not be compared with other HashCodes.", error: true)]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full error message should be "HashCode is a mutable struct and should not be compared with other HashCodes. Use ToHashCode to retrieve the computed hash code."

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the exception message too? Or just the obsolete message?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think some developers are going to mistakenly try to call GetHashCode instead of ToHashCode. It'd be worth including "Use ToHashCode to retrieve the computed hash code." in both the obsolete and exception message for GetHashCode as a hint of what to do.

Copy link

@jamesqo jamesqo Nov 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jcdickinson I would suggest making leaving off the Use ToHashCode ... for the Equals obsolete message and its exception message.

[EditorBrowsable(EditorBrowsableState.Never)]
public override int GetHashCode() => throw new NotSupportedException();
# pragma warning restore 0809
public override int GetHashCode() => throw new NotSupportedException(SR.HashCode_EqualityNotSupported);

[Obsolete("HashCode is a mutable struct and should not be compared with other HashCodes.", error: true)]
[EditorBrowsable(EditorBrowsableState.Never)]
public override bool Equals(object obj) => throw new NotSupportedException(SR.HashCode_EqualityNotSupported);
#pragma warning restore 0809
}
}