Fix GetNormalizedLength on iOS and similar platforms #110723

tarekgh · 2024-12-15T20:59:12Z

Fixes #110720

tarekgh · 2024-12-15T21:00:32Z

/azp run runtime-androidemulator
/azp run runtime-ioslikesimulator,runtime-maccatalyst

azure-pipelines · 2024-12-15T21:00:41Z

Azure Pipelines successfully started running 1 pipeline(s).

ivanpovazan · 2024-12-16T09:10:06Z

Note: If #110725 gets merged in the meantime the tests should be reenabled

tarekgh · 2024-12-16T20:01:07Z

could someone paste the failure logs for the failures listed in https://dev.azure.com/dnceng-public/public/_build/results?buildId=895827&view=logs&j=a2a39d6b-70c5-5040-933a-4b8b6e52bf19&t=278c4d44-b3fb-5bc8-7d46-7d739aeb5a1f? I don't know how to reach these logs.

matouskozak · 2024-12-16T20:07:31Z

could someone paste the failure logs for the failures listed in https://dev.azure.com/dnceng-public/public/_build/results?buildId=895827&view=logs&j=a2a39d6b-70c5-5040-933a-4b8b6e52bf19&t=278c4d44-b3fb-5bc8-7d46-7d739aeb5a1f? I don't know how to reach these logs.

2024-12-15 17:22:51.154 Df System.Globalization.Extensions.Tests[17177:2fa76] 1) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ÄÇ", normalizationForm: FormD, expected: "ÄÇ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.154 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ÄÇ' to a buffer of length 3 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.154 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0016457
2024-12-15 17:22:51.154 Df System.Globalization.Extensions.Tests[17177:2fa76] 2) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ÄÇ", normalizationForm: FormC, expected: "ÄÇ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.155 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ÄÇ' to a buffer of length 1 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.155 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0002661
2024-12-15 17:22:51.155 Df System.Globalization.Extensions.Tests[17177:2fa76] 3) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormC, expected: "ẛ̣")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ẛ̣' to a buffer of length 1 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0002417
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76] 4) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormD, expected: "ẛ̣")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ẛ̣' to a buffer of length 2 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0001462
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76] 5) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ﬁ", normalizationForm: FormKC, expected: "fi")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ﬁ' to a buffer of length 1 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0006702
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76] 6) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ﬁ", normalizationForm: FormKD, expected: "fi")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ﬁ' to a buffer of length 1 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0001803
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76] 7) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormKD, expected: "ṩ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    Exception messages: Trying to normalize 'ẛ̣' to a buffer of length 2 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-15 17:22:51.156 Df System.Globalization.Extensions.Tests[17177:2fa76]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

tarekgh · 2024-12-16T20:17:09Z

@matouskozak thanks for pasting the logs. Did this failure occur before? Looks like it is different than what used to fail. Also, it looks like iOS legs succeeded. So, the failures are localized to maccatalyst only. Is that right? If so, any idea what the difference between it and iOS in term of is the support?

matouskozak · 2024-12-16T20:24:25Z

@matouskozak thanks for pasting the logs. Did this failure occur before? Looks like it is different than what used to fail. Also, it looks like iOS legs succeeded. So, the failures are localized to maccatalyst only. Is that right? If so, any idea what the difference between it and iOS in term of is the support?

On rolling builds, I see failing

2024-12-16 05:29:48.444 Df System.Globalization.Extensions.Tests[57174:a70cb] 1) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ÄÇ", normalizationForm: FormD, expected: "ÄÇ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-16 05:29:48.444 Df System.Globalization.Extensions.Tests[57174:a70cb]    Exception messages: Trying to normalize 'ÄÇ' to a buffer of length 3 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.444 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0013492
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb] 2) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ÄÇ", normalizationForm: FormC, expected: "ÄÇ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
   Exception messages: Assert.Equal() Failure: Values differ
Expected: 2
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb] Actual:   0   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0009139
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb] 3) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ﬁ", normalizationForm: FormC, expected: "ﬁ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
   Exception messages: Assert.Equal() Failure: Values differ
Expected: 1
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb] Actual:   0   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0001048
2024-12-16 05:29:48.445 Df System.Globalization.Extensions.Tests[57174:a70cb] 4) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ﬁ", normalizationForm: FormD, expected: "ﬁ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
   Exception messages: Assert.Equal() Failure: Values differ
Expected: 1
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] Actual:   0   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0001036
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] 5) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormC, expected: "ẛ̣")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
   Exception messages: Assert.Equal() Failure: Values differ
Expected: 2
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] Actual:   0   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0000991
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] 6) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormD, expected: "ẛ̣")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    Exception messages: Trying to normalize 'ẛ̣' to a buffer of length 2 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0000792
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] 7) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ﬁ", normalizationForm: FormKC, expected: "fi")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    Exception messages: Trying to normalize 'ﬁ' to a buffer of length 1 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0000785
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] 8) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ﬁ", normalizationForm: FormKD, expected: "fi")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    Exception messages: Trying to normalize 'ﬁ' to a buffer of length 1 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0000806
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] 9) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormKC, expected: "ṩ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
   Exception messages: Assert.Equal() Failure: Values differ
Expected: 1
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] Actual:   0   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.0001101
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb] 10) 	[FAIL] System.Globalization.Tests.StringNormalizationTests.Normalize   Test name: System.Globalization.Tests.StringNormalizationTests.Normalize(value: "ẛ̣", normalizationForm: FormKD, expected: "ṩ")   Test case: System.Globalization.Tests.StringNormalizationTests.Normalize
   Assembly:  [System.Globalization.Extensions.Tests, Version=10.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    Exception messages: Trying to normalize 'ẛ̣' to a buffer of length 2 succeeded!   Exception stack traces:    at System.Globalization.Tests.StringNormalizationTests.Normalize(String value, NormalizationForm normalizationForm, String expected)
   at System.Object.InvokeStub_StringNormalizationTests.Normalize(Object , Span`1 )
2024-12-16 05:29:48.446 Df System.Globalization.Extensions.Tests[57174:a70cb]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

Also, it looks like iOS legs succeeded. So, the failures are localized to maccatalyst only. Is that right? If so, any idea what the difference between it and iOS in term of is the support?

I don't see iOS pipeline running on this PR. The only run iOS job is the one that is part of regular runtime pipeline which only runs smoke tests. I'll try to kick them off. I don't think we will see difference in failing tests between let's say iOS and MacCatalyst or tvOS.

matouskozak · 2024-12-16T20:24:34Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2024-12-16T20:24:56Z

Azure Pipelines successfully started running 2 pipeline(s).

matouskozak · 2024-12-16T20:31:56Z

Btw. this is the way I'm getting the logs out (I know it's not great, we need to improve it dotnet/xharness#1188)

matouskozak · 2024-12-17T08:37:47Z

@tarekgh something is going on with the CI tests that I don't understand. Looks like everything else but maccatalyst is passing. However, when I try it locally even the other platforms are failing for me. I'm investigating.

matouskozak · 2024-12-17T10:06:03Z

@tarekgh I haven't figured out the CI issue but I think that I know what's wrong with the tests currently.

The Apple-mobile string normalize API is returning 0 when the normalization fails (which it should fail since we are trying to normalize into too small buffer)

runtime/src/native/libs/System.Globalization.Native/pal_normalization.m

Lines 56 to 66 in e78ee77

    
           Function: 
        
           NormalizeString 
        
           Used by System.StringNormalizationExtensions.Normalize to normalize a string 
        
           into a certain Unicode Normalization Form. 
        
           Return values: 
        
           0: internal error during normalization. 
        
           >0: the length of the normalized string (not counting the null terminator). 
        
           */ 
        
           int32_t GlobalizationNative_NormalizeStringNative(NormalizationForm normalizationForm, const uint16_t* lpSource, int32_t cwSourceLength, uint16_t* lpDst, int32_t cwDstLength)

But

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs

Lines 156 to 160 in e78ee77

    
           if (realLen <= destination.Length) 
        
           { 
        
               charsWritten = realLen; 
        
               return true; 
        
           }

will pass 0 as success as well.

Do you know what does
https://github.com/dotnet/runtime/blob/e78ee77456c94b83c83e485bb1a16a1d3d3a2573/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs#L147C21-L147C138 return when the destination buffer is too short? We should probably align the behavior between Apple-mobile and other ICU platforms.

tarekgh · 2024-12-17T20:24:05Z

Do you know what does
https://github.com/dotnet/runtime/blob/e78ee77456c94b83c83e485bb1a16a1d3d3a2573/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs#L147C21-L147C138 return when the destination buffer is too short? We should probably align the behavior between Apple-mobile and other ICU platforms.

When there is not enough buffer space, we return the returned length from unorm2_normalize ICU method.

runtime/src/native/libs/System.Globalization.Native/pal_normalization.c

Line 78 in e654570

    
           return (U_SUCCESS(err) || (err == U_BUFFER_OVERFLOW_ERROR)) ? normalizedLen : 0;

You can see how it is used here:

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs

Line 85 in e654570

if (realLen <= buffer.Length)

Will you continue looking at fixing this? or do you need any help from me?

matouskozak · 2024-12-18T07:25:44Z

Do you know what does
https://github.com/dotnet/runtime/blob/e78ee77456c94b83c83e485bb1a16a1d3d3a2573/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs#L147C21-L147C138 return when the destination buffer is too short? We should probably align the behavior between Apple-mobile and other ICU platforms.

When there is not enough buffer space, we return the returned length from unorm2_normalize ICU method.

runtime/src/native/libs/System.Globalization.Native/pal_normalization.c

Line 78 in e654570

return (U_SUCCESS(err) || (err == U_BUFFER_OVERFLOW_ERROR)) ? normalizedLen : 0;

You can see how it is used here:

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs

Line 85 in e654570

if (realLen <= buffer.Length)

Will you continue looking at fixing this? or do you need any help from me?

Thank you. I can take over, enjoy the vacation.

of normalized string instead of 0 for insufficiently sized buffers

matouskozak · 2024-12-19T07:55:57Z

/azp run runtime-ioslike,runtime-ioslikesimulator,runtime-maccatalyst

azure-pipelines · 2024-12-19T07:56:13Z

Azure Pipelines successfully started running 3 pipeline(s).

matouskozak · 2024-12-19T11:17:47Z

src/native/libs/System.Globalization.Native/pal_normalization.m

+        if (lpDst == NULL || cwDstLength == 0)
+        {
+            return (int32_t)[normalizedString length];
+        }


I think this part is no longer necessary, but I prefer to keep it so that it is clear what happens when destination buffer is 0.

matouskozak · 2024-12-19T11:19:31Z

@tarekgh I fixed the error handling so that it is aligned with the pal_normalization.c implementation. The tests are passing now.

tarekgh · 2024-12-19T19:37:27Z

I fixed the error handling so that it is aligned with the pal_normalization.c implementation. The tests are passing now.

@matouskozak thanks. ~~Could you please link where you did that?~~ never mind, I saw your commit now :-)

tarekgh · 2024-12-21T21:07:37Z

@akoeplinger @ivanpovazan @matouskozak could one of you please help approving this PR if there is no more feedback? Thanks!

ivanpovazan

LGTM

Fix GetNormalizedLength on iOS and similar platforms

778dca4

dotnet-issue-labeler bot added the area-System.Globalization label Dec 15, 2024

dotnet-policy-service bot assigned tarekgh Dec 15, 2024

tarekgh mentioned this pull request Dec 15, 2024

[apple] Failure in System.Globalization.Tests.StringNormalizationTests.Normalize tests #110720

Closed

tarekgh added the os-ios Apple iOS label Dec 15, 2024

tarekgh requested review from matouskozak and BrzVlad December 15, 2024 21:06

matouskozak self-assigned this Dec 18, 2024

matouskozak added 3 commits December 19, 2024 08:03

Merge branch 'main' into FixGetNormalizedLengthOniOS

41c69e3

reenable try normalize tests

9c4f658

Fix Apple ICU normalize to return length

8ca39d2

of normalized string instead of 0 for insufficiently sized buffers

matouskozak reviewed Dec 19, 2024

View reviewed changes

matouskozak requested a review from akoeplinger December 19, 2024 11:20

matouskozak requested a review from ivanpovazan December 19, 2024 18:52

ivanpovazan approved these changes Dec 23, 2024

View reviewed changes

matouskozak merged commit 74bddba into dotnet:main Dec 23, 2024
173 of 181 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix GetNormalizedLength on iOS and similar platforms #110723

Fix GetNormalizedLength on iOS and similar platforms #110723

tarekgh commented Dec 15, 2024

tarekgh commented Dec 15, 2024

azure-pipelines bot commented Dec 15, 2024

ivanpovazan commented Dec 16, 2024

tarekgh commented Dec 16, 2024

matouskozak commented Dec 16, 2024

tarekgh commented Dec 16, 2024

matouskozak commented Dec 16, 2024 •

edited

Loading

matouskozak commented Dec 16, 2024

azure-pipelines bot commented Dec 16, 2024

matouskozak commented Dec 16, 2024

matouskozak commented Dec 17, 2024

matouskozak commented Dec 17, 2024

tarekgh commented Dec 17, 2024

matouskozak commented Dec 18, 2024

matouskozak commented Dec 19, 2024

azure-pipelines bot commented Dec 19, 2024

matouskozak Dec 19, 2024

matouskozak commented Dec 19, 2024

tarekgh commented Dec 19, 2024 •

edited

Loading

tarekgh commented Dec 21, 2024

ivanpovazan left a comment

Fix GetNormalizedLength on iOS and similar platforms #110723

Fix GetNormalizedLength on iOS and similar platforms #110723

Conversation

tarekgh commented Dec 15, 2024

tarekgh commented Dec 15, 2024

azure-pipelines bot commented Dec 15, 2024

ivanpovazan commented Dec 16, 2024

tarekgh commented Dec 16, 2024

matouskozak commented Dec 16, 2024

tarekgh commented Dec 16, 2024

matouskozak commented Dec 16, 2024 • edited Loading

matouskozak commented Dec 16, 2024

azure-pipelines bot commented Dec 16, 2024

matouskozak commented Dec 16, 2024

matouskozak commented Dec 17, 2024

matouskozak commented Dec 17, 2024

tarekgh commented Dec 17, 2024

matouskozak commented Dec 18, 2024

matouskozak commented Dec 19, 2024

azure-pipelines bot commented Dec 19, 2024

matouskozak Dec 19, 2024

Choose a reason for hiding this comment

matouskozak commented Dec 19, 2024

tarekgh commented Dec 19, 2024 • edited Loading

tarekgh commented Dec 21, 2024

ivanpovazan left a comment

Choose a reason for hiding this comment

matouskozak commented Dec 16, 2024 •

edited

Loading

tarekgh commented Dec 19, 2024 •

edited

Loading