getContentMd5() of blob returns encrypted MD5 value due to SERVER SIDE ENCRYPTION TRUE by default on azure storage side?

[harisingh-highq]

Hi
I found the case when we get MD5 of blob file using blobClient.getProperties().getContentMd5(), will it return encrypted MD5 of blob due to SERVER SIDE ENCRYPTION TRUE by default on azure storage side?

Below is code snippet:-
`blobClient.upload(bin, length);

BlobProperties prop = blobClient.getProperties();
java.security.MessageDigest md5 = java.security.MessageDigest.getInstance("MD5");
md5.update(prop.getContentMd5());
byte[] encrBytes = md5.digest();
StringBuilder encrHash = new StringBuilder();
for (int i = 0; i < encrBytes.length; i++) {
encrHash.append(Integer.toString((encrBytes[i] & 0xff) + 0x100, 16).substring(1));
}`

Above StringBuilder encrHash value does not match with if I download same file from azure UI and calculate MD5 of downloaded file in local system.

Is it valid behavior?

OR below logic of converting byte array of getContentMd5() to string builder is wrong from my side?

Can you please help me to sort out this issue?

[gapra-msft]

Hi @harisingh-highq

Thanks for posting this question.

I've linked a sample from our tests to clarify how to validate the contentMd5 returned from the server.
"defaultText" is just a placeholder for the data. So you will need to replace that with the actual data you are calculating the md5 for.

Hope that helps!

[harisingh-highq]

I upload same file in azure storage and cloudian S3 storage. It give me different MD5 value of same file in different storage.
Is it valid behaviour?

In azure blob storage, I'm getting MD5 value using azure-storage-blob sdk as per below code:-
java.security.MessageDigest md5 = java.security.MessageDigest.getInstance("MD5");
md5.update(prop.getContentMd5());
byte[] encrBytes = md5.digest();
StringBuilder md5Hash = new StringBuilder();
for (int i = 0; i < encrBytes.length; i++) {
md5Hash .append(Integer.toString((encrBytes[i] & 0xff) + 0x100, 16).substring(1));
}
In cloudian S3 storage, I'm getting MD5 value using aws-java-sdk-s3 sdk as per below code:-
String md5Hash = amazonS3Client.getObjectMetadata(bucketName, fileName).getETag();

Can anyone help me into this?

[harisingh-highq]

@gapra-msft
Thank you for sharing sample code.

My concern is Azure stored file has different MD5 checksum than local file??

If I compare Azure uploaded file contentMD5 with local same file MD5, it is not matched. Is it valid behavior?

[gapra-msft]

Hi @harisingh-highq

How are you calculating the md5 of the local file?

In the code you shared above, it seems like you are calculating the md5 of the md5, rather than the data itself, which could be why you are seeing discrepancies

[harisingh-highq]

Hi @gapra-msft
Thank you for quick response.

FYI, I calculate local file MD5 with online utility OR using command prompt

I tried with below code, no one is matched with local file MD5:-

1st Try:-
java.security.MessageDigest md5 = java.security.MessageDigest.getInstance("MD5");
md5.update(blobClient.getProperties().getContentMd5());
StringBuilder uploadedFileMD5= new StringBuilder();
if (md5 != null) {
for (byte b : md5.digest()) {
uploadedFileMD5.append(String.format("%02x", b));
}
}

2nd Try:-
BlobProperties prop = blobClient.getProperties();
String uploadedFileMD5= Base64.encodeBase64String(prop.getContentMd5());

3rd Try:-
BlobProperties prop = blobClient.getProperties();
String uploadedFileMD5= Base64.encodeBase64String(MessageDigest.getInstance("MD5").digest(prop.getContentMd5()));

[gapra-msft]

@harisingh-highq

Thank you for the extra information. I wrote a small sample script to verify that the service is indeed computing the md5 correctly. It looks like the cert util tool returns the md5 as a hex encoded string of the md5 bytes. The SDK simply returns the raw bytes. So to make them match, you need to convert one of them.

I've attached a sample below. Hope this helps!

    private static final String account = "XX";
    private static final String url = "http://XX.blob.core.windows.net";
    private static final String key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
    private static final String container = "XXXXXXXXXXX";
    private static final String blob = "XXXXXXXXXXXXXXXXXXXXXX";

    private static final String certUtilMD5 = "XXXXXXXXXXXXXXXXXXX"; // This is the hex representation of the md5 as a string
    private static final char[] HEX_ARRAY = "0123456789abcdef".toCharArray();

    public static void main(String[] args) throws IOException, InterruptedException {

        final StorageSharedKeyCredential credential = new StorageSharedKeyCredential(account, key);

        BlobServiceClient serviceClient = new BlobServiceClientBuilder()
                .credential(credential)
                .endpoint(url)
                .buildClient();

        BlobContainerClient containerClient = serviceClient.getBlobContainerClient(container);

        BlobClient blobClient = containerClient.getBlobClient(blob);

        byte[] contentMd5 = blobClient.getProperties().getContentMd5();

        char[] hexChars = new char[contentMd5.length * 2];
        for (int j = 0; j < contentMd5.length; j++) {
            int v = contentMd5[j] & 0xFF;
            hexChars[j * 2] = HEX_ARRAY[v >>> 4];
            hexChars[j * 2 + 1] = HEX_ARRAY[v & 0x0F];
        }
        System.out.println(new String(hexChars));
        System.out.println(certUtilMD5);
    }

[gapra-msft]

This should also resolve your other issue #17758

[harisingh-highq]

Hi @gapra-msft

I would like to inform you that now local file MD5 and azure store file MD is getting matched with above solution.

Thank you so much for detailed solution. Really your help is appreciable.

[gapra-msft]

@harisingh-highq Great to hear! Since your issue seems to have been resolved, I will go ahead and close this issue.