Skip to content

Commit

Permalink
add DigestHash to list of possible file attributes
Browse files Browse the repository at this point in the history
Summary:
# Context:

We are currently overloading the pre-existing attributes `FILE_SIZE` and `BLAKE3` to mean different things depending on whether directory or file attributes are being fetched from the readdir/getAttributesFromFilesV2 endpoint.

In the file case, file_size means the size of the file on disk. In the directory case, it means the size of the directory digest that's stored in RE's Content Addressed Store (CAS).

In the file case, blake3 means the blake3 hash of that single file. In the directory case, blake3 means the blake3 hash of the directory digest that's stored in RE CAS.

It's no longer fine to overload the blake3 hash case since existing users of the blake3 attribute do NOT expect the number of results to increase. Therefore, we will have to introduce a separate attribute for requesting hashes of directories that won't interfere with the current usage of blake3 attribute requests.

# Solution

We should introduce a new attribute for query RE CAS digest size information for directories. This will allow us to avoid overloading the existing size/blake3 attributes and instead allow users to specifically query for CAS digest size/hash if they want it. This attribute will be queriable for both directories and files. However, for files it will behave the exact same way as requesting the existing file_size/blake3 attribute.

# This diff

This diff introduces the digest hash attribute. The actual implementation of this attribute is done in a later diff (to reduce the number of changes per diff).

Reviewed By: jdelliot

Differential Revision: D61688934

fbshipit-source-id: ea71e0b720f70739da19bd04eacca33e7b511901
  • Loading branch information
MichaelCuevas authored and facebook-github-bot committed Sep 9, 2024
1 parent 1c7a5d7 commit 9ad00db
Showing 1 changed file with 24 additions and 4 deletions.
28 changes: 24 additions & 4 deletions eden/fs/service/eden.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -331,21 +331,30 @@ enum FileAttributes {
OBJECT_ID = 8,

/**
* Returns the BLAKE3 hash of a file or directory. Returns an error for
* symlinks and non-regular files. Note: for directories, the blake3 hash is
* the digest hash of the directory's augmented manifest.
* Returns the BLAKE3 hash of a file. Returns an error for
* symlinks, directories, and non-regular files. Note: the digest_hash can be
* requested for directories as an alternative to blake3_hash.
*/
BLAKE3_HASH = 16,

/**
* Returns the digest size of a given file or directory. This can be used
* together with BLAKE3_HASH to determine the key that should be used to
* together with DIGEST_HASH to determine the key that should be used to
* fetch a given file/directory from Content Addressed Stores (i.e. RE CAS).
* For directories, the size of the augmented manifest that represents the
* the directory is returned. For files, this field is the same as FILE_SIZE.
* Returns an error for any non-directory/non-file types (symlink, exe, etc).
*/
DIGEST_SIZE = 32,

/**
* Returns the digest hash of a given file or directory. This can be used
* together with DIGEST_SIZE to determine the key that should be used to
* fetch a given file/directory from Content Addressed Stores (i.e. RE CAS).
* For files, this hash is just the blake3 hash of the given file. For
* directories, this hash is blake3 hash of all the directory's descendents.
*/
DIGEST_HASH = 64,
/* NEXT_ATTR = 2^x */
} (cpp2.enum_type = 'uint64_t')

Expand Down Expand Up @@ -425,6 +434,16 @@ union DigestSizeOrError {
2: EdenError error;
}

union DigestHashOrError {
// Similar to ObjectIdOrError, it's possible for `digest hash` to be unset
// even if there is no error.
//
// Notably, no digest hash will be returned if any child file or directory
// has been modified.
1: BinaryHash digestHash;
2: EdenError error;
}

/**
* Subset of attributes for a single file returned by getAttributesFromFiles()
*
Expand All @@ -439,6 +458,7 @@ struct FileAttributeDataV2 {
4: optional ObjectIdOrError objectId;
5: optional Blake3OrError blake3;
6: optional DigestSizeOrError digestSize;
7: optional DigestHashOrError digestHash;
}

/**
Expand Down

0 comments on commit 9ad00db

Please sign in to comment.