Support cross-platform compatibility by making xattr conditionally dependent

[marczalik]

What is the use case for the feature?
OFRAK, and in particular FilesystemEntry, currently supports working with xattr attributes, which exist for Linux and macOS files. Rather than xattr, Windows uses Alternative Data Streams. As such, OFRAK has a platform-specific dependency for xattr that prevents Windows users from using OFRAK. Since we currently only manipulate xattr attributes for completeness (copying them to files OFRAK has modified), we can remove this dependency for now in order to enable OFRAK on Windows.

Rather than removing support for xattr, we will make it a dependency conditional upon the platform running OFRAK. On platforms which do not support xattr (e.g., Windows), we will use a stub xattr library that will warn users that extended attributes are not supported when using OFRAK on Windows.

Does the feature contain any proprietary information about another company's intellectual property?
No.

How would you implement this feature?
Remove references to xattr from the FilesystemEntry class.

Make xattr dependency conditional on platform_system. Any file importing xattr will fall back to the stub library if xattr is not present.

Are there any (reasonable) alternative approaches?
It makes sense to remove xattr for now in order to get Windows users up and running on OFRAK. We should have design discussions around whether/how we will support xattr/ADS in the future.

Are you interested in implementing it yourself?
Yes.

[rbs-jacob]

Here are some of the places I could find that will need to be modified if references to xattr are to be removed.

examples/ex4_filesystem_modification.py
51:    # Modify the program permission bits and xattrs before repacking
53:    print(f"Initial xattrs: {hello_world_program.xattrs}")
56:    await hello_world_program.modify_xattr_attribute("user.foo", b"bar")
59:    print(f"Modified xattrs: {hello_world_program.xattrs}")

examples/test_examples.py
5:import xattr
79:     * xattrs
89:    assert xattr.getxattr(target_file, "user.foo") == b"bar"

ofrak_tutorial/notebooks_with_outputs/5_filesystem_modification.ipynb
178:    "print(f\"current extended attributes: {hello_world_program.xattrs}\")"
216:    "    # Modify the program permission bits and xattrs before repacking:\n",
220:    "    await hello_world_program.modify_xattr_attribute(\"user.foo\", \"bar\".encode(\"utf-8\"))\n",
311:    "xattr -l modified_squashfs_dir/src/hello_world"

ofrak_core/mypy.ini
16:[mypy-xattr.*]

ofrak_core/test_ofrak/components/test_tar_component.py
117:            command = ["tar", "--xattrs", "-C", directory, "-cf", archive.name, "."]
134:            command = ["tar", "--xattrs", "-C", extract_dir, "-xf", tar.name]
214:        command = ["tar", "--xattrs", "-C", root, "-xf", self.testtar_path]
229:            command = ["tar", "--xattrs", "-C", extract_dir, "-xf", tar.name]

ofrak_core/test_ofrak/components/test_filesystem_component.py
182:    async def test_modify_xattr_attribute(self, filesystem_root: FilesystemRoot):
184:        Test that FilesystemEntry.modify_xattr_attribute modifies the entry's xattr attributes.
187:        assert child_textfile.xattrs == {}
188:        await child_textfile.modify_xattr_attribute("user.foo", b"bar")
189:        assert child_textfile.xattrs == {"user.foo": b"bar"}
320:            command = ["tar", "--xattrs", "-C", directory, "-cf", archive.name, "."]
337:            command = ["tar", "--xattrs", "-C", extract_dir, "-xf", tar.name]
345:            command = ["tar", "--xattrs", "-C", directory, "-cf", archive_name, "."]

ofrak_core/pytest_ofrak/patterns/pack_unpack_filesystem.py
7:import xattr
23:        self.check_xattrs in a subclass.
25:        self.check_xattrs = True
60:        permissions and xattrs. Return the path to the root directory. Override this method to
61:        test with a different directory structure, or to add xattrs or stat values.
98:        if self.check_xattrs:
99:            self._validate_xattrs_equality(old_path, new_path)
117:    def _validate_xattrs_equality(self, old_path: str, new_path: str):
118:        old_xattrs = dict()
119:        for attr in xattr.listxattr(old_path, symlink=True):
120:            old_xattrs[attr] = xattr.getxattr(old_path, attr, symlink=True)
121:        new_xattrs = dict()
122:        for attr in xattr.listxattr(new_path, symlink=True):
123:            new_xattrs[attr] = xattr.getxattr(new_path, attr, symlink=True)
125:            old_xattrs == new_xattrs
126:        ), f"{old_path} and {new_path} have different xattrs\nold: {old_xattrs}\nnew: {new_xattrs}"

ofrak_core/ofrak/core/tar.py
56:                command = ["tar", "--xattrs", "-C", temp_dir, "-xf", temp_archive.name]
79:            command = ["tar", "--xattrs", "-C", flush_dir, "-cf", temp_archive.name, "."]

ofrak_core/ofrak/core/filesystem.py
7:import xattr
30:    xattrs: Optional[Dict[str, bytes]]
90:    async def modify_xattr_attribute(self, attribute: str, value: bytes):
92:        Modify the extended file attributes ("xattrs") for the filesystem entry.
97:        if self.xattrs is None:
98:            self.xattrs = dict()
99:        self.xattrs[attribute] = value
137:        if self.xattrs:
138:            for attr, value in self.xattrs.items():
139:                xattr.setxattr(path, attr, value)
303:                folder_attributes_xattr = self._get_xattr_map(absolute_path)
310:                            folder_attributes_xattr,
318:                        folder_attributes_xattr,
340:                file_attributes_xattr = self._get_xattr_map(absolute_path)
347:                            file_attributes_xattr,
357:                            file_attributes_xattr,
362:                        FIFOPipe(relative_path, file_attributes_stat, file_attributes_xattr),
367:                        BlockDevice(relative_path, file_attributes_stat, file_attributes_xattr),
372:                        CharacterDevice(relative_path, file_attributes_stat, file_attributes_xattr),
425:                if entry.xattrs:
426:                    for attr, value in entry.xattrs.items():
427:                        xattr.setxattr(link_name, attr, value, symlink=True)  # Don't follow links
508:        folder_xattrs: Optional[Dict[str, bytes]] = None,
518:        :param folder_xattrs: xattrs for the folder
538:                    Folder(directory, folder_stat_result, folder_xattrs),
563:        file_xattrs: Optional[Dict[str, bytes]] = None,
574:        :param file_xattrs: xattrs for the file
593:            File(filename, file_stat_result, file_xattrs),
654:    def _get_xattr_map(cls, path):
655:        xattr_dict = {}
656:        for attr in xattr.listxattr(path, symlink=True):  # Don't follow links
657:            xattr_dict[attr] = xattr.getxattr(path, attr)
658:        return xattr_dict

ofrak_core/ofrak/ofrak_context.py
82:                FilesystemRoot._get_xattr_map(full_file_path),

ofrak_core/requirements.txt
17:xattr==0.10.1

ofrak_patch_maker/mypy.ini
22:[mypy-xattr.*]

[marczalik]

Need to have an offline discussion about how to preserve existing xattr functionality while also supporting OFRAK on Windows.

[andresito00]

ext2, ext3, ext4, JFS, Squashfs, UBIFS, Yaffs2, ReiserFS, Reiser4, XFS, Btrfs, OrangeFS, Lustre, OCFS2 1.6, ZFS, and F2FS filesystems support extended attributes (abbreviated xattr) when enabled in the kernel configuration... Currently, four namespaces exist: user, trusted, security and system. The user namespace has no restrictions with regard to naming or contents. The system namespace is primarily used by the kernel for access control lists. The security namespace is used by SELinux, for example.

Red Balloon Security, Inc. has previously encountered extended attributes in an embedded device filesystem. With widening adoption of SELinux kernel features, we should expect to see its usage more in the future.

Improper (read: lack of) handling extended attributes lead to consistent crashes on boot in our previous efforts. In that case, the kernel was not verbose in the root cause of its failure, leading to extended debug times to root cause the issue.

To save OFRAK users time and heartache, I recommend not removing this functionality, but, however possible, maintaining its support and warning users on platforms that do not support xattr handling.

[marczalik]

Resolved by #228