README.md

Input/Output Data Transport

An input/output data transport utility objects 🚅 was created for full managed object of configuration file that include load from any file format like .yaml, .json, or .toml, and manage retention and version of config files lifecycle.

This package has 3 parts of core module; Register, Stores, and Files. The core part of this project is Files module that is an open file object with specific format that want to open include open directory objects (I think the open dir object do not should to use now because I do not do with its code.)

📍 Installation

pip install -U ddeutil-io

Python version supported:

Python Version	Installation	Support Fixed Bug
>=3.9,<3.14	pip install -U ddeutil-io	✔️

Note

This package need to install ddeutil first to be core package namespace. You do not need to pip it because I include this package to the required list.

For optional dependencies that should to additional installation;

Module	Installation	Additional dependencies
YamlFl	ddeutil-io[yaml]	pip install PyYaml
TomlFl	ddeutil-io[toml]	pip install toml
MsgpackFl	ddeutil-io[msgpack]	pip install msgpack

🎯 Features

The features of this package is Input/Output data transport utility objects.

Module	Name	Description	Remark
files	PathSearch	Path Search object that use to search path tree from an input root path.
	rm	Remove a file or dir from an input path.
	touch	Create an empty file with specific name and modified time of path it an input times was set.
files.file	Fl	Open File object that use to open any normal or compression file from current local file system
	EnvFlMixin	Environment Mapping to read method of open file object mixin.
	EnvFl	Dot env open file object which mapping search engine to data context that reading from dot env file format (.env).
	YamlEnvFl	Yaml open file object which mapping search environment variable.
	YamlFl	Yaml open file object that read data context from Yaml file format (.yml, or .yaml).
	YamlFlResolve	Yaml open file object with resolve boolean convert value problem such as convert 'on' value to true instead a string of 'on' value.
	JsonEnvFl	Json open file object which mapping search environment variable before parsing with json package.
	JsonLineFl	Json open file object that read data context from Json file format (.json) with a newline seperator.
	JsonFl	Json open file object that read data context from Json file format (.json).
	CsvFl	CSV open file object with comma (,) seperator charactor.
	CsvPipeFl	CSV open file object with pipe (|) seperator charactor.
	TomlEnvFl	TOML open file object which mapping search environment variable before parsing with toml package from TOML file format (.toml).
	TomlFl	TOML open file object that read data context from TOML file format (.toml).
	PickleFl	Pickle open file object that read data context from Pickle file format (.pickle).	no cover
	MarshalFl	Marshal open file object that read data context from Marshal file format.	no cover
	MsgpackFl	Msgpack open file object that read data context from Msgpack file format.	no cover
stores	Store	Store File Loading Object for get data from configuration and stage.
	StoreJsonToCsv	Store object that getting the Json context data and save it to stage with CSV file format.
	StoreToJsonLine	Store object that getting the YAML context data and save it to stage with Json line file format.
register	Register	Register Object that contain configuration loading methods and metadata management.
	ArchiveRegister	Archiving Register object that implement archiving management on the Register object such as self.purge, and self.remove methods.

🍻 Usages

I will show some usage example of function in this package. If you want to use complex or adjust some parameter, please see doc-string or real source code (I think it does not complex, and you can see how that function work).

Files

For example, I will represent YamlEnvFl object that passing environment variable to reading content before passing to the Yaml loader.

data:
  get: HELLO ${HELLO}

import os
from ddeutil.io import YamlEnvFl

os.environ["HELLO"] = "WORLD"
content = YamlEnvFl('./source.yaml').read(safe=True)
assert content['data']['get'] == "HELLO WORLD"

Note

This module do not implement special function on IO like the build-in open function. It also makes standard read and write file objects.

Store

Store object is the storing dir system handler object that manage any files in that dir path with get, move, load, save, or ls operations.

from ddeutil.io.stores import Store

store: Store = Store(path='./conf', compress="gzip")

data = store.get(name='config_file.yaml')
store.save('./stage/file.json', data)

conf/
 ├─ examples/
 │   ╰─ config_file.yaml
 ╰─ stage/
     ╰─ file.json

Register

The Register Object is the metadata generator object for the config data. If you passing name and configs to this object, it will find the config name in any stage storage and generate its metadata to you.

from ddeutil.io.register import Register
from ddeutil.io.config import Params

registry: Register = Register(
    name='examples:conn_data_local_file',
    params=Params(**{
        "stages": {
            "raw": {"format": "{naming:%s}.{timestamp:%Y%m%d_%H%M%S}"},
        },
    }),
)
registry.move(stage="raw")

The raw data of this config was written in conn_file.yaml file.

conf/
 ╰─ examples/
     ╰─ conn_file.yaml

When call move method, it will transfer data from .yaml file to json file with the data hashing algorithm.

data/
 ├─ __METADATA/
 │   ├─ exampleconn_data_local_file.base.json
 │   ╰─ exampleconn_data_local_file.raw.json
 ╰─ raw/
     ╰─ conn_file_20240101_000000.json

💬 Contribute

I do not think this project will go around the world because it has specific propose, and you can create by your coding without this project dependency for long term solution. So, on this time, you can open the GitHub issue on this project 🙌 for fix bug or request new feature if you want it.