Frictionless Python Framework Roadmap: Limits of the Package

[pdelboca]

We just had an excellent conversation about the future and the roadmap of the Frictionless Python Framework. We talked about:

• the current status,
• the work needed,
• support for v2 of Datapackage,
• a common API with other framewors (like the R package)
• and more!

We agreed that an excellent next step will be to define the limits of the Python Framework both in terms of what we want it to do and what we do not want it to do anymore (as a preparation to define how the API should look like in the future) The idea is to come up with an agreed list of items that will guide future actions.

The benefits of having a clear set of limits are:

• Make it more maintainable by reducing and limiting the scope.
• Create consensus among the community on the future.
• Align expectations.
• Empower maintainers to take action and implement changes.

I'll kickoff with some points based on the discussion we had, feel free to adapt/add/remove items.

What frictionless-py should do:

• xxx
• xxx

What frictionless-py should not do:

• It should not be a swiss knife for all the task related to data analysis. (Let's expand this into more explicit items. What things in the code should no longer exist?)
• It should not implement support for data portals.

[pierrecamilleri]

Thanks a lot Patricio for continuing this conversation.

Before giving my opinion, I'd like to make it clear that I'm probably biased by our use case and have limited knowledge of other use cases, so I'm obviously (and as always!) open to any contradiction.

That said, I come from the school where a tool has to do one thing and do it well, and for frictionless, I think that what brings the most value is data validation.

So for me, and as a first take:

What frictionless-py should do:

• Validate table data, accepting numerous data sources, with the Table Schema and Data Package specifications.
  • I think providing explicit errors in case of malformed csv, or malformed json, is valuable (or any other non-binary format). Ideally, I'd prefer this to be handled by a dependency.
  • Areas for improvement in this area (apart from adapting to DP v2) include performance, documentation and ease of use (some options for Excel, for example, are fairly cryptic), extensibility (writing new checks today requires a certain familiarity with frictionless-py code), native support for i18n and this with no compromise to reliability.

What frictionless-py should not do:

• Data transformation. Many tools probably do this better.
• Data portals - as mentioned in our conversation, some of this functionality has been transferred to other tools (Zenodo was mentioned).

I have mixed feelings about this one :

• Guesswork. Guess file format, infer column formats etc. While guessing these may seem attractive as a way of hiding complexity from the user, it's the source of many bugs (to name but a few, this one, this one and this one), it introduces real complexity into the code, it complicates documentation and typing and I don't think it's worth it: the user usually knows the format of the data he wants to validate. However, I acknowledge that, especially for the command line client, is is nice to not have to enter 4 options to be able to perform a validation - but to wrap it up, I would prefer the validation to fail if there is some uncertainty rather than being unreliable for edge case.

[peterdesmet]

I agree. I (only) use frictionless-py for its full-feature validation. It's such a massive feature that I didn't replicate it in R. @mpadge and I discussed an idea once that it would be nice to have a fast Data Package validation library (in C??) that could then be used by both frictionless-py and frictionless-r (and frictionless-java, etc.).

I think frictionless-py and frictionless-r should do:

• Input/output
• Human readable print
• Manipulation: adding/removing files and data frames as resources + adding/removing metadata properties
• Validation (using a common library that is not a burden to install) with human/machine readable error report (e.g. a data frame with the errors)

Their APIs could be discussed together, but I don't think they should necessarily be the same: they should make sense in their relative context. Frictionless-r for example is currently designed to get out of the way quickly, returning your data as a data frame that you can use in so many other R libraries. It was one of the things I was missing in the previous datapackage-r package.

[khughitt]

Their APIs could be discussed together, but I don't think they should necessarily be the same.

Agreed. Trying to force the APIs to be identical might make sense for something with a very narrow scope (e.g. Arrow), but isn't likely to work for every project.

How about trying to coordinate on a "common core" API for the elements you listed above?

Frictionless-r for example is currently designed to get out of the way quickly, returning your data as a data frame that you can use in so many other R libraries

This is also mostly how I think about things / what I'm most interested in as a user.

I wonder if it would be helpful to think about two (or more) different types of users:

1. people using frictionless-py / frictionless-r to interact with data effortlessly
2. people using frictionless-py / frictionless-r as part of larger systems for data ingestion, validation, etc.?

[peterdesmet]

How about trying to coordinate on a "common core" API for the elements you listed above?

Yeah, definitely open for that.

The different types of users aren't necessarily useful to me, we should cater to both in the design of the API. It is why frictionless-r functions can be piped (|>) together:

create_package() |>
  remove_resource("wrong")
  add_resource("data-1", df) |>
  add_resource("data-2", df) |>
  write_package()

[khughitt]

Thanks for staying on top of things and starting the conversation, Patricio!

I strongly agree with the "do one thing and do it well" approach..

As we discussed in the meeting yesterday, if we decide there are other things we would like to do well, we can consider creating separate modular/dedicated libraries aimed at them (long-term ~> frictionless "universe"?)

Would it be useful to generalize this conversation, and pose the same questions for frictionless-r now? (@peterdesmet ?)

Once we agree on the high-level points / find as much common ground as is sensible, we can then separately discuss what we want to do for the current language implementations.

At the end of this discussion, let's compile a list of action items / separate discussions to have?

e.g. "What things in the code should no longer exist?"

What frictionless-py should do:

1. validation
2. i/o (https://github.com/frictionlessdata/dplib-py's job now?)

What frictionless-py should not do:

1. "transform your data"

• I agree that this should probably not be handled by frictionless-py..
• this is a massive and open-ended task
• it will depend heavily on the data type
• other dedicated libraries may already exist
• if we decide take this on, a dedicated frictionless-transforms-py library would be a better place, imo

2. CLI (update 12/19)

After thinking about it some more, I would probably move CLI from "maybe" -> "should not".

It doesn't make sense to force every language binding to implement a CLI interface.

Also, separating the API & CLI would lead to more modular and simpler code bases for each.

What frictionless-py maybe should do? 🤷

1. "Describe your data" ?

This is useful, but it's also very open-ended and might be better to split off?

2. CLI?

click and typer make it pretty straight-forward to create simple CLI's these days..

If the CLI just mirrors the frictionless-py API, as it appears to be doing now, then this is perhaps fine.

I could also imagine the CLI being a separate effort that builds on top of and interfaces with the APIs in frictionless-py, and perhaps other "core" libraries we may eventually decide to create.

I have no strong opinion on this for now; just raising awareness..

Questions

• what are the things frictionless-py currently does besides validation?
• how does https://github.com/frictionlessdata/dplib-py fit in to this?
  • what is its intended scope?
  • how does it relate to frictionless-py?
  • would it make sense to create a similar library in R, dplib-r?
• where should functionality for interfacing with other related efforts such as RO-Crates reside?

Possible next steps based on discussion so far:

1. Performance

• profile code / identify bottlenecks
• create benchmark suite (separate repo), and start to test different approaches to address bottlenecks identified

2. Scope

• create explicit lists of things frictionless-py / frictionless-r currently do besides validation?