Skip to content

Commit

Permalink
Attempt to detect when frictionless did not really validate
Browse files Browse the repository at this point in the history
  • Loading branch information
thbar committed Mar 23, 2024
1 parent a9e36b5 commit 0950fbe
Showing 1 changed file with 23 additions and 0 deletions.
23 changes: 23 additions & 0 deletions scripts/irve/dynamic-irve.exs
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,29 @@ defmodule FrictionlessValidator do
|> Enum.map(& &1["message"])
|> Enum.take(5)
end

# See https://github.com/frictionlessdata/frictionless-py/issues/1646
# Frictionless will generate a "valid" report for files it did not actually
# look at it full. As a temporary work-around, I noticed that when this
# happens, the `labels` field is an empty array, whereas when validation did
# occur for real, it contains the fields we expect. So this heuristic leverages
# that empiric observation to try to work-around that uncertainty.
def did_validation_really_occur?(output) do
%{
"tasks" => [
%{
"labels" => labels
}
]
} = output

"id_pdc_itinerance" in labels
end

def really_valid?(output) do
did_validation_really_occur?(output) &&
match?(%{"errors" => 0, "warnings" => 0}, output["stats"])
end
end

IO.puts("========== #{resources |> length()} candidates ==========\n\n")
Expand Down

0 comments on commit 0950fbe

Please sign in to comment.