Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Omit GeoJSON properties with null values #206

Merged

Conversation

kylebarron
Copy link
Member

In geoarrow/geoarrow-rs#588 I got a bug report about a GeoJSON file that failed to read. The issue here is a column that's numeric for the first N rows but then is null for some row N + 1. Currently the GeoJSON reader coerces a null value to the string "null":

// Null, Array(Vec<Value>), Object(Map<String, Value>)
_ => processor.property(i, key, &ColumnValue::String(&value.to_string()))?,

This is hard to work with because in my processor I'd have to check every input string value (from any geozero reader, for any column) against the string "null" and manage type conversions. E.g. if the first few rows of a column are the string "null" but then I see an f64 value, I need to be able to coerce that column to a Float64Array (which in Arrow has its own null bitmask).

I'd argue that the simplest approach in this specific case is to do a no-op for GeoJSON null values. That {"a": 1, "b": null} is equivalent to {"a": 1}. That seems like a much smaller change than, say, adding a ColumnValue::Null variant. But interested to hear others' thoughts.

Copy link
Member

@michaelkirk michaelkirk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a mostly uncontroversial improvement, and so we should merge it.

Beyond that, I have thoughts, but no conclusions, about introducing an explicit Null type into geozero...

Even though I think your change is mostly uncontroversially an improvement, I'm going to let it sit for a couple days before merging in case anyone else wants to weigh in.

@michaelkirk michaelkirk added this pull request to the merge queue Apr 16, 2024
Merged via the queue into georust:main with commit cdda9a8 Apr 16, 2024
1 check passed
@kylebarron kylebarron deleted the kyle/omit-geojson-null-properties branch April 16, 2024 20:56
kylebarron added a commit to geoarrow/geoarrow-rs that referenced this pull request May 8, 2024
### Change list

- Support writing Struct, Array, Datetime columns from GeoArrow through
geozero.
- Vendor geozero GeoJSON updates
(georust/geozero#208 and
georust/geozero#206)
- Handle null attribute values when writing to geozero.
- add `writeGeoJSON` to JS binding
- 

Closes #588. 

~~I'm still getting issues with the GeoJSONWriter excluding fields. In
particular, writing Overture data to GeoJSON completely omits list and
struct fields, giving invalid GeoJSON with ", ,"~~

---- 

Note: tested this via the Python API
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants