Add rules for the Fares v2 base implementation

[isabelle-dr]

After the PR # 286 was merged in May 2022.

Materials

Google Doc to keep track of the Validator implementation

Additional files:

• fare_products.txt
• fare_leg_rules.txt
• fare_transfer_rules.txt
• areas.txt
• stop_areas.txt

Changes to existing files

• routes.txt: additional network_id field
• fare_rules.txt: the file becomes conditionally required

New requirement concept

Conditionally forbidden: The field or file must not be included under conditions outlined in the field or file description.

New field type

Currency amount: A decimal value indicating a currency amount. The number of decimal places is specified by ISO 4217 for the accompanying Currency code. All financial calculations should be processed as decimal, currency, or another equivalent type suitable for financial calculations depending on the programming language used to consume data. Processing currency amounts as float is discouraged due to gains or losses of money during calculations.

[bdferris-v2]

So a couple things now that I've had a chance to run #1228 against the feeds in the validator acceptance check.

• There are a significant number of feeds that don't pass validation as currently implemented. The majority of them are Trillium feeds, fwiw. The issues:
  • May fare_leg_rules.txt are missing a fare_product_id. Instead, these leg rules are specifying a cost amount directly in the rules file. fare_product_id is a required field in the spec as adopted, but I'm wondering if it didn't start that way in the initial spec revision?
  • Most currency amounts do not specify the number of decimal places to match the ISO 4217 spec. Not sure if that should be an error? Spec seems to imply it should match.
• I've seen some other validation errors that follow the general theme that feeds implementing parts of the Fares V2 spec that haven't been adopted yet generally don't validate correctly. That's often because they've added additional primary keys or other features to distinguish new types of matching in fare_leg_rules.txt or fare_transfer_rules.txt. To be honest, this was the original reason I wanted to implement validation because there seemed to be few if any feeds that had implemented the spec as adopted. I think there is a question of how we manage validation given the number of feeds that are implementing additional experimental fields.

So that brings up the questions of next steps. I wasn't involved with first round of Fare V2 adoption. Before opening it up to the entire community, I'd be interested in having a quick discussion with some key stakeholders to talk through these issues. @isabelle-dr thoughts?

[isabelle-dr]

Hello Brian,

I also had a quick look at the acceptance test report generated by your PR (the one attached here, which I hope is accurate because I see the GH action wasn't completed).
Here are some answers to your questions:

May fare_leg_rules.txt are missing a fare_product_id. Instead, these leg rules are specifying a cost amount directly in the rules file. fare_product_id is a required field in the spec as adopted, but I'm wondering if it didn't start that way in the initial spec revision?

That's exactly it, the initial implementation didn't require fare_product_id (cc @omar-kabbani).

Most currency amounts do not specify the number of decimal places to match the ISO 4217 spec. Not sure if that should be an error? Spec seems to imply it should match.

Do you mean most currency amounts do not have the number of decimal places required? We have allocated the severity ERROR for validation of every field type so yes, an invalid currency should be an error.

I think there is a question of how we manage validation given the number of feeds that are implementing additional experimental fields.

My quick answer to this is that we should first only add rules according to what was adopted, which we believe will help agencies update their data to match the latest version of it.
Adding validation for the additional experimental fields that are widely used would be a good use case for the custom validation functionality.
That being said, having datasets being flagged invalid if they have additional fields is an issue, and it's a bit of a surprise, isn't this breaking the backward compatibility?
Is the addition of the future new types of matching in fare_leg_rules.txt or fare_transfer_rules.txt making existing data invalid? (@omar-kabbani, do you have the answer?)

[bdferris-v2]

Regarding currency amounts, that's correct: they do not have the number of decimal places required.

Regarding backwards compatibility, I think this is a little subtle. Per the Guiding Principles, backwards compatibility is first and foremost about preserving backwards compatibility for feed producers. In that sense, adding new optional fields to fare_leg_rules.txt is probably ok. But it does make it pretty painful for feed consumers. Ideally, changes wouldn't break existing parsers, but I don't think that's going to be possible? Right now, to support Fares V2 as a consumer means you probably need to support the experimental spec as well.

What we do about validation is an open question. I'd like to learn more about the plan for the incremental adoption and extension of Fares V2 (if there is a plan?). @omar-kabbani maybe that's something I can chat with you about?

[omar-kabbani]

Hi @bdferris-v2, thanks for bringing this up!
I'll just add on to what @isabelle-dr mentioned:

MobilityData has published a plan for the next steps for GTFS-Fares v2 which includes adding 5 features to the specification. These features require adding optional files and fields that do not affect the way fares are modelled. We do not want to introduce any breaking changes that will break compatibility with previous GTFS datasets.

With regards to feeds not passing validation, I understand that some data producers are still using amount and currency instead of fare_product_id, so I believe that would be the main reason for validation errors.

As for the additional (experimental) fields, I did not think they would trigger errors or warnings, the validator should simply ignore them. The only case I can think of where the validator raises warnings is if the Primary Keys have changed with the addition of fields.
Would you be able to share with us a dataset that failed due to additional fields so we can investigate?

Also, we are happy to discuss things in more detail!

[isabelle-dr]

This has been completed, the validator has been updated to reflect changes to the Fares v2 base implementation.

We have decided not to include the changes done on fare_rules.txt and fare_attributes.txt in this issue, please see google/transit/issues/354 for additional context.