Make most fields in big query diagnostic schema as optional

[pritamghanghas]

While working on product-management#72, it was observed that we might miss some diagnostic reports due to schema validation of required fields.

We don't want to miss diagnostic reports if some of the current mandatory fields are malformed or missing due to hardware/software issues. Except for timestamp, there isn't any need to mark fields as required.
In case of errors, this might lead to diag not being uploaded and thus problem being masked. Even if there is no large problem being masked, the diag upload frequency which we use to calculate uptime becomes unreliable.

Required Fields as of date that need conversion to optional:
"BN",
"ID",
"BA",
"FR",
"FW",
"VA",
"serial_number",
"PK",
"OK"
"AN",

Acceptance Criterion

• Make all fields except timestamp optional in nebra-production.hotspot_diagnostics_data.diagnostics

[pritamghanghas]

@marvinmarnold This will require making changes to the table. @htsili can probably do it. I have checked. Converting previously required fields to optional is allowed by big query. I can make the device side changes after that.

[htsili]

@pritamghanghas i think this is now done. Can you please check it?

[pritamghanghas]

@htsili LGTM I will work on rest of it.

[marvinmarnold]

We are seeing diagnostics coming through again so I think this is production already. Is any additional work needed or can this be closed @pritamghanghas ?

[pritamghanghas]

@marvinmarnold No, this is in QA now. I will check a few devices from testnet tomorrow and close the ticket.

[KevinWassermann94]

@pritamghanghas This must be in production already. We are getting around 470k upload per day since Jul 28th

[marvinmarnold]

@pritamghanghas can you make sure this gets into production before the end of the sprint.

@KevinWassermann94 this has changes in two places - Google Cloud schema and code. Changing the cloud schema is in prod and restored uploads for most devices. The code change still needs to be deployed but will prob not have a big impact.