Enhance API data with timezone at location's position

[jpmens]

We might consider enhancing API data with timezone at a location's position.

Inspired by #443

[jpmens]

Two new fields:

• tz is the timezone at the location (lat, lon)
• isolocal is the local time at the location

  {
   "_type": "location",
   "acc": 11,
   "alt": 1562,
   "batt": 55,
   "bs": 1,
   "cog": 264,
   "conn": "m",
   "lat": 17.069333,
   "lon": -96.717628,
   "m": 1,
   "p": 84.441,
   "t": "p",
   "tid": "jp",
   "tst": 1703116330,
   "vac": 4,
   "vel": 0,
   "ghash": "9g51p5k",
   "cc": "MX",
   "addr": "C Narciso Mendoza, 68080 Oaxaca, OAX, Mexico",
   "locality": "Oaxaca",
   "isorcv": "2023-12-20T23:52:10Z",
   "isotst": "2023-12-20T23:52:10Z",
   "disptst": "2023-12-20 23:52:10",
   "tz": "America/Mexico_City",
   "isolocal": "2023-12-20T17:52:10-0600"
  }

[centralhardware]

@jpmens I am running latest version, but don't see new fields.

[jpmens]

Do you see TZDATADB diagnostic when launching recorder,? Does the file exist? Is it readable?

[centralhardware]

it was problem with file, all works as expected. Are you planning to allow access this fields inside lua hook?

[jpmens]

it was problem with file, all works as expected.

Let me know if the installer (make) failed, please.

Are you planning to allow access this fields inside lua hook?

I’ve not tested, but they should be there automatically. Are they not?

[centralhardware]

Let me know if the installer (make) failed, please.

I got this error:
0.265 install: can't create '/usr/share/owntracks/recorder/timezone16.bin': No such file or directory

error don't appear
if add dir creation to Docker file
RUN mkdir -p /usr/share/owntracks/recorder
but produced image doesn't contain timezone16.bin. I download file manualy and recorder start working properly.

I’ve not tested, but they should be there automatically. Are they not?

function otr_init()
	otr.log("example.lua starting; writing to /tmp/lua.out")
end

function otr_hook(topic, _type, data)
        otr.log(data['isolocal'])
        otr.log(data['tz'])
end

function otr_exit()
end

print nulls

[ckrey]

I see "tz" and "isolocal" in ocat output:

 {
      "_type": "location",
      ...
      "isorcv": "2024-01-30T22:11:18Z",
      "isotst": "2024-01-30T22:11:18Z",
      "disptst": "2024-01-30 22:11:18",
      "tz": "Europe/Berlin",
      "isolocal": "2024-01-30T23:11:18+0100"
    }

but not in api:

    {
      "_type": "location",
      ...
     "isorcv": "2024-01-30T22:11:18Z",
      "isotst": "2024-01-30T22:11:18Z",
      "disptst": "2024-01-30 22:11:18"
    }

[jpmens]

@ckrey one possible cause is that your Recorder process can't access the TZDATADB file (permissions, path)? Another that the Recorder hasn't been restarted?

The data produced by ocat and ot-recorder should be identical otherwise.

[ckrey]

@ckrey one possible cause is that your Recorder process can't access the TZDATADB file (permissions, path)? Another that the Recorder hasn't been restarted?

The data produced by ocat and ot-recorder should be identical otherwise.

you are absolutely right, I had a mélange of package installed and locally built versions

[jpmens]

So, we're changing the strategy a bit, also to accomodate @centralhardware's wish to have the timezone in Lua, which I'll admit to have not previously considered.

The code I'll be pushing in a moment now does the following:

1. When a new publish arrives, we perform reverse geo lookup on the lat/lon if configured. When using OpenCage (and only when using OpenCage), we obtain the timezone name from the annotations in their data. This we store alongside the actuall address data in our lmdb database (note tzname):

   $ ocat --dump
   d5dj6kr {"cc":"MX","addr":"CAPA, Calle 18 Norte, 77720 Playa del Carmen, ROO, Mexico","tst":1706694151,"locality":"Playa del Carmen","tzname":"America/Cancun"}

2. This tzname element is made available to Lua hooks, as the following example demonstrates:

   $ ot-recorder --lua-script tzdata.lua ...
   Jan 31 09:42:25  ot-recorder[45698] <Info>: TZDATADB is at sundry/zonedetect/ZoneDetect/out/timezone16.bin
   Jan 31 09:42:25  ot-recorder[45698] <Debug>: Subscribing to owntracks/# (qos=2)
   | tzname	America/Cancun
   | tid	JJ
   | _type	location
   | device	moto
   | ghash	d5dj6kr
   | username	jjolie
   ...
   | addr	CAPA, Calle 18 Norte, 77720 Playa del Carmen, ROO, Mexico
   | cc	MX
   - 09:42:30 owntracks/jjolie/moto               t=u tid=JJ loc=20.63451,-87.07807 [MX] CAPA, Calle 18 Norte, 77720 Playa del Carmen, ROO, Mexico (d5dj6kr)

3. When serving data (via ocat or the API), the data is enriched with tzname and isolocal, the latter being the local time at the source of the location publish.

   $ ocat -S JP -u jjolie -d moto
   {"count":1,"locations":[{"_type":"location","cog":67,"batt":11,"lat":20.634506,"t":"u","lon":-87.078073,"acc":5,"tid":"JJ","vel":12,"vac":-1,"alt":0,"tst":1706694150,"ghash":"d5dj6kr","cc":"MX","addr":"CAPA, Calle 18 Norte, 77720 Playa del Carmen, ROO, Mexico","locality":"Playa del Carmen","tzname":"America/Cancun","isorcv":"2024-01-31T09:42:30Z","isotst":"2024-01-31T09:42:30Z","disptst":"2024-01-31 09:42:30"}]}

4. For historical records which don't have a cached tzname, we use the TZDATADB as described above to obtain tzname, incurring a not quite insignificant penalty in terms of runtime.

[centralhardware]

Will be tz field available even without reverse geo coding (via tzdatadb)?

[jpmens]

The field is now named tzname (no longer tz) and is available in Lua hooks only from reverse geo. The reason is that Lua hooks are invoked when a publish arrives at the recorder.

To clarify, in the case of no reverse geo, tzname will still be generated from the tzdatadb when serving data.

[jpmens]

Also to clarify, OpenCageData have a free tier you can sign up for. They offer one API key and up to 2500 reverse geo queries per day which is likely enough for many of us. :-)

[centralhardware]

Also to clarify, OpenCageData have a free tier you can sign up for. They offer one API key and up to 2500 reverse geo queries per day which is likely enough for many of us. :-)

it's good news, but I can't get it to work. I am trying OTR_GEOKEY and rebuild image with --geokey opencage:, but it 's still don't work

[jpmens]

I'll need a bit more info than that ...

First, you should not need to rebuild. I'm assuming your config.mk has something like GHASHPREC = 7 in it which is good.

Then, please remember, that if you post or publish a location you've already been at (!) the query will not be performed, as the Recorder has already "not cached" (yes, it sounds strange) the location.

Move your STORAGE/ghash/*.mdb files aside, launch recorder once with the --initialize flag to re-create the empty databases, and then launch normally.

Now publish a position. Do you see the reverse geo on the console?

What does ocat --dump show?

[centralhardware]

Move your STORAGE/ghash/*.mdb files aside, launch recorder once with the --initialize flag to re-create the empty databases, and then launch normally.

It's work now, thank you

[jpmens]

I believe we might consider this complete now (famous last words). We spent quite some time (thank you, @ckrey) optimizing output and have been able to lower processing time on locations/ API by an order of magnitude by avoiding constant timezone switching.