Collect Statistics from Matomo #1113

sarahsporck · 2023-09-04T08:17:22Z

Credentials for matomo can be found in passbolt. For now I tracked:

search
card activation
card verification
card creation

in each request I tried to include:

the user ip address
user agent (for device information)
language (especially interesting after we implement i18next)
region (if possible)

Furthermore I decided to implement tracking completely in the backend, as this is where all information is present. This also means we don't have to implement tracking in frontend AND administration.

What's still missing?

option to disable tracking in the frontend and probably also in administration
a matomo health endpoint as errors are currently only logged, so it could go unnoticed if matomo is down or behaves strangely (@svenseeberg or does this already exist?)
production credentials

administration/src/cards/createCards.ts

sarahsporck · 2023-09-04T08:20:19Z

backend/src/main/kotlin/app/ehrenamtskarte/backend/matomo/Matomo.kt

+enum class EventCategory {
+    Cards
+}


We could also change the category. I guess adding Cards as category is kind of meaningless

sarahsporck · 2023-09-04T08:20:46Z

backend/src/main/kotlin/app/ehrenamtskarte/backend/matomo/Matomo.kt

+
+object Matomo {
+    val logger = LoggerFactory.getLogger(Matomo::class.java)
+    private const val ACTION_NAME = "Request"


Without an action name the request is not displayed properly in matomo.

Never mind. I tested this again. Should work fine without the action name

Do you plan to remove this then?

backend/src/main/kotlin/app/ehrenamtskarte/backend/matomo/Matomo.kt

frontend/lib/graphql/configured_graphql_provider.dart

michael-markl

Just some drive-by comments;

I didn't pay attention to what's collected and if we can deduce all desired statistics from that.

backend/src/main/kotlin/app/ehrenamtskarte/backend/config/BackendConfiguration.kt

administration/src/cards/createCards.ts

backend/src/main/kotlin/app/ehrenamtskarte/backend/matomo/Matomo.kt

f1sh1918

Hm i can not find matomo credential in passbolt.
Maybe i don't have sufficient rights?
Please provide me access to test :)

michael-markl · 2023-09-05T13:49:41Z

On a side note: It might be beneficial to also add a matomo docker container to the dev setup, so we can more easily test the tracking locally. I am not sure how complicated that would be, though.

sarahsporck · 2023-09-06T10:07:35Z

On a side note: It might be beneficial to also add a matomo docker container to the dev setup, so we can more easily test the tracking locally. I am not sure how complicated that would be, though.

Added fancy docker setup for this :) Besides there is a dev site in our live matomo where we can test against the live system.

Hm i can not find matomo credential in passbolt. Maybe i don't have sufficient rights? Please provide me access to test :)

Don't have the rights to do this @svenseeberg

f1sh1918 · 2023-09-06T13:18:46Z

docs/development-setup.md

+14. Set up the matomo instance [http://localhost:5003](http://localhost:5003) (The public version is available at https://matomo-entitlementcard.tuerantuer.org)
+15. (optional) Add your matomo config to the backend config
+```yaml
+matomo:


shouldn't this be added to config-local.yaml?
Even i do that i can not start the backend since

Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "matomo" (class app.ehrenamtskarte.backend.config.BackendConfiguration), not marked as ignorable (7 known properties: "server", "production", "projects", "geocoding", "map", "postgres", "csvWriter"])

Where exactly do i have to add this @sarahsporck

you need to add this to the project config. that's in the config.local.yml e.g.

projects: - id: nuernberg.sozialpass.app ... matomo: ...

so not in the first level? probably this was my problem. i got an error when i started the backend with this additional parameters

could you please update the docs that its easier to know where to add matomo

f1sh1918 · 2023-09-06T13:18:57Z

docs/development-setup.md

+matomo:
+  siteId: 1
+  url: http://localhost:5003
+  accessToken: <matomo-access-token>


how can i find the token?

visit localhost:5003 and go to the settigns. Either search for "token" or go to "Persönlich->Sicherheit"

i had to adjust the config.ini.php on the docker container to change the trusted hosts and add the port
Not sure if that is just my problem. If not maybe this can be adjusted somehow

[General] salt = "36941bec3f510a9c9411000c89ee712b" trusted_hosts[] = "localhost:5003"

@sarahsporck please add how to findthe matomo-access-token and update the url to http://localhost:5003/matomo.php

f1sh1918

For me everything looks fine i just could need some help how to configure matomo locally to proper test this pr. The docker instance and everything works so far :)
And don't we have to put the matomo config for the backend in the config.yml for the different stages?

f1sh1918

works fine. just some updates to docs are missing :)
Amazing work!

f1sh1918 · 2023-09-19T08:22:42Z

docs/development-setup.md

+matomo:
+  siteId: 1
+  url: http://localhost:5003
+  accessToken: <matomo-access-token>


@sarahsporck please add how to findthe matomo-access-token and update the url to http://localhost:5003/matomo.php

frontend/lib/graphql/configured_graphql_provider.dart

steffenkleinle · 2023-09-20T11:03:22Z

frontend/lib/graphql/configured_graphql_provider.dart

+  State<StatefulWidget> createState() => ConfiguredGraphQlProviderState();
+}
+
+class ConfiguredGraphQlProviderState extends State<ConfiguredGraphQlProvider> {


What are we using this whole thing for? What events/things are we tracking from the frontend? How is the tracking with matomo connected to graphql?

steffenkleinle · 2023-09-20T11:14:58Z

backend/src/main/kotlin/app/ehrenamtskarte/backend/matomo/Matomo.kt

+
+object Matomo {
+    val logger = LoggerFactory.getLogger(Matomo::class.java)
+    private const val ACTION_NAME = "Request"


Do you plan to remove this then?

steffenkleinle · 2023-09-20T11:28:47Z

backend/src/main/kotlin/app/ehrenamtskarte/backend/matomo/Matomo.kt

+import java.io.IOException
+import java.util.concurrent.ExecutionException
+
+object Matomo {


Optional: I think in this file there is some duplicated stuff which might be reduced.

f1sh1918 · 2023-10-04T08:08:48Z

@sarahsporck when do you expect to finish/merge this pr?
I would like to create a release this week but could also do it without that

sarahsporck · 2023-10-05T07:17:35Z

@sarahsporck when do you expect to finish/merge this pr? I would like to create a release this week but could also do it without that

In the best szenario I'll finish today.

f1sh1918

for me its fine to merge @sarahsporck

sarahsporck added 2 commits August 16, 2023 08:38

875: add basic matomo client

a46c76f

875: add basic tracking requests

6b846ae

sarahsporck requested review from steffenkleinle and f1sh1918 as code owners September 4, 2023 08:17

sarahsporck commented Sep 4, 2023

View reviewed changes

administration/src/cards/createCards.ts Show resolved Hide resolved

875: revert adding graphql as direct dependency

26a3ce7

sarahsporck commented Sep 4, 2023

View reviewed changes

sarahsporck linked an issue Sep 4, 2023 that may be closed by this pull request

[Statistics] Collect Statistics #875

Closed

michael-markl reviewed Sep 5, 2023

View reviewed changes

f1sh1918 reviewed Sep 5, 2023

View reviewed changes

sarahsporck added 3 commits September 6, 2023 10:50

add matomo docker container

13e26a2

sanity check matomo config

7b6ca56

add support for category search

a4d313f

send async tracking request

1297ea6

sarahsporck force-pushed the collect-statistics branch from 6c5f020 to 1297ea6 Compare September 6, 2023 12:45

f1sh1918 reviewed Sep 6, 2023

View reviewed changes

fix formatting

e0cc104

f1sh1918 reviewed Sep 7, 2023

View reviewed changes

Merge branch 'main' into collect-statistics

826ef7f

f1sh1918 approved these changes Sep 19, 2023

View reviewed changes

steffenkleinle reviewed Sep 20, 2023

View reviewed changes

sarahsporck added 3 commits October 5, 2023 09:17

Merge branch 'main' into collect-statistics

aa7c642

remove unused code

071048b

Merge branch 'main' into collect-statistics

16e0bb6

f1sh1918 approved these changes Oct 5, 2023

View reviewed changes

sarahsporck merged commit 0b5523c into main Oct 5, 2023

sarahsporck deleted the collect-statistics branch October 5, 2023 13:12

sarahsporck mentioned this pull request Oct 5, 2023

Add Matomo Credentials #1133

Closed

sarahsporck mentioned this pull request Oct 16, 2023

Statistik: Anträge/Karten pro Stadt #585

Open

5 tasks

sarahsporck mentioned this pull request Oct 27, 2023

Move matomo.env into docker folder #1192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collect Statistics from Matomo #1113

Collect Statistics from Matomo #1113

sarahsporck commented Sep 4, 2023

sarahsporck Sep 4, 2023

sarahsporck Sep 4, 2023

sarahsporck Sep 6, 2023

steffenkleinle Sep 20, 2023

michael-markl left a comment

f1sh1918 left a comment

michael-markl commented Sep 5, 2023 •

edited

Loading

sarahsporck commented Sep 6, 2023

f1sh1918 Sep 6, 2023

sarahsporck Sep 6, 2023

f1sh1918 Sep 7, 2023

f1sh1918 Sep 7, 2023

f1sh1918 Sep 6, 2023

sarahsporck Sep 6, 2023

f1sh1918 Sep 7, 2023 •

edited

Loading

f1sh1918 Sep 19, 2023

f1sh1918 left a comment •

edited

Loading

f1sh1918 left a comment

f1sh1918 Sep 19, 2023

steffenkleinle Sep 20, 2023

steffenkleinle Sep 20, 2023

steffenkleinle Sep 20, 2023

f1sh1918 commented Oct 4, 2023 •

edited

Loading

sarahsporck commented Oct 5, 2023

f1sh1918 left a comment

Collect Statistics from Matomo #1113

Collect Statistics from Matomo #1113

Conversation

sarahsporck commented Sep 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-markl left a comment

Choose a reason for hiding this comment

f1sh1918 left a comment

Choose a reason for hiding this comment

michael-markl commented Sep 5, 2023 • edited Loading

sarahsporck commented Sep 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

f1sh1918 Sep 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

f1sh1918 left a comment • edited Loading

Choose a reason for hiding this comment

f1sh1918 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

f1sh1918 commented Oct 4, 2023 • edited Loading

sarahsporck commented Oct 5, 2023

f1sh1918 left a comment

Choose a reason for hiding this comment

michael-markl commented Sep 5, 2023 •

edited

Loading

f1sh1918 Sep 7, 2023 •

edited

Loading

f1sh1918 left a comment •

edited

Loading

f1sh1918 commented Oct 4, 2023 •

edited

Loading