Introspected sources are overwritten on refresh

[boaz-wiesner]

Hey there :)

We're using introspection exclusively to get the current datasources. We have some temporary datasources in our cluster that are created and deleted daily. When turnilo auto introspects (as configured), it logs that it needs to remove some datasources from its state and add some new ones. However, in reality, all datasources are removed from turnilo's state, and only the newly added datasources remain. Even on further auto introspections, the removed datasources are not re-added.

I believe this to be a logical bug somewhere here:

turnilo/src/server/utils/cluster-manager/cluster-manager.ts

Line 317 in a78295a

public scanSourceList = (): Promise<void> => {

Restarting the turnilo server successfully resets the state and all datasource exist as they should, until the next deletion -> re-add.

Would appreciate any insight on this 😄

Thanks!

[adrianmroz-allegro]

We don't really use introspection and can't reproduce such scenarios on our end. I would need more information about what's happening and what do you expect to happen. More detailed logs would also help. Linked method looks good at first glance, it creates tasks that add/remove some externals based on new sources returned from Druid.

[boaz-wiesner]

No problem, I'll try to debug this on my end a bit more, was hoping you possibly had some more context. Will get back to you on this.

[boaz-wiesner]

I was able to replicate the issue on a toy local cluster I spun up, and ends up the issue is simply in the deletion of introspected sources.

The function deleteDataCube is called, and the filter there is wrong:

turnilo/src/common/models/sources/sources.ts

Lines 130 to 134 in a78295a

	export function deleteDataCube(sources: Sources, dataCube: DataCube): Sources {
	return {
	...sources,
	dataCubes: sources.dataCubes.filter(dc => dc.name === dataCube.name)
	};

Should instead be:

dataCubes: sources.dataCubes.filter(dc => dc.name !== dataCube.name

The current behavior is pretty clear 😄 Only the deleted datacubes are stored in the dataCubes object.

[adrianmroz-allegro]

It should be fixed with this version: https://github.com/allegro/turnilo/releases/tag/1.38.2-beta.0