Unique value count

[MaximMoinat]

This change adds the number of unique values in the overview tab, together with the fraction unique (100% meaning a column with only unique values, 0% with a constant value).

Also, the fraction columns are styled as percentage (rounded to integer percent).

[schuemie]

Hmmm, one caveat here: there's a maximum on the number of unique values that WhiteRabbit will keep in memory: https://github.com/OHDSI/WhiteRabbit/blob/master/src/org/ohdsi/whiteRabbit/scan/SourceDataScan.java#L49

The default number of rows to scan (100,000) is the same as this maximum, but if people set the number of rows higher these new statistics might suggest fewer unique values than actually encountered.

I suggest that if the number of unique values in memory is equal to the maximum (or maybe we set a flag if the max is hit), we modify the statistics here to say 'greater than or equal'. So for example:
N unique >= 100,000, Fraction unique >= 10%.

[MaximMoinat]

In that case, the number of counted unique values will actually be greater than the actual number. The discarded values are potentially counted multiple times. See these lines:
https://github.com/thehyve/OHDSI-WhiteRabbit/blob/unique-stat/src/org/ohdsi/whiteRabbit/scan/SourceDataScan.java#L479-L480
(these lines were added because of issues caused by trimming of the fieldInfo object directly after processing all values)

Lets indeed go with your proposal. At maximum 100,000 unique values should be counted and if that limit is hit, report the stats as you mentioned (>= 100,000, >= xx%). We should be able to reuse the tooManyValues flag for this.

[schuemie]

Ah, I now see you introduced the uniqueCount variable, which I think is redundant with valueCounts.size().

[MaximMoinat]

My initial implementation indeed used valueCounts.size(). However, when I tried this the unique count was at maximum the 'max distinct values' (by default 1000). This is caused by the truncation of the valueCounts object before generating the scan report. So not all unique values are kept after the file/table has been analysed.

WhiteRabbit/src/org/ohdsi/whiteRabbit/scan/SourceDataScan.java

Lines 343 to 346 in d8ff935

	for (FieldInfo fieldInfo : fieldInfos)
	fieldInfo.trim();
	return fieldInfos;

[MaximMoinat]

Amended as suggested by @schuemie. The unique count and unique fraction will now be marked with '<=' when the unique count is more than the MAX_VALUES_IN_MEMORY. In these cases the unique count will be an overestimate, as an intermittent trimming of the value counts is done (keeping only topN).

By the way, what is the reason that the trimming is done only once? This does not guarantee that the size of the valueCounts is always smaller than the MAX_VALUES_IN_MEMORY.

WhiteRabbit/src/org/ohdsi/whiteRabbit/scan/SourceDataScan.java

Lines 436 to 439 in d8ff935

	if (!tooManyValues && valueCounts.size() > MAX_VALUES_IN_MEMORY) {
	tooManyValues = true;
	valueCounts.keepTopN(maxValues);
	}

[blootsvoets]

Looks good, some minor comments