[Bug] Categorical Colormaps Failing

[acblackford]

Context

Datasets with categorical colormap renders are not being displayed in the dashboard. Selecting predefined colormap names for the renders do work, however. This does not throw any errors, and allows for a preview to deploy when working on a config PR, but the dataset will not populate in a story visualization or in the exploration/dataset tabs.

URL:

Blocking a release? (y/n): y

Regression issue? (y/n):
[If yes, add Regression label]

Priority: P2

Steps to reproduce the issue

1. Example sourceParams markdown (also issue in renders):
   colormap: '{"6":"#1e90c9", "12":"#ab94d6", "18": "#5a1680", "24": "#d943d4"}'

does not populate the dataset.

Selecting a markdown with colormap name:

colormap_name: blues_r

works as intended.

[hanbyul-here]

Can you share the complete configuration for this layer? We expect some flags for categorical legend work, and I'd like to check them.

[anayeaye]

I have looked at a few different working and not working categorical colormaps. Here are some notes and examples:

Notes
There are a couple ways that categorical colormaps are handled: for simple mappings, the colormap can be defined in the sourceParams directly (example: geoglam). For more complicated colormaps that are blocked by the firewall, we have been creating custom named colormaps in the raster-api that are passed as colormap_name in the sourceParams (example: nlcd). In all working examples colormap or colormap_name the raster tiles for a categorical map do not work properly without a categorical legend in the mdx file--I can work up an example to share.

Examples

collection: geoglam working categorical colormap

• categorical colormap
• working tile request https://openveda.cloud/api/raster/searches/5eed9169c72f068fa7a9cf9e779b04c3/tiles/WebMercatorQuad/1/0/0?assets=cog_default&colormap=%7B%221%22%3A+%5B120%2C+120%2C+120%5D%2C+%222%22%3A+%5B130%2C+65%2C+0%5D%2C+%223%22%3A+%5B66%2C+207%2C+56%5D%2C+%224%22%3A+%5B245%2C+239%2C+0%5D%2C+%225%22%3A+%5B241%2C+89%2C+32%5D%2C+%226%22%3A+%5B168%2C+0%2C+0%5D%2C+%227%22%3A+%5B0%2C+143%2C+201%5D%7D&bidx=1&unscale=false&resampling=nearest&max_size=1024&return_mask=true

collection: soil-texture working colormap_name + categorical legend

• categorical custom colormap (WAF restricted) colormap_name
• working tile request https://openveda.cloud/api/raster/searches/1e27a2ff56a95c11859f96664385bcca/tiles/WebMercatorQuad/1/0/1?assets=soil_texture_0cm_250m&colormap_name=soil_texture&nodata=255

collection: nlcd working colormap_name + categorical legend

• categorical custom colormap colormap_name
• working tile request https://openveda.cloud/api/raster/searches/76f81983c7b415eb77a306ebe88fbc54/tiles/WebMercatorQuad/3/2/3?assets=landcover&bidx=1&nodata=0&resampling=nearest&colormap_name=nlcd

collection: campfire-nlcd not working, categorical colormap

• categorical colormap probably would be blocked by WAF, should use custom colormap_name nlcd
• mdx https://github.com/NASA-IMPACT/veda-config/blob/34be34189e2923320edf587eb5107f071ed0d512/datasets/camp-fire-nlcd.data.mdx?plain=1#L2
• buggy map explore: https://www.earthdata.nasa.gov/dashboard/exploration?search=&datasets=%5B%7B%22id%22%3A%22campfire-nlcd%22%2C%22settings%22%3A%7B%22isVisible%22%3Atrue%2C%22opacity%22%3A100%2C%22analysisMetrics%22%3A%5B%7B%22id%22%3A%22mean%22%2C%22label%22%3A%22Average%22%2C%22chartLabel%22%3A%22Average%22%2C%22themeColor%22%3A%22infographicB%22%7D%2C%7B%22id%22%3A%22std%22%2C%22label%22%3A%22St+Deviation%22%2C%22chartLabel%22%3A%22St+Deviation%22%2C%22themeColor%22%3A%22infographicD%22%7D%5D%7D%7D%5D&taxonomy=%7B%7D

[hanbyul-here]

Recording some more findings:
There is a discrepancy between staging and production - while staging returns a valid response when the custom colormap value is passed as a hex value, the production returns an invalid response.

Compare staging: https://staging.openveda.cloud/api/raster/searches/46caa563f1a22309ab6d83cbea5ea80a/WebMercatorQuad/tilejson.json?assets=cog_default&colormap=%7B%2211%22%3A%22%23486DA2%22%7D

to production: https://openveda.cloud/api/raster/searches/46caa563f1a22309ab6d83cbea5ea80a/WebMercatorQuad/tilejson.json?assets=cog_default&colormap=%7B%2211%22%3A%22%23486DA2%22%7D

I could work around it by replacing hex value to rgb values

[anayeaye]

RE staging hex colormaps are working but not prod:

The WAF in production is blocking requests with the AWS#AWSManagedRulesSQLiRuleSet#SQLi_QUERYARGUMENTS rule. The sampled blocked requests are the hex colormaps. We could investigate an exemption for tiles requests if we can identify a safe pattern for that exemption