An unexpected error occurred when hmac-auth credential resources created.

[sh1yu]

Summary

Kong Ingress Controller throw a error when KongConsumer and KongCredential resources created:

failed to sync: posting new config to /config: 500 Internal Server Error {"message":"An unexpected error occurred"}

Kong Ingress controller version
Kong Ingress controller
Release: 0.10.0
Build: 91835c8
Repository: git@github.com:Kong/kubernetes-ingress-controller.git
Go: go1.15.2

Kong version
2.1.4

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:22:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Environment

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master Ready master 220d v1.17.2 172.31.114.129 CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://18.9.7
k8s-node-116 Ready 211d v1.17.2 172.31.114.116 CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.7
k8s-node-128 Ready 220d v1.17.2 172.31.114.128 CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://18.9.7
k8s-node-135 Ready 219d v1.17.2 172.31.114.135 CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://18.9.7

What happened

Controller got a error when KongCredential resource created:

time="2020-09-30T05:55:48Z" level=warning msg="deprecated KongCredential resource in use; please use secret-based credentials, KongCredential resource will be removed in future" component=store kongconsumer_name=demo-hmac-consumer kongconsumer_namespace=default
time="2020-09-30T05:55:48Z" level=error msg="failed to update kong configuration: posting new config to /config: 500 Internal Server Error {\"message\":\"An unexpected error occurred\"}" component=controller
time="2020-09-30T05:55:48Z" level=error msg="failed to sync: posting new config to /config: 500 Internal Server Error {\"message\":\"An unexpected error occurred\"}" component=sync-queue
time="2020-09-30T05:55:48Z" level=warning msg="requeuing sync for 'default/default-token-nxlbf'" component=sync-queue

and Proxy (kong) print a error stack:

2020/09/30 05:57:45 [error] 23#0: *252371 lua coroutine: runtime error: ...are/lua/5.1/kong/db/schema/others/declarative_config.lua:337: attempt to concatenate field 'value' (a table value)
stack traceback:
coroutine 0:
	...are/lua/5.1/kong/db/schema/others/declarative_config.lua: in function 'validate_references'
	...are/lua/5.1/kong/db/schema/others/declarative_config.lua:639: in function 'flatten'
	/usr/local/share/lua/5.1/kong/db/declarative/init.lua:234: in function 'parse_table'
	/usr/local/share/lua/5.1/kong/api/routes/config.lua:76: in function 'fn'
	/usr/local/share/lua/5.1/kong/api/api_helpers.lua:258: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:241>
coroutine 1:
	[C]: in function 'resume'
	/usr/local/share/lua/5.1/lapis/application.lua:397: in function 'handler'
	/usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
	/usr/local/share/lua/5.1/lapis/application.lua:167: in function </usr/local/share/lua/5.1/lapis/application.lua:165>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/lapis/application.lua:173: in function 'dispatch'
	/usr/local/share/lua/5.1/lapis/nginx.lua:230: in function 'serve'
	/usr/local/share/lua/5.1/kong/init.lua:1210: in function 'admin_content'
	content_by_lua(nginx-kong.conf:243):2: in main chunk, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/2.0", host: "localhost:8444"
2020/09/30 05:57:45 [error] 23#0: *252371 [lua] api_helpers.lua:429: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:401: ...are/lua/5.1/kong/db/schema/others/declarative_config.lua:337: attempt to concatenate field 'value' (a table value)
stack traceback:
	...are/lua/5.1/kong/db/schema/others/declarative_config.lua: in function 'validate_references'
	...are/lua/5.1/kong/db/schema/others/declarative_config.lua:639: in function 'flatten'
	/usr/local/share/lua/5.1/kong/db/declarative/init.lua:234: in function 'parse_table'
	/usr/local/share/lua/5.1/kong/api/routes/config.lua:76: in function 'fn'
	/usr/local/share/lua/5.1/kong/api/api_helpers.lua:258: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:241>

stack traceback:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/lapis/application.lua:401: in function 'handler'
	/usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
	/usr/local/share/lua/5.1/lapis/application.lua:167: in function </usr/local/share/lua/5.1/lapis/application.lua:165>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/lapis/application.lua:173: in function 'dispatch'
	/usr/local/share/lua/5.1/lapis/nginx.lua:230: in function 'serve'
	/usr/local/share/lua/5.1/kong/init.lua:1210: in function 'admin_content'
	content_by_lua(nginx-kong.conf:243):2: in main chunk, client: 127.0.0.1, server: kong_admin, request: "POST /config?check_hash=1 HTTP/2.0", host: "localhost:8444"
127.0.0.1 - - [30/Sep/2020:05:57:45 +0000] "POST /config?check_hash=1 HTTP/2.0" 500 42 "-" "Go-http-client/2.0"

Expected behavior

KongConsumer and KongCredential resource create success and kubernetes ingress controller process normally.

Steps To Reproduce

1. install controller with helm3:

 helm install kong/kong --generate-name --set ingressController.installCRDs=false

2. apply KongConsumer resource:

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: demo-hmac-consumer
  annotations:
    kubernetes.io/ingress.class: kong
username: demo-user
#credentials:
#- demo-hmac-apikey

3. apply KongCredential resource:

apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: demo-hmac-credential
consumerRef: demo-hmac-consumer
type: hmac-auth
config:
  username: demo-user
  secret: "123456"

4. view logs:

kubectl logs  kong-1601176600-kong-755549d499-mthqg  -c  ingress-controller --tail 10 -f
kubectl logs  kong-1601176600-kong-755549d499-mthqg  -c  proxy --tail 10 -f

[mflendrich]

TBD: try to reproduce this issue

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.