Performance and failure risk due to complete user-owned image scan

[gbmeuk]

Hi,

Recently I came across, what I believe to be an issue with the way Moray checks and cross references all image information between docker_images table and imgapi_images tables.

First let me explain the situation I found myself in. Due to limited resources in my lab environment, I amended the standard packages on many of the Triton components because the systems were oversubscribed and wouldn't really ever be under heavy usage. One of my mistakes was not considering how quickly diskspace would be consumed when I compared zfs used and altered accordingly.

However, the situation that this left me in was one where Postgres instances ran out of space, and I can only presume were unable to complete some operation which left records in table docker_images that referred to a image_uuids in imgapi_images that were not present.

What brought extra confusion was when I was trying to build from previous docker files and images that had built correctly only days before. Upon further investigation, I found the cause of the issue, deleted the superfluous records from docker_images and the issue was resolved. However, it did raise an issue in my head that could be a problem for others, particularly performance issues for users with large numbers of images.

There may be some very good reason that I'm unaware of, it's just that it strikes me that SELECT *, '89ce1152-7d51-46f2-9110-371b778d2b78' AS req_id FROM docker_images WHERE ( owner_uuid = $1 AND owner_uuid IS NOT NULL ) LIMIT 1000 (and similar for imgapi_image should also be including some other field to more specifically return the relevant details. This would ensure better performance under situations where heavy build users have lots of records of images and provide more unique failure scenarios that can be handled/troubleshooted better.

Some log details
[root@1d73f8a1-e08a-432c-bfa4-7c665c3d9366 (GB-Home:moray0) ~]# grep c734531f-cd72-66b9-7d3e-8436063ea380 /var/log/moray.log | bunyan [2016-09-13T11:38:15.987Z] DEBUG: moray/PGClient/93295 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: query: row (moray_id=1) client: { "id": 1, "currentQuery": "SELECT *, '89ce1152-7d51-46f2-9110-371b778d2b78' AS req_id FROM docker_images WHERE ( owner_uuid = $1 AND owner_uuid IS NOT NULL ) LIMIT 1000", "timeout": 30000, "txn": true } -- row: { "_id": 385, "_txn_snap": null, "_key": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae-docker.io-5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "_value": "{\"author\":\"SequenceIQ\",\"architecture\":\"\",\"comment\":\"\",\"created\":1421312452025,\"config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":null,\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"container_config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:f1cd99b17a20d0bc64f00e989b745a3eec515ce8b042861e2ed59b3f398771cd in /usr/local/hadoop/etc/hadoop/core-site.xml.template\"],\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"docker_id\":\"5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1\",\"image_uuid\":\"c734531f-cd72-66b9-7d3e-8436063ea380\",\"index_name\":\"docker.io\",\"owner_uuid\":\"1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae\",\"parent\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"private\":false,\"heads\":[\"c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806\"],\"size\":154,\"virtual_size\":371624627}", "_etag": "508963B2", "_mtime": "1473197083280", "_vnode": null, "docker_id": "5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "head": null, "image_uuid": "c734531f-cd72-66b9-7d3e-8436063ea380", "index_name": "docker.io", "owner_uuid": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae", "parent": "bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5", "heads": [ "c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806" ], "_rver": 4, "req_id": "89ce1152-7d51-46f2-9110-371b778d2b78" } [2016-09-13T11:38:15.987Z] DEBUG: moray/93295 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: getRecords: row found (req_id=89ce1152-7d51-46f2-9110-371b778d2b78) row: { "_id": 385, "_txn_snap": null, "_key": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae-docker.io-5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "_value": "{\"author\":\"SequenceIQ\",\"architecture\":\"\",\"comment\":\"\",\"created\":1421312452025,\"config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":null,\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"container_config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:f1cd99b17a20d0bc64f00e989b745a3eec515ce8b042861e2ed59b3f398771cd in /usr/local/hadoop/etc/hadoop/core-site.xml.template\"],\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"docker_id\":\"5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1\",\"image_uuid\":\"c734531f-cd72-66b9-7d3e-8436063ea380\",\"index_name\":\"docker.io\",\"owner_uuid\":\"1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae\",\"parent\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"private\":false,\"heads\":[\"c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806\"],\"size\":154,\"virtual_size\":371624627}", "_etag": "508963B2", "_mtime": "1473197083280", "_vnode": null, "docker_id": "5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "head": null, "image_uuid": "c734531f-cd72-66b9-7d3e-8436063ea380", "index_name": "docker.io", "owner_uuid": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae", "parent": "bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5", "heads": [ "c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806" ], "_rver": 4, "req_id": "89ce1152-7d51-46f2-9110-371b778d2b78" } [2016-09-13T11:49:59.922Z] DEBUG: moray/PGClient/93287 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: query: row (moray_id=4) client: { "id": 4, "currentQuery": "SELECT *, '956a5d18-0504-4e65-8c54-66164c17846a' AS req_id FROM docker_images WHERE ( owner_uuid = $1 AND owner_uuid IS NOT NULL ) LIMIT 1000", "timeout": 30000, "txn": true } -- row: { "_id": 385, "_txn_snap": null, "_key": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae-docker.io-5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "_value": "{\"author\":\"SequenceIQ\",\"architecture\":\"\",\"comment\":\"\",\"created\":1421312452025,\"config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":null,\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"container_config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:f1cd99b17a20d0bc64f00e989b745a3eec515ce8b042861e2ed59b3f398771cd in /usr/local/hadoop/etc/hadoop/core-site.xml.template\"],\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"docker_id\":\"5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1\",\"image_uuid\":\"c734531f-cd72-66b9-7d3e-8436063ea380\",\"index_name\":\"docker.io\",\"owner_uuid\":\"1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae\",\"parent\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"private\":false,\"heads\":[\"c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806\"],\"size\":154,\"virtual_size\":371624627}", "_etag": "508963B2", "_mtime": "1473197083280", "_vnode": null, "docker_id": "5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "head": null, "image_uuid": "c734531f-cd72-66b9-7d3e-8436063ea380", "index_name": "docker.io", "owner_uuid": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae", "parent": "bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5", "heads": [ "c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806" ], "_rver": 4, "req_id": "956a5d18-0504-4e65-8c54-66164c17846a" } [2016-09-13T11:49:59.922Z] DEBUG: moray/93287 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: getRecords: row found (req_id=956a5d18-0504-4e65-8c54-66164c17846a) row: { "_id": 385, "_txn_snap": null, "_key": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae-docker.io-5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "_value": "{\"author\":\"SequenceIQ\",\"architecture\":\"\",\"comment\":\"\",\"created\":1421312452025,\"config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":null,\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"container_config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:f1cd99b17a20d0bc64f00e989b745a3eec515ce8b042861e2ed59b3f398771cd in /usr/local/hadoop/etc/hadoop/core-site.xml.template\"],\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"docker_id\":\"5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1\",\"image_uuid\":\"c734531f-cd72-66b9-7d3e-8436063ea380\",\"index_name\":\"docker.io\",\"owner_uuid\":\"1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae\",\"parent\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"private\":false,\"heads\":[\"c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806\"],\"size\":154,\"virtual_size\":371624627}", "_etag": "508963B2", "_mtime": "1473197083280", "_vnode": null, "docker_id": "5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "head": null, "image_uuid": "c734531f-cd72-66b9-7d3e-8436063ea380", "index_name": "docker.io", "owner_uuid": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae", "parent": "bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5", "heads": [ "c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806" ], "_rver": 4, "req_id": "956a5d18-0504-4e65-8c54-66164c17846a" } [2016-09-13T11:50:11.612Z] DEBUG: moray/PGClient/93317 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: query: row (moray_id=6) client: { "id": 6, "currentQuery": "SELECT *, 'dce36351-d75e-472e-b788-5a9cfbee9c1c' AS req_id FROM docker_images WHERE ( ( ( docker_id = $1 AND docker_id IS NOT NULL ) ) AND ( ( owner_uuid = $2 AND owner_uuid IS NOT NULL ) ) AND ( ( index_name = $3 AND index_name IS NOT NULL ) ) ) LIMIT 1000", "timeout": 30000, "txn": true } -- row: { "_id": 385, "_txn_snap": null, "_key": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae-docker.io-5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "_value": "{\"author\":\"SequenceIQ\",\"architecture\":\"\",\"comment\":\"\",\"created\":1421312452025,\"config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":null,\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"container_config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:f1cd99b17a20d0bc64f00e989b745a3eec515ce8b042861e2ed59b3f398771cd in /usr/local/hadoop/etc/hadoop/core-site.xml.template\"],\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"docker_id\":\"5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1\",\"image_uuid\":\"c734531f-cd72-66b9-7d3e-8436063ea380\",\"index_name\":\"docker.io\",\"owner_uuid\":\"1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae\",\"parent\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"private\":false,\"heads\":[\"c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806\"],\"size\":154,\"virtual_size\":371624627}", "_etag": "508963B2", "_mtime": "1473197083280", "_vnode": null, "docker_id": "5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "head": null, "image_uuid": "c734531f-cd72-66b9-7d3e-8436063ea380", "index_name": "docker.io", "owner_uuid": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae", "parent": "bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5", "heads": [ "c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806" ], "_rver": 4, "req_id": "dce36351-d75e-472e-b788-5a9cfbee9c1c" } [2016-09-13T11:50:11.613Z] DEBUG: moray/93317 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: getRecords: row found (req_id=dce36351-d75e-472e-b788-5a9cfbee9c1c) row: { "_id": 385, "_txn_snap": null, "_key": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae-docker.io-5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "_value": "{\"author\":\"SequenceIQ\",\"architecture\":\"\",\"comment\":\"\",\"created\":1421312452025,\"config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":null,\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"container_config\":{\"Hostname\":\"70b4a57bb473\",\"Domainname\":\"\",\"User\":\"root\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"HOME=/\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin\",\"JAVA_HOME=/usr/java/default\",\"HADOOP_PREFIX=/usr/local/hadoop\",\"HADOOP_COMMON_HOME=/usr/local/hadoop\",\"HADOOP_HDFS_HOME=/usr/local/hadoop\",\"HADOOP_MAPRED_HOME=/usr/local/hadoop\",\"HADOOP_YARN_HOME=/usr/local/hadoop\",\"HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop\",\"YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ADD file:f1cd99b17a20d0bc64f00e989b745a3eec515ce8b042861e2ed59b3f398771cd in /usr/local/hadoop/etc/hadoop/core-site.xml.template\"],\"Image\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"Volumes\":null,\"VolumeDriver\":\"\",\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"Labels\":null},\"docker_id\":\"5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1\",\"image_uuid\":\"c734531f-cd72-66b9-7d3e-8436063ea380\",\"index_name\":\"docker.io\",\"owner_uuid\":\"1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae\",\"parent\":\"bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5\",\"private\":false,\"heads\":[\"c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806\"],\"size\":154,\"virtual_size\":371624627}", "_etag": "508963B2", "_mtime": "1473197083280", "_vnode": null, "docker_id": "5c09c81ffffd767bbbb42cfc094856b680c31e78a7dce5559edc861335b6fca1", "head": null, "image_uuid": "c734531f-cd72-66b9-7d3e-8436063ea380", "index_name": "docker.io", "owner_uuid": "1c7d5ff0-6c69-c5e2-b7d5-aa7853714fae", "parent": "bd0a4bca6161c19e55a7137a90331b70cff7cf071cc433b533b4d0871081a4d5", "heads": [ "c6af36cfa682f2a47bd7bebd531ffa6a2f9260c9faeb16f15338af084b400806" ], "_rver": 4, "req_id": "dce36351-d75e-472e-b788-5a9cfbee9c1c" } [2016-09-13T11:50:11.616Z] DEBUG: moray/93317 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: getObject: entered (req_id=b01f0b50-fcd6-c8dc-fc89-8db1eeceb4d5, bucket=imgapi_images, key=c734531f-cd72-66b9-7d3e-8436063ea380) opts: { "req_id": "b01f0b50-fcd6-c8dc-fc89-8db1eeceb4d5", "headers": {}, "no_count": false, "sql_only": false, "noCache": true } [2016-09-13T11:50:11.616Z] DEBUG: moray/93317 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: loadObject: entered (req_id=b01f0b50-fcd6-c8dc-fc89-8db1eeceb4d5, bucket=imgapi_images, key=c734531f-cd72-66b9-7d3e-8436063ea380) [2016-09-13T11:50:11.616Z] DEBUG: moray/PGClient/93317 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: pg.query: started (moray_id=2) client: { "id": 2, "currentQuery": "SELECT *, 'b01f0b50-fcd6-c8dc-fc89-8db1eeceb4d5' AS req_id FROM imgapi_images WHERE _key=$1", "timeout": 30000, "txn": false } -- sql: SELECT *, 'b01f0b50-fcd6-c8dc-fc89-8db1eeceb4d5' AS req_id FROM imgapi_images WHERE _key=$1 -- args: [ "c734531f-cd72-66b9-7d3e-8436063ea380" ] [2016-09-13T11:50:11.619Z] DEBUG: moray/93317 on 1d73f8a1-e08a-432c-bfa4-7c665c3d9366: getObject: failed (req_id=b01f0b50-fcd6-c8dc-fc89-8db1eeceb4d5) ObjectNotFoundError: imgapi_images::c734531f-cd72-66b9-7d3e-8436063ea380 does not exist at EventEmitter.<anonymous> (/opt/smartdc/moray/lib/objects/get.js:71:16) at EventEmitter.g (events.js:180:16) at EventEmitter.emit (events.js:95:17) at endOrError (/opt/smartdc/moray/lib/pg.js:97:13) at f (/opt/smartdc/moray/node_modules/once/once.js:16:25) at onQueryEnd (/opt/smartdc/moray/lib/pg.js:124:9) at EventEmitter.emit (events.js:95:17) at Query.handleReadyForQuery (/opt/smartdc/moray/node_modules/pg/lib/query.js:86:8) at null.<anonymous> (/opt/smartdc/moray/node_modules/pg/lib/client.js:159:19) at EventEmitter.emit (events.js:117:20) [root@1d73f8a1-e08a-432c-bfa4-7c665c3d9366 (GB-Home:moray0) ~]#

Originally I raised this as a Moray issue but quite rightly seems like it is a client side problem