Support for s3 for docker magic

[akranga]

At present '_component_builder.py' throws "ERROR:Error: 's3://....' should be a GCS path."

    220     self._gcs_base = gcs_base
    221     if not self._check_gcs_path(self._gcs_base):
--> 222       raise Exception('ImageBuild __init__ failure.')
    223     self._gcs_path = os.path.join(self._gcs_base, self._tarball_name)
    224     self._target_image = target_image

Because kaniko supports s3. It will probably will make sense to support it as well. In this case user can benefit from "local" minio.

[xiaozhouX]

Agree with that, I think we also need to use other type storage, like by defining a default pvc for kaniko.
And so do the Image url and registry credential.
Is pipeline have a plan for make some feature integration well with other cloudProvider? I would like to help for that, maybe with some PRs? @IronPan

[akranga]

I shall be happy to contribute as well. Few questions I have got in my mind.

1. To support backends like minio. We need somehow to configure boto and let kfp or magic now about it.

2. Kaniko spec generation. Currently it is not super flexible (like deliver registry secret). Which is ok. But need some directions in these area

[gaoning777]

The GCS directory is merely used for temporary build files and we are planning to replace the cloud dependency with k8s native configmap. Feel free to send a PR to use configmap to store temporary files such as kaniko spec, dockerfile, etc.
And, pushing to other registries is not supported for now. Kaniko supports it, though. This feature requires a little more work involving creating k8s secrets, mounting secrets to the kaniko build pod, etc. Feel free to send a PR. Thanks

[Ark-kun]

@gaoning777 AFAIK, GCS supports the s3:// protocol, so we could enable such URIs.

[IronPan]

Having kaniko cloud provider agnostic would be ideal. Kaniko supports local directory (ref) and my guess is creating a K8s emptyDir volume and mount to the kaniko container is good enough as the temporary storage.

@akranga @xiaozhouX does this sounds a good plan? Contribution is very welcomed.

[IronPan]

Also kaniko spec is currently located here Ideally, it should be parameterized so that the caller can decide where the resulted image stores to.

[gaoning777]

@gaoning777 AFAIK, GCS supports the s3:// protocol, so we could enable such URIs.

Yes, however the users need to set up the credentials for pushing to the S3.
Instructions are here.

[gaoning777]

duplicate of #345.