Add System Certs when they are not available or overridden

[HumairAK]

The issue resolved by this Pull Request:

Resolves RHOAIENG-4709

Description of your changes:

When user or odh provides ca bundles, we combine them and use them as the authoritative cert file in dsp and dsp user pipeline pods. If these certs are missing the system certs, then we end up in a situation where user pods cannot download from pypi. So this pr makes it so that in such an event, we include these certs in our custom ca bundle.

Testing instructions

First reproduce the issue:

1. Deploy the odh-trusted-ca-bundle configmap:

odh-trusted-ca-bundle.yaml

apiVersion: v1
data:
  odh-ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    MIIFLTCCAxWgAwIBAgIUEkxySciftIMcrCGLZhtlodPa+xQwDQYJKoZIhvcNAQEL
    BQAwJjELMAkGA1UEBhMCWFgxFzAVBgNVBAMMDnJoLWRzcC1kZXZzLmlvMB4XDTI0
    MDMyMTE2NDYzNFoXDTM0MDMxOTE2NDYzNFowJjELMAkGA1UEBhMCWFgxFzAVBgNV
    BAMMDnJoLWRzcC1kZXZzLmlvMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKC
    AgEApqY3QFi0nVvBhUPHsdHSs2a0OMu8d5j3h1V/hoJ2K6YfX7a3/wdSNoOSckWq
    /mgmm7Bt+ffUfEBchywqo1RflFG3aURn1P7qkvDvCcEnP4zGtvtHTPCjpMMfjo3o
    iqEonExanoMNkzpWRcjt4bJM+jgvew9LgNaHul3dXVMN0fM0ojpv01zLDIfSQltJ
    zcFa910dYs/TpNpX9Eje1C5KFJ1CWgaqDZ8sQLKWMz1yaWfeDjDcsmWk6Dv8cOVP
    9cqZBHP/+CZYetFPgkii1eNIEVs//oyyJ016PeqbwAJj3JCx+XQqxWRU6RtfS9Wi
    /csWrwD7vBgFBT9jBBb8nwnJ035cP/v9rXAZdnef7r9WOsHsFudI7FEfvkkRpn+0
    QH1WSJ4maUguYr3mvocS6GsGoS0sMIfOihYVzF+yLkoVFotiVLACAEkoQcqisYUO
    /+ZlRXx7lOIYManwu9ixjkkVPYizqISn3xkeo4zLJ8CtDn1sAkmyf3c6ZVHnpGi1
    6SfvPfFCEevec/p7K3+caIRy3gYjMaCgcwVEbTqSrmAb+hZHTA6GUsNPOjIwl3tr
    niAkqufZGPagH3qDf2uCOPNgar2E9V+MMjqgmdTplKXjZwmsnPys2+GEtub8S9wy
    XxhQpRhcpv0ExROacBKWooQJdBQWY4joHcjxbDkk98XlObUCAwEAAaNTMFEwHQYD
    VR0OBBYEFGZKO5jPSR6pLT2mhdXUqNRBiSb5MB8GA1UdIwQYMBaAFGZKO5jPSR6p
    LT2mhdXUqNRBiSb5MA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIB
    AEBCftZdo+V9v5zv/p+y6UPG+W9fY7GLzjYkv9ku80X/I4isFjoovhQidMMz61U8
    xEbz7OcjgfbtqYeM7VDlslsJ8Rw9eql0CQJ/2b6mkfTq7/0Q483BjZPUNxdhIw57
    zDnxW24MSh5CMc5Ftze+9AhWH6hjlpmsvJaYhLG7vZF9g6vOt8bG66+8ehnYiJX7
    km8OgSrUOhPu5jKFk/lHGSc5G5i4IXcqpNQCHER4AP1GbGdElCx9+hX71oDq99TI
    rCUaHqsMq1NnZPf4yPMy6KM4uiYLwSp41t49dq5uugTkrzII5ohbz++k5+VjgF2r
    jkADHVPvgvXHczhK4eUm8cWUtPk4uIBJcmPf8JRWPM5IoToZUGrhx780CEoPuAtA
    dBZ1Gjy6UObkPjgql/BqnFFoL3X1H3Z/8oUi7uHIFD0+wsr1m8zqdgBMvr40IBXR
    plOwerecb3lVY/jCOw7CrCHRrrnm/+w9LTC5qUfkzt4uPiLklwZlf0cxK7Lk3yw7
    /sjbxnfwFmolbUVr+J6IfBN3a1ztR8UUAEgu/fz2yHOEZMHbh+GXI+g3XwBtchqs
    7NMUos1kSX/PGT7PZfWKeHRxHnf7eAHidXklhr8VJrFw7WjFVkt1yTMuMivTQlOX
    LlMbrgyUK4zc+x/tYiqaXbn6iODIOHLB2Ys/5oXrx9PE
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  annotations:
    config.openshift.io/inject-trusted-cabundle: "true"
  name: odh-trusted-ca-bundle

2. Then Deploy a simple dspa
3. Run the following pipeline (note the package_install)

example_pipeline.yaml

# PIPELINE DEFINITION
# Name: hello-pipeline
# Inputs:
#    recipient: str
# Outputs:
#    Output: str
components:
  comp-say-hello:
    executorLabel: exec-say-hello
    inputDefinitions:
      parameters:
        name:
          parameterType: STRING
    outputDefinitions:
      parameters:
        Output:
          parameterType: STRING
deploymentSpec:
  executors:
    exec-say-hello:
      container:
        args:
        - --executor_input
        - '{{$}}'
        - --function_to_execute
        - say_hello
        command:
        - sh
        - -c
        - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
          \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.3.0'\
          \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"'  &&\
          \  python3 -m pip install --quiet --no-warn-script-location 'boto3' 'botocore'\
          \ && \"$0\" \"$@\"\n"
        - sh
        - -ec
        - 'program_path=$(mktemp -d)


          printf "%s" "$0" > "$program_path/ephemeral_component.py"

          _KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         "$program_path/ephemeral_component.py"                         "$@"

          '
        - "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import\
          \ *\n\ndef say_hello(name: str) -> str:\n    print(\"starting boto connection\"\
          )\n\n    import boto3\n    from botocore.client import Config\n    print(\"\
          done connection\")\n    return \"done\"\n\n"
        image: quay.io/opendatahub/ds-pipelines-ci-executor-image:v1.0
pipelineInfo:
  name: hello-pipeline
root:
  dag:
    outputs:
      parameters:
        Output:
          valueFromParameter:
            outputParameterKey: Output
            producerSubtask: say-hello
    tasks:
      say-hello:
        cachingOptions:
          enableCache: true
        componentRef:
          name: comp-say-hello
        inputs:
          parameters:
            name:
              componentInputParameter: recipient
        taskInfo:
          name: say-hello
  inputDefinitions:
    parameters:
      recipient:
        parameterType: STRING
  outputDefinitions:
    parameters:
      Output:
        parameterType: STRING
schemaVersion: 2.1.0
sdkVersion: kfp-2.3.0

The pipeline should fail, confirm that you see the error in the launcher main container:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))': /simple/kfp/

Then deploy the DSPO with the fixes, and try running the same pipeline again. It should succeed.

Checklist

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[dsp-developers]

A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-614
An OCP cluster where you are logged in as cluster admin is required.

To use this image run the following:

cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines-operator.git
cd data-science-pipelines-operator/
git fetch origin pull/614/head
git checkout -b pullrequest 72cd5dcc8f48abb95c3ae31ba28aaa7c76dcd1e1
oc new-project opendatahub
make deploy IMG="quay.io/opendatahub/data-science-pipelines-operator:pr-614"

More instructions here on how to deploy and test a Data Science Pipelines Application.

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-614

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-614

[amadhusu]

I was able to reproduce the original issue and verified this fix did the trick, also did a sanity check with our sample pipeline to make sure.

P.S. Noticed some pip package errors due to dependencies with kfp 2.3.0 which was used to create this IR.yaml but I think those can be ignored.

Original Issue

Working Solution

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amadhusu
Once this PR has been reviewed and has the lgtm label, please ask for approval from humairak. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment