Project 3 Part 2
Demonstrating use of various capabilities of using service mesh(istio) in our Project part 2.
Problem 1: Currently the authorization of end-user access to our service is done using boilerplate code i.e handled inside a micro-service. But there are chances of having security issues like man-in-the-middle attacks and many more and bugs could be easily incorporated in.
Problem 2: If we want to add a new role or permission(authorization) in the code-base, in order to reflect it to the production, we have to update all the services and deploy them once again.
Problem 3: Currently, only one version of the application is deployed in the production, suppose, we want to test the application on two different versions and record the user-interaction or response(metrics) at the same time, that part is not yet implemented.
Problem 4: If we want to add a new feature or update an existing feature in the production, the application needed some boot time in order to update all the services and it somewhere affected the end-users using it.
Istio offers two types of authentication
- Peer Authentication
- Request Authentication
Peer Authentication : We have implemented Peer Authentication using Mutual TLS which is used for Service to Service authentication. This provides handling the Man-in-the-middle attach on the services. Peer Authentication provides below benefits : i) Provides each service with a strong identity representing its role to enable interoperability across clusters and clouds. ii) Secures service-to-service communication. iii) Provides a key management system to automate key and certificate generation, distribution, and rotation.
Istio has inbuild Mutual TLS authentication. You can select STRICT or PERMISSIVE mTLS. Before enabling mTLS between pods, we should inject Sidecar Injection which creates local envoys of each pods i.e proxy pods. Authentication Process : Istio re-routes the outbound traffic from a client to the client’s local sidecar Envoy. The client side Envoy starts a mutual TLS handshake with the server side Envoy. During the handshake, the client side Envoy also does a secure naming check to verify that the service account presented in the server certificate is authorized to run the target service. The client side Envoy and the server side Envoy establish a mutual TLS connection, and Istio forwards the traffic from the client side Envoy to the server side Envoy. After authorization, the server side Envoy forwards the traffic to the server service through local TCP connections. To Enable Strict or Permissive mTLS checkout to Istio authentication and Authorization of IstioPart3 branch and run below command
kubectl apply -f tls-strict.yaml
To enforce Strict mTLS change mode to STRICT. By default it is PERMISSIVE.
Request Authentication : We have already implemented JWT authorization in our project 2. Request Authentication is used for end-user authentication to verify the credential attached to the request. Istio provides this authentication using custom authentication provider such as Auth0. We tried using the istio with Auth0 but we faced issue and Jetstream platform was not compatible with Auth0.
Check auth-policy.yaml file in same folder mentioned above.
A/B Testing The another problem, we identified was, if we have two versions of the application and we are not sure which will increase user interaction, so we tried deploying both versions at the same time and collect metrics based on the end-user metrics. In order to run this, Checkout the folder named -> A/B Testing
kubectl apply -f ab-testing-deployment.yaml
kubectl apply -f abdestinationrule.yaml
We created, two deployments of the UI service, version 1 and version 2. The only difference between two versions was UI background color change on the Sign-in Page
Like: Version 1:
Version 2:
We have achieved the result using Consistent Hash LoadBalancing. We hit the url maintaining two different IP’s and tested for the same (on two different computers by hitting the URL multiple times for each user independently).
The first step was to generate a consistent hashing according to the contents of the version.
To Test A/B testing we tested by passing many header specific to user and tried the curl command:
Here you can see that the REACT rendered main.chunk.js. But if we had have used HTML, CSS for designing front end, depending on the CSS changed for background color we would have got the respective css file changes. But in case of react, react only renders a single JS page. But if you see using UI, you can see that the session is maintained.
Conclusion: Depending on the header passed, that is unique to each user and with the help of consistent hashing we have maintained separate session affinity and from now on we can easily monitor end-users activity and do the analysis for both the versions.
There was no efficient rollout policy for the application. Any small change in the application required a full pipeline to be run and a complete replacement of existing service with the new service. The problem with this approach was that it was risky. If some error occurs in the production environment then it could lead to catastrophe. Hence we needed a mechanism where we can gradually rollout our application to a small percentage of users initially and then depending on the response and monitoring the requests we can increase our target audience for the new service.
Enhancement in the problem statement: While exploring canary deployments in Istio we got to know that there are also some advanced techniques for canary deployment (Dark Release). These techniques allow us to deploy a new version using Istio and making it available to the audience who fits certain criteria or fulfills certain conditions. UseCase of this enhancement: In the real world normally organizations have staging and production environments. The staging environment is used for all the testing before deploying the application on the production environment. But there is always some catch or some differences in the environments which cause the application to fail in the production environment. This technique helps us to deploy a new release on the production environment and making it available only for special users ( eg. Developers and Testers). Once the developers are sure that the basic functionalities are working then they can go ahead with rolling out to a percentage of audience and gradually increasing the number.
We explored different techniques for this strategy which looked similar but have a different impact. Canary releases suit the best for our needs. We explored the implementation of canary deployment in Kubernetes and Istio. The problem with Kubernetes canary releases is that there is no efficient mechanism to choose what percentage of traffic you want for the new canary deployment. It depends on the number of pods. If you want 10% traffic for version2 and 90% for version one then you'll have to create 10 replicas for the service and Kubernetes will automatically divide the traffic. But this is unnecessary scaling and utilization of resources. Istio provides a very flexible mechanism for weighted traffic hence we chose to go with Istio implementation.
Standard Canary Deployment
- The first step is to create a different version of the service and upload it to the docker hub. We changed the UI service for demo purposes. The change is in the login page where the email id place holder is replaced with login id. (Frontend/src/user/Signin.js)
- In the standard Kubernetes file we changed the uiDeployment.yaml. We added one more deployment with the new version and attached it to the same UI service.(Frontend/uiDeployment.yaml)
- We added an Istio specific YAML where we created a new virtual service with destination rules. ( IstioFiles/istio_rules_canary.yaml)
- We also explored advanced techniques for canary deployment with the imposition of certain conditions.
Dark Release:
- The implementation was basically header-based routing. If the request has a particular header then Istio will route the request to the new release.
- We decided to use our backend service "DataAnalysis" to implement dark release. But after failing several attempts we found that Istio does not support "header-propagation". This meant that we have to make changes in our code so that we can save and forward headers through different microservices.
- We are currently implementing this feature.
Github Commit Links: 86, 87, 88, 90 https://github.com/airavata-courses/devengers/commit/26db69181dc0af486bd7cb33241e3e90a417d509 https://github.com/airavata-courses/devengers/commit/cc693009a173048b254e86f73aa39a2c36a491ef https://github.com/airavata-courses/devengers/commit/7f2b210a023ac52df39677f7c36da773fa8f0aec https://github.com/airavata-courses/devengers/commit/41d53d0bae726d7c41c3553110e337fe4ca80072 https://github.com/airavata-courses/devengers/commit/25b49ee035cbf51bf90d9a768747e2f5b99f9879 https://github.com/airavata-courses/devengers/commit/523bd76ab7e57fbcc290ad7abf39e56bf8dedadd https://github.com/airavata-courses/devengers/commit/691f1c4912754934359f2c979efd635d1fcbe969
Evaluation: For evaluation, you can go to the homepage and refresh the page multiple times to see the change. the URL: http://129.114.104.27:32177/ If you want to change the settings then just change the percentage of weights in IstioFiles/istio_rules_canary.yaml and commit. Note: As per Istio implementation 50:50 ratio does not mean that alternate request will be transmitted to different versions. The implementation is not round-robin but random instead. So it is not guaranteed that ever 2nd request will be split but overall at random, the request is split as per the given ratio.
Conclusion: Istio is reliable and more flexible in the implementation of application rollouts and provides many additional features than the standard Kubernetes.
Contributions: Asim : Git Issues: 79,80,81,82,83,84,85,86,87,88, 89, 90,91,92,93,94, 95, 96 97
Git CommitS: Environment Setup https://github.com/airavata-courses/devengers/commit/a0711c4a00a615d4d0def9093d73f8f5e4a65e16 https://github.com/airavata-courses/devengers/commit/dfd176bf4a834d98723a3ed4d32d7fed9ffc80fc https://github.com/airavata-courses/devengers/commit/a1f7419f27fa842e196723a3526049b58b2362d1 https://github.com/airavata-courses/devengers/commit/b16307193fe7e0459d046e12d8be70ad4d588f95 https://github.com/airavata-courses/devengers/commit/54ba684574408567b24dcc1dfdd46442f86ee414 https://github.com/airavata-courses/devengers/commit/f2c963a10b40368905b374a16ba58bb6450f34ea https://github.com/airavata-courses/devengers/commit/d40bfd5f6a2c95ae0f8e20b324103e07b87b6d8d https://github.com/airavata-courses/devengers/commit/ba890cc21ff4811bb8116dad7d75b4e1829a3666
Istio Setup https://github.com/airavata-courses/devengers/commit/d53165779b5566da123fa8c4bc5d5bcf9f2c7293 https://github.com/airavata-courses/devengers/commit/c96e1e18d293f8cd24d3c36e8023f48b01c77675 https://github.com/airavata-courses/devengers/commit/40f01baf5fb7f42de9d4aea43088bac4f364c6b2 https://github.com/airavata-courses/devengers/commit/1b78e68b529bd870f86cfbf752f014e57c623d5b https://github.com/airavata-courses/devengers/commit/941ad81460dc97e5eb6465e811f28a14f548edf2 https://github.com/airavata-courses/devengers/commit/87634e0feecd9ec14836d639f8d50eeb7a99a7e5
Problem4: https://github.com/airavata-courses/devengers/commit/cc693009a173048b254e86f73aa39a2c36a491ef https://github.com/airavata-courses/devengers/commit/26db69181dc0af486bd7cb33241e3e90a417d509 https://github.com/airavata-courses/devengers/commit/7f2b210a023ac52df39677f7c36da773fa8f0aec https://github.com/airavata-courses/devengers/commit/41d53d0bae726d7c41c3553110e337fe4ca80072 https://github.com/airavata-courses/devengers/commit/25b49ee035cbf51bf90d9a768747e2f5b99f9879 https://github.com/airavata-courses/devengers/commit/523bd76ab7e57fbcc290ad7abf39e56bf8dedadd https://github.com/airavata-courses/devengers/commit/691f1c4912754934359f2c979efd635d1fcbe969
Load Balancing Exploration: https://github.com/airavata-courses/devengers/commit/79571c5e99a3dec0e4e0f7968c513ebec28e7d77