Blog Post: Vulnerable Microservices

[davidhadas]

This PR adds a blog post discussing "Vulnerable Microservices"

1. It explains why it s realistic to assume all services deployed on Kubernetes are vulnerable
2. It shows that using proper security-behavior instrumentation can protect deployed vulnerable services
3. It details 4 cyber use cases that users should cover in their service deployments
4. It shows that microservice architecture is well-suited to security-behavior instrumentation
5. It points to the Guard Open Source (Part of the CNCF Knative project) to offer security-behavior instrumentation

The idea to create this post came up while discussing options to raise the security awareness of 4 cyber use cases that need to be covered by Kubernetes users and the desire to point them to the external open-source project that targets covering these use cases. See #37356

[k8s-ci-robot]

Welcome @davidhadas!

It looks like this is your first PR to kubernetes/website 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/website has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[netlify]

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	8d326ad
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/63c1319ab0321f00094062e9
😎 Deploy Preview	https://deploy-preview-38104--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[sftim]

I'd like to get a review on this from SIG Security.

/sig security

[sftim]

Thanks for the article PR @davidhadas.

Overall, I think this article would find a better home on the Knative blog (update: I missed that Knative is a CNCF project).

If you remove the references to your own blog, and check before you publish it, we may be able to mirror / copy that article to the Kubernetes blog. However, as Knative is a Google project and not part of the Kubernetes project, we usually wouldn't do that.

[davidhadas]

Overall, I think this article would find a better home on the Knative blog.

Allow me to challenge this assessment.
Although there are aspects of this post that is useful also in the Knative blog,
this post was specifically created to raise the awareness of the Kubernetes community about a significant security measure that is needed when deploying microservices on Kubernetes. It uses language used in the Kubernetes community and analyzes the issues from the perspective of Kubernetes users.

Knative language is different and is presently focused on serverless use cases - so a post on the Knative blog will look different and will not be suitable as a Kubernetes blog post.

(btw, I am working with the Knative community to consider changing the focus from serverless to a more generic "Kubernetes automation suite", as much of the value of Knative automation can be used by users not targeting a pure "serverless" use case)

Also, afaik mirroring the exact same post on different blogs is a bad practice.

If you remove the references to your own blog, and check before you publish it, we may be able to mirror / copy that article to the Kubernetes blog.

I suggest this post will be published in the Kubernetes blog (after fixing any references and other aspects based on the review).

However, as Knative is a Google project and not part of the Kubernetes project, we usually wouldn't do that.

This is incorrect.
Historically, Knative use to be a google project, but this is no longer the case. This is a CNCF project (google has one member out of 6 members on the steering committee). See https://www.cncf.io/projects/knative/ - "Knative was accepted to CNCF on March 2, 2022 and is at the Incubating project maturity level."

btw, Kubernetes also use to be a google project, but today is a CNCF project.

[sftim]

I thought knative was a CNCF project. I missed the big CNCF logo when I checked.

Sorry about that.

[sftim]

IMO, either:

• we need to shift the message and call-to-action away from “come and get involved with this Knative subproject, Guard” and towards a more generic article about protecting cloud native services on Kubernetes,

or

• this belongs on the Knative blog, perhaps with a copy on the Kubernetes blog and synchronized publication

[davidhadas]

IMO, either:

• we need to shift the message and call-to-action away from “come and get involved with this Knative subproject, Guard” and towards a more generic article about protecting cloud native services on Kubernetes,

Sure, will do that.
Will change the call for action to be more appropriate.

[davidhadas]

/retest

[k8s-ci-robot]

@davidhadas: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

Let's aim to publish this on 2023-01-15 (between now and then, we'll release Kubernetes 1.26)

[sftim]

Overall: “Security-Behavior” is not idiomatic English. Write “security behavior”, and put the first use of this term in italics to emphasize that it is jargon.

---

(I'm reviewing as an editor for the Kubernetes blog team)

/hold
I'm concerned about several instances of misusing HTML and Markdown to get a particular rendering. This is not an approach we like because it is hard to maintain. We could add a convention for flagging articles that we know break our style conventions, so that we can regression test them.

However, it's much easier on other contributors if you follow the conventions rather than stretch or break them.
IMO, we should hold this PR until those style concerns are cleared.

[davidhadas]

Let's aim to publish this on 2023-01-15 (between now and then, we'll release Kubernetes 1.26)
Ignore the previous remark - I thought you suggested Dec 15th.
Will move to January 15th

[sftim]

BTW, the post-release blog schedule is in kubernetes/sig-release#2105

[sftim]

Thanks again for this PR.

---

• Now that you have switched to SVG images, you can omit the PNG versions.
• If you can, avoid Latin phrases and their abbreviations

[PushkarJ]

Not sure if I asked this earlier, but is there a link to a Getting Started Guide or a slack channel / mailing list or other place where people can actually go to get involved ? Right now this section encourages involvement without clear guidance on how.

[davidhadas]

ok, I will see what I can add

[davidhadas]

I added a link to the slack channel of the maintainers (it is not being moved to cncf)

[sftim]

I don't agree with this title. We should build consensus that the Kubernetes project is happy to endorse this message.

A less controversial message is to design as if your code is riddled with risk, but not to actually assert that it truly is.

[sftim]

(I don't want people to think that we're announcing a Kubernetes vulnerability, for example).

[davidhadas]

@sftim,
The title and content of this blog were discussed with the 6 reviewers from the Security-Tooling WG who reviewed it during a joint meeting we had to discuss this blog.

We discussed the main message and the reasoning for this title. The title was changed from "Vulnerable Microservices" as a result.

I stand behind this title 100% since it is exactly what we do want people to understand. That the microservices deployed are all vulnerable and should take this into account. The assumption that the deployed microservices are not vulnerable is exactly what we are trying to root out in this article. No, in practice, it is impossible to build a weakness-free microservice that will not be vulnerable.

This title does not say Kubernetes is vulnerable - it says that the deployed microservices are all vulnerable and it is about time the community will start accepting this reality and start acting accordingly.

[sftim]

/hold

Let's make sure the project as a whole is happy with the comms side of things before we ship it. It is a contentious title.

[sftim]

Aside: https://github.com/kubernetes/community/ doesn't list a WG Security Tooling.

[davidhadas]

Please let me know where should it be discussed beyond the discussion we already had in the sig-security-tooling WG meeting. I will try to make the case for accepting the post and its message. I thought we already passed this phase after the sig-security-tooling WG review, but I now understand that we did not.

I think it is great to get this posted in the k8s community blog (as it is very applicable to the community users and also important to the community developers) and be happy to explain the security aspects to whoever wishes to listen.

If this blog is not accepted, np and I will publish it elsewhere.

[reylejano]

The title and content of this blog were discussed with the 6 reviewers from the Security-Tooling WG who reviewed it during a joint meeting we had to discuss this blog.
...
Please let me know where should it be discussed beyond the discussion we already had in the sig-security-tooling WG meeting. I will try to make the case for accepting the post and its message. I thought we already passed this phase after the sig-security-tooling WG review, but I now understand that we did not.

SIG Docs' Blog subproject manages the review process for blogs. Discussion on the content and title can continue on this PR or on the #sig-docs-blog Slack channel. Folks from SIG Security can continue with the technical review on this PR and provide a /lgtm to confirm the technical review for this blog post.

[reylejano]

From @sftim

Let's make sure the project as a whole is happy with the comms side of things before we ship it. It is a contentious title.
...
From @davidhadas
I stand behind this title 100% since it is exactly what we do want people to understand. That the microservices deployed are all vulnerable and should take this into account. The assumption that the deployed microservices are not vulnerable is exactly what we are trying to root out in this article. No, in practice, it is impossible to build a weakness-free microservice that will not be vulnerable.

Let's tune the title so that it's not contentious and it conveys the author's message/intention.

@davidhadas, since deployed microservices are vulnerable (I agree with you on this), imo it's why a zero trust model should be applied everywhere, and this blog introduces security behavior, how about:

• Microservices Are Vulnerable — Analyze Their Security Behavior
• Yes, Your Microservices Are Vulnerable or Your Microservices Are Vulnerable
• Analyze the Security Behavior of Microservices in Kubernetes

I'm not the best writer but what I'm trying to do above is to keep the impact of "All Microservices are Vulnerable" but defer the context of "microservices" a bit

[divya-mohan0209]

Agree with Rey above.

[davidhadas]

Changed to "All Your Microservices Are Vulnerable"
If will not be accepted, will change to "Your Microservices Are Vulnerable"

[sftim]

This article seems to be about lateral movement, at least in part. It'd be nice to mention that jargon term in the article text.

[sftim]

Please rebase #38104 against the latest upstream main and squash it down to 1 commit. That'll help us rule out some issues.

[davidhadas]

This article seems to be about lateral movement, at least in part. It'd be nice to mention that jargon term in the article text.

The article is not about lateral movement.
It mostly discusses the first step in an attack aimed to compromise a microservice.

[davidhadas]

Please rebase #38104 against the latest upstream main and squash it down to 1 commit. That'll help us rule out some issues.

@sftim, I already rebased yesterday as you suggested on slack - this did not help. I will next try to squash to 1 commit...

[divya-mohan0209]

As a co-chair for SIG Docs, I agree with what Tim, our tech lead, has to say here. I believe a discussion with sig-security and others in the community needs to be had before pushing it forward since the Kubernetes blog is widely read and the title is contentious even if you, personally, stand by it. One of the best ways to move ahead is to have this brought up on the #sig-security slack channel and discussed.
cc: @reylejano @natalisucks

[divya-mohan0209]

Also, I understand that this article is trying to reference war and battles in the context of software. However, those wouldn't be my first choice of words in the spirit of keeping the article as accessible to and as inclusive of as many community members as possible.

[PushkarJ]

Thank you @divya-mohan0209 :) From purely technical perspective this blog makes good points and brings up an important topic of security behavior analytics of micro-services. Thank you @davidhadas for working on this for several days in the last couple of months.

However, in my personal opinion, there is definitely scope to improve on the language and the way the message is conveyed as mentioned earlier in above review comments from SIG Docs folks. I am not an expert on this though so I have limited my reviews so far only to technical aspects of the blog. I would also recommend adding a short note at the top saying that security guard is an example of a tool that helps solve the problem space described and is not an official recommendation by the community.

Since SIG Docs is the right place to discuss such topics and have folks with the right expertise to gain consensus on this, having a discussion about this in #sig-docs-blog instead of #sig-security would be a great next step.

[davidhadas]

My understanding is that we have already done all due process in sig-security to ensure that the meaning that is conveyed (in both title and the blog body) is correct, well-phrased from a technical point of view etc. If anyone disagrees, we can discuss this in sig-security channel/ WG meetings.

As for how we word what we wish to say (in both title and the blog body) - It always makes sense to continue and improve the wording while doing the utmost to keep the meaning and clarify this meaning rather than change it. If I understand correctly, sig-docs is the right place for such improvements.

[divya-mohan0209]

@davidhadas : Summarising our stand on this currently based on the past couple of comments,

• SIG Security will continue providing feedback on this from a technical perspective.
• We will need to articulate the content/title better in the spirit of keeping it more inclusive & accessible. Tim has already been working with you and we'll have more folks chime in with their suggestions. Some suggestions have already been provided by Rey and we'll continue to work closely with you to get this aligned to our content guidelines.
• We'd also like as many folks from the community providing feedback as possible. Since this falls under the blog subproject for sig docs, we'd request for consensus driving purposes that you post a link to this PR on the #sig-docs-blog slack channel with some context. If you need any help, please feel free to reach out to Tim, Rey, Natali, or me on Slack.

We understand that our review process may seem tedious. However, it is to ensure we are putting out content that is reflective of who we are as a community. Thank you for your patience as we work together to get this merged!

[divya-mohan0209]

Left some suggestions. Hope they're helpful

[divya-mohan0209]

Agree with Rey above.

[sftim]

Thanks again for this PR. The overall direction is reducing the number of things that are blockers for a merge, and you're also aligning with our style overall.

If you're willing to use sentence case for the Markdown headings, that's even better.

[sftim]

Suggested change

	Considering the current spread of offensive tools, sophistication of offensive players, and ever-growing cyber financial gains to offenders, any cyber strategy that relies on constructing a non-vulnerable, weakness-free service in 2023 is clearly too naïve. It seems the only viable strategy is to:
	Considering the current spread of offensive tools, sophistication of offensive players, and ever-growing cyber financial gains to offenders, any cyber strategy that relies on constructing a non-vulnerable, weakness-free service in 2023 is clearly too naïve. I'm asserting that the only viable strategy is to:

As this is:

• an opinion
• it's yours

and this is in a blog article, it's OK to bring in the first person.

Some readers are going to be less happy with “all deployed microservices are vulnerable” - it'll bristle with people who take pride in their work or who see the statement as simplistic. I'm proposing a framing to helps make it clear that you're stating this as opinion, not as the kind of factual statement that'd earn a [citation needed] on Wikipedia.

[davidhadas]

This is not a personal opinion.
It is a well-understood reality among hackers black and white hat.
It is also frequently spoken of in various security-related forums.

[sftim]

Is it plausible that a service doesn't have a vulnerability?

Honestly, I think we agree on the facts. What you're describing sounds like the motivation for zero trust architecture: treat network peers as hostile and avoid assuming that any service you interact with is well-behaved.

Guard helps meet UK NCSC zero trust design principle 3:

You should monitor health signals from your users and devices continuously, to evaluate confidence in their trustworthiness.

That framing would work fine. What I'm concerned about is making the assertion that not only should people design as if their code is vulnerable (for which they may read: awful), it actually is. This could and would offend.
If we publish something where it's clearly you as author making that assertion about their app / architecture, fine - you're welcome to. I definitely want to avoid also implying that the Kubernetes project officially endorses that view.

[davidhadas]

@sftim, we can have a technical discussion on this in sig-security. I am not at all sure we agree on the facts since while reading what you write, my understanding is that you seem to believe that developers can create non-vulnerable microservices, while I very clearly say the opposite.

Similar to the fact that you can't secure your home (or office) and make it non-vulnerable - protect it against a professional, well-motivated thief, you also can't secure your microservice and make it non-vulnerable - protect it against a professional, well-motivated hacker. This is exactly why governmental entities keep their sensitive services air-gapped and will never in their right mind keep highly sensitive information in any internet-connected system. They do so since they understand that everything is vulnerable and use the air gap to make it harder on the offenders (btw, as proven more than once, even with air-gaped systems they are vulnerable). Now, we can have a full discussion about why this is, but I think you will find that this concept that "everything is vulnerable" is well agreed upon by people in the field (not only by those working for governments ;) ).

Ergo, and more specifically to the applicable audience, All Microservices are Vulnerable.

I do understand this requires a mind shift from where your current beliefs are, the same as it requires a mind shift from where many readers will be when reading this post. But this is why this blog post was created. To help (a bit) create this mind shift. So, making the crystal clear statement that "Your microservices are vulnerable" is important in this context while tweaking this statement to say something softer is not helpful.

[usarid]

I guess the heart of this disagreement is whether people would treat the assertion that their services are vulnerable as an assertion that their code is awful and hence would be offended. I actually don’t think they would. I’m not quite sure how this is different than asserting that services are prone to failures, so one needs to design for failure. That’s the spirit of Kubernetes. We could even make it more explicit that tue assertion of vulnerability is not about the quality of the code but rather the nature of software and technology stacks and connectivity. And that should put the reader back in the mindset of realizing that designing for vulnerability is necessary, which is a really healthy outcome.

…

On Sat, Jan 7, 2023 at 11:39 AM David Hadas ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In content/en/blog/_posts/2023-01-15-Security-Bahavior-Analysis/index.md <#38104 (comment)>: > +layout: blog +title: "All Microservices Are Vulnerable" +date: 2023-01-15 +slug: security-behavior-analysis +--- + +**Author:** +David Hadas (IBM Research Labs) + +## Avoiding a False Sense of Security + +_This post warns Devops from a false sense of security. Following security best practices when developing and configuring microservices do not result in non-vulnerable microservices. The post shows that although all deployed microservices are vulnerable, there is much to be done to ensure microservices are not exploited. It explains how analyzing the behavior of clients and services from a security standpoint, named here **"Security-Behavior Analysis"**, can protect the deployed vulnerable microservices. It points to [Guard](http://knative.dev/security-guard), an open source project offering security-behavior monitoring and control of Kubernetes microservices presumed vulnerable._ + +As cyber war games continue to intensify in sophistication, organizations deploying cloud services, continue to grow their cyber investments aiming to produce safe and non-vulnerable services. However, the year-by-year growth in cyber investments, does not result in a parallel reduction in cyber incidents. Instead, the number of cyber incidents continues to grow annually. Evidently organizations are fighting a battle they are doomed to fail at. No matter how much effort is made to detect and remove cyber weaknesses from deployed services, it seems offenders have the upper hand. + +Considering the current spread of offensive tools, sophistication of offensive players, and ever-growing cyber financial gains to offenders, any cyber strategy that relies on constructing a non-vulnerable, weakness-free service in 2023 is clearly too naïve. It seems the only viable strategy is to: @sftim <https://github.com/sftim>, if you wish we can have a technical discussion on this in sig-security, but I really don't think we do agree on the facts as long as you believe that developers can create non-vulnerable microservices when they try hard enough (or that most of their microservices are not vulnerable and only a few are). Similar to the fact that you can't secure your home (or office) and make it non-vulnerable - protect it against a professional, well-motivated thief, you also can't secure your microservice and make it non-vulnerable - protect it against a professional, well-motivated hacker. This is exactly why governmental entities keep their sensitive services air-gapped and will never in their right mind keep highly sensitive information in any internet-connected system. They do so since they understand that everything is vulnerable and use the air gap to make it harder on the offenders (btw, as proven more than once, even with air-gaped systems they are vulnerable). Now, we can have a full discussion about why this is, but I think you will find that this concept that "everything is vulnerable" is well agreed upon by people in the field (not only by those working for governments ;) ). Ergo, and more specifically to the applicable audience, All Microservices are Vulnerable. I do understand this requires a mind shift from where your current beliefs are, the same as it requires a mind shift from where many readers will be when reading this post. But this is why this blog post was created. To help (a bit) create this mind shift. So, making the crystal clear statement that "Your microservices are vulnerable" is important in this context while tweaking this statement to say something softer is not helpful. — Reply to this email directly, view it on GitHub <#38104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAJXWYLWCNNJ5SYJA5O6PTWRHA5ZANCNFSM6AAAAAASMOMQFQ> . You are receiving this because you commented.Message ID: ***@***.***>

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[davidhadas]

Moved to #38918