Istio, mTLS and the OSI layer

I have been playing a lot with Istio and recently tested mTLS encryption. The test, which I describe in this post, really materialized the OSI layer in front of my eyes. which is always interesting how new stuff can dust off your old basic knowledge.

The entire concept of service mesh and Istio is exciting and revolutionary in my view… but just like any new groundbreaking tech, it takes a few cycles to realize how it manifests beyond the papers, blogs and theory, at least for me. So, as I usually do, I share my experiences on this blog and in my sessions with other in the thought that if I can help even one person understand it better I have achieved my goal.

Quick background.

Istio is a service mesh open source project from Google providing a control plane on top of Envoy proxies. As I wrote in a previous post it provides an abstraction layer for micro-services, abstracting operations that are not part the micro-service’s business logic (its main function). The features this abstraction layer provides are many and significant and all operate at the service layer (Layer 7) of that good ol‘ OSI layer.( If you don’t know what the OSI layer is you have some reading to do).

Being a layer 7 service mean that it is operating within the boundaries of the protocols it understands which are gRPC, HTTP 1.1 & HTTP 2. It also means that when we speak of things like “routing”, it does not refer to network L3 routing rather steering traffic to a certain direction based on application level decisions such as header information or just application decision (send 5% of my traffic to service A).

Lately, I have been giving Istio’s mTLS encryption a lot of thought. “If I can encrypt ALL of my traffic, how does that affect micro-segmentation?” since mTLS will encrypt the traffic between all micro-services does that mean that all communications will pass over 443 now, disabling the ability to set FW policies effectively? (if everything is the same port I cannot differentiate traffic)

How very wrong. It’s like I overlooked every basic networking training I’ve had in my life, but I guess sometimes one needs to test something to really understand it, and that’s what I did.

I setup Kubernetes with NSX-T as the networking layer and Istio installed on top injecting the Envoy sidecar to each deployed pod and deployed the bookinfo app. I also deployed vRealize Log Insight in my network to be able to capture all the logs from the NSX-T distributed Firewall and set the default rule to forward all the packet by packet traffic logs to it

I then started clicking through the bookinfo app, generating micro-service to micro-service traffic and checked the logs

You can see from the log that the services talk with each other all on port 9080. Cool. now to enable mTLS encryption in Istio:

First I configured a “mesh policy” that forces all proxies to accept only mTLS encrypted communications like this:

That of course broke my application, because all micro-services are set to accept only encrypted traffic but no one yet sends traffic encrypted so nothing works. The next step then is to create a destination rule in k8s to send encryoted traffic like this:

That made my app work again, so now its time to look at the wire with vRLI

What do you know, all the services still communicate on the same port! And that my friends, is when I realized what I missed big time. Its not the communications that’s is encrypted, ports etc is layer 4 stuff. What is encrypted is the data, the payload, the important stuff. And it has nothing to do with L4 ports or the l4 firewall being affected.

I felt a bit silly, but I had to test it to see it and it gave me a new found respect for teh good ol‘ OSI layer.

Feel the same? feel free to comment or tweet at me, would love to get your perspective on things.

Niran