Kubernetes Networking

• Introduction
• NetworkPolicy
• Nginx Ingress Controller
• Contour Ingress Controller
• Gateway API
• Kube-proxy
• Multicloud communication for Kubernetes
• Multi-Cluster Kubernetes Networking
• Kubernetes Network Policy
  • Cilium
  • Kubernetes Network Policy Samples
• Kubernetes Ingress Specification
• Xposer Kubernetes Controller To Manage Ingresses
• Software-Defined IP Address Management (IPAM)
• CNI Container Networking Interface
  • List of existing CNI Plugins (IPAM)
  • Project Calico
• DNS Service with CoreDNS
• Kubernetes Node Local DNS Cache
• Kubernetes Sidecars
• Videos
• Tweets

Introduction

• kubernetes.io: The Kubernetes network model. How to implement the Kubernetes networking model
• ovh.com - getting external traffic into kubernetes: clusterip, nodeport, loadbalancer and ingress
• learnk8s.io: Load balancing and scaling long-lived connections in Kubernetes 🌟 Kubernetes doesn’t load balance long-lived connections, and some Pods might receive more requests than others. If you’re using HTTP/2, gRPC, etc. or any other long-lived connection, you might want to consider client-side load balancing
• stackrox.com: Kubernetes Networking Demystified: A Brief Guide
• medium.com: Fighting Service Latency in Microservices With Kubernetes
• medium.com: Kubernetes NodePort vs LoadBalancer vs Ingress? When should I use what? 🌟
• blog.alexellis.io: Get a LoadBalancer for your private Kubernetes cluster
• dustinspecker.com: How Do Kubernetes and Docker Create IP Addresses?!
• youtube: Kubernetes Ingress Explained Completely For Beginners
• AWS and Kubernetes Networking Options and Trade-Offs (part 1)
• AWS and Kubernetes Networking Options and Trade-Offs (part 2)
• AWS and Kubernetes Networking Options and Trade-Offs (part 3)
• medium: Service Types in Kubernetes? 🌟 A Service enables network access to a set of Pods in Kubernetes.
• containo.us: Kubernetes Ingress & Service API Demystified
• speakerdeck.com: Kubernetes and networks. Why is this so dan hard? 🌟
• eevans.co: Deconstructing Kubernetes Networking
• externalTrafficPolicy=local on kubernetes. How to preserve the source IP in kubernetes externalTrafficPolicy=local is an annotation on the Kubernetes service resource that can be set to preserve the client source IP. When it is set, the actual IP address of a client is propagated to the K8s service instead of the IP address of the node.
• ronaknathani.com: How a Kubernetes Pod Gets an IP Address 🌟
• opensource.com: Why I use Ingress Controllers to expose Kubernetes services Kubernetes ingress controllers will make or break your cloud architecture.
• blog.nody.cc: Verify your Kubernetes Cluster Network Policies: From Faith to Proof
• medium: How to setup Hetzner load balancer on a Kubernetes cluster
• zhimin-wen.medium.com: Sticky Sessions in Kubernetes 🌟
• infoq.com: Kubernetes Ingress Is Now Generally Available
• Learnk8s: Comparison of Kubernetes Ingress Controllers 🌟🌟 How do you choose the right Kubernetes Ingress controller when: Not all Ingress controllers support UDP, Only Kong has a free LDAP integration, Nginx Ingress and HAProxy are the only two ingress without CRDs.
• blog.alexellis.io: Get kubectl access to your private cluster from anywhere
• jmrobles.medium.com: How to setup Hetzner load balancer on a Kubernetes cluster
• kubernetes.io: Scaling Kubernetes Networking With EndpointSlices EndpointSlices are a new Kubernetes API that provides a scalable and extensible alternative to the Endpoints API.
• medium: Create a Custom Annotation for the Kubernetes ingress-nginx Controller
• haproxy.com: Announcing HAProxy Kubernetes Ingress Controller 1.5 🌟
• devclass.com: HAProxy Ingress Controller 1.5 introduces mTLS support, gives load balancing experts more power
• thenewstack.io: HAProxy Kubernetes Ingress Controller Moves Outside the Cluster
• suse.com: NGINX Guest Blog: NGINX Kubernetes Ingress Controller 🌟
• blog.cloudflare.com: Moving k8s communication to gRPC
• K8GB - Kubernetes Global Balancer - openshift.com: K8GB - Kubernetes Global Balancer
• altoros.com: Kubernetes Networking: How to Write Your Own CNI Plug-in with Bash
• Network Node Manager network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes. By simply deploying and configuring network-node-manager, you can solve kubernetes network issues that cannot be resolved by kubernetes or resolved by the higher kubernetes Version. Below is a list of kubernetes’s issues to be resolved by network-node-manager. network-node-manager is based on kubebuilder v2.3.1.
• getenroute.io: Drive API Security At Kubernetes Ingress Using Helm And Envoy 🌟
• ithands-on.com: Kubernetes 101 : External services - ExternalName, DNS and Endpoints
• ibm.com: Multizone Kubernetes and VPC Load Balancer Setup Securely expose your Kubernetes app by setting up a Load Balancer for VPC in a different zone.
• opensource.googleblog.com: Kubernetes: Efficient Multi-Zone Networking with Topology Aware Routing
• nbailey.ca: Domesticated Kubernetes Networking
• sookocheff.com: A Guide to the Kubernetes Networking Model 🌟
• build.thebeat.co: A curious case of AWS NLB timeouts in Kubernetes A debugging adventure that allowed us to solve the tail latencies our Kubernetes applications were experiencing when talking with our AWS NLB.
• dzone: Multizone Kubernetes and VPC Load Balancer Setup Securely expose your Kubernetes app by setting up a Load Balancer for VPC in a different zone.
• ingressbuilder.jetstack.io 🌟🌟 Ingress Builder allows users to select any annotation from the list of available controllers, to add to the ingress manifest.
• itnext.io: Generating Kubernetes Network Policies Automatically By Sniffing Network Traffic 🌟 This blog post is about an experiment to automate creation of Kubernetes Network Policies based on actual network traffic captured from applications running on a Kubernetes cluster - code
• medium: Using nginx-ingress controller to restrict access by IP (ip whitelisting) for a service deployed to a Kubernetes (AKS) cluster
• openshift.com: gRPC or HTTP/2 Ingress Connectivity in OpenShift 🌟
• inlets.dev: Fixing Ingress for short-lived local Kubernetes clusters
• nginx.com: How to Simplify Kubernetes Ingress and Egress Traffic Management
• blog.teamhephy.info: Running Workflow Without Any LoadBalancer
• blog.alexellis.io: Get a public LoadBalancer for your private Kubernetes cluster 🌟
• searchitoperations.techtarget.com: Differences between Kubernetes Ingress vs. load balancer To manage Kubernetes cluster traffic, admins have a few choices. Compare Kubernetes Ingress vs. load balancers, as well as the NodePort and ClusterIP service types.
• monzo.com: Controlling outbound traffic from Kubernetes
• medium: Access Application Externally In Kubernetes Cluster using Load Balancer Service Learn how to create a Pod and how to create a Load Balancer service using Kubernetes cluster. And access the application from outside.
• itnext.io: Why and How of Kubernetes Ingress (and Networking) 🌟
• techdozo.dev: gRPC load balancing on Kubernetes (using Headless Service)
• thenewstack.io: ZeroLB, a New Decentralized Pattern for Load Balancing
• ungleich.ch: Making kubernetes kube-dns publicly reachable
• ungleich.ch: Building Ingress-less Kubernetes Clusters Building Ingress-less Kubernetes Clusters with IPv6
• thenewstack.io: Ingress Controllers: The More the Merrier
• levelup.gitconnected.com: Setting up Application Load Balancer (Ingress) for the Pods running in AWS EKS Fargate
• devopscube.com: Kubernetes Ingress Tutorial For Beginners 🌟 In this Kubernetes ingress tutorial, you will learn the basic concepts of ingress, the native ingress resource object, and the concepts involved in ingress controllers
• ystatit.medium.com: How to Change Kubernetes Kube-apiserver IP Address
• monzo.com: Controlling outbound traffic from Kubernetes
• tech2fun.net: Using Service Endpoints and Alias for accessing External Service in K8s
• nginx.com: Reducing Kubernetes Costs by 70% in the Cloud with NGINX, Opsani, and Prometheus
• ithands-on.com: Kubernetes 101 : Changing a service type If we realize that our service, a ClusterIP doesn’t suit our needs anymore, we could change its type to a nodePort service for example.
• cloud.redhat.com: Global Load Balancer Approaches 🌟
• loft.sh: Kubernetes NGINX Ingress: 10 Useful Configuration Options 🌟 Kubernetes Ingress is the object that provides routing rules into your cluster. To best serve traffic to your app, you need to correctly configure it. This is an incredible article from loft.sh with 10 useful options for configuring NginX Ingress
• technos.medium.com: Kubernetes Services for Absolute Beginners — NodePort 🌟

• fransemalila.medium.com: Kubernetes Networking To access the application over the network, K8s services must be used to expose the pods to external traffic and load balancing the traffic across multiple pods.

  • Cluster IP
  • Target Ports
  • Node Port
  • External IPs
  • Load Balancer

• netris.ai: A Cloud-Like On-Prem Load Balancer for Kubernetes? (a practical guide)

• thenewstack.io: Ingress Controllers: The Swiss Army Knife of Kubernetes
• nginx.com: Kubernetes Networking 101
• medium.com/the-programmer: Working With ClusterIP Service Type In Kubernetes Working with services in Kubernetes Using ClusterIP
• olamiko.medium.com: Technical Series: Kubernetes Networking
• learnk8s.io: Tracing the path of network traffic in Kubernetes 🌟
• devopslearners.com: Kubernetes Ingress Tutorial For Beginners - https://devopscube.com/kubernetes-ingress-tutorial
• devopscube.com: How To Configure Ingress TLS/SSL Certificates in Kubernetes
• armosec.io: Getting Started with Kubernetes Ingress | Ben Hirschberg
• itnext.io: Kubernetes Service Type LB for On Prem Deployments
• medium.com/techbeatly: Kubernetes Networking Fundamentals

NetworkPolicy

• opensource.com: What you need to know about Kubernetes NetworkPolicy Understanding Kubernetes NetworkPolicy is one of the fundamental requirements to learn before deploying an application to Kubernetes.

Nginx Ingress Controller

• tech2fun.net: K8s Nginx Ingress Handling TLS Traffic and Using Pod Readiness Probes
• blog.teamhephy.info: Learn how to use the Nginx Ingress controller to serve traffic over SSH with TCP load balancing
• nginx.com: A Guide to Choosing an Ingress Controller, Part 4: NGINX Ingress Controller Options
• NGINX Ingress Controller - v1.0.0 NGINX Ingress Controller v1.0.0 released today! The biggest change is the support to stable/v1 ingress object, and dropping support to v1beta1.
• amy-ma.medium.com: Nginx Ingress Configuration Configure NGINX basic routing with TLS on HPCC. This tutorial provides steps on how to set up basic routing for ECLWatch with the NGINX Ingress controller and configure certificates using Cert-Manager.
• devopscube.com: How to Setup Nginx Ingress Controller On Kubernetes – Detailed Guide 🌟

Contour Ingress Controller

• trstringer.com: Kubernetes Ingress with Contour

Gateway API

• gateway-api.sigs.k8s.io 🌟 Gateway API is an open source project managed by the SIG-NETWORK community. It’s is a collection of resources that model service networking in Kubernetes. These resources - GatewayClass,Gateway, HTTPRoute, TCPRoute, Service, etc - aim to evolve Kubernetes service networking through expressive, extensible, and role-oriented interfaces that are implemented by many vendors and have broad industry support.
• kubernetes.io: Evolving Kubernetes networking with the Gateway API
• thenewstack.io: Unifying Kubernetes Service Networking (Again) with the Gateway API 🌟 The Gateway API, formerly known as the Services API and before that Ingress V2, was first discussed in detail — and in-person — at Kubecon 2019 in San Diego. There were already many well-known and well-documented limitations of Ingress and Kubernetes networking APIs. The Gateway API was intended as a redo of these APIs, built on the lessons from Services, Ingress and the service mesh community.

Kube-proxy

• dustinspecker.com: iptables: How Kubernetes Services Direct Traffic to Pods In this article you will learn how Kubernetes’s kube-proxy uses iptables to direct traffic to pods randomly. You’ll focus on the ClusterIP type of Kubernetes services.
• arthurchiao.art: Cracking kubernetes node proxy (aka kube-proxy) This post analyzes the Kubernetes node proxy model, and provides 5 demo implementations (within couples of lines of code) of the model, each based on different tech-stacks (userspace/iptables/ipvs/tc-ebpf/sock-ebpf).

Multicloud communication for Kubernetes

• developers.redhat.com: Use Skupper to connect multiple Kubernetes clusters 🌟 - skupper.io Multicloud communication for Kubernetes. Skupper is a layer 7 service interconnect. It enables secure communication across Kubernetes clusters with no VPNs or special firewall rules. With Skupper, your application can span multiple cloud providers, data centers, and regions.

Multi-Cluster Kubernetes Networking

• itnext.io: Multi-Cluster Kubernetes Networking with Netmaker
  • NetMaker Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.

Kubernetes Network Policy

• howtoforge.com: Network Policy in Kubernetes 🌟 By default, pods accept traffic from any source. A network policy helps to specify how a group of pods can communicate with each other and other network endpoints.
• medium: How to Provision Network Policies in Kubernetes | AWS 🌟
• learncloudnative.com: Kubernetes Network Policy
• bionconsulting.com: Kubernetes Network Policies
  • bionconsulting.com: Kubernetes Network Policies - Part 2
• thenewstack.io: The Kubernetes Network Security Effect 🌟 Kubernetes has a built-in object for managing network security: NetworkPolicy. While it allows the user to define the relationship between pods with ingress and egress policies, it is basic and requires very precise IP mapping of a solution — which changes constantly, so most users I’ve talked to are not using it.
• faun.pub: Control traffic flow to and from Kubernetes pods with Network Policies
• openshift.com: Network Policies: Controlling Cross-Project Communication on OpenShift
• loft-sh.medium.com: Kubernetes Network Policies: A Practitioner’s Guide 🌟
• loft.sh: Kubernetes Network Policies: A Practitioner’s Guide 🌟
• medium: Kubernetes Network Policies: Are They Really Useful? 🌟
• loft.sh: Kubernetes Network Policies for Isolating Namespaces 🌟
• arthurchiao.art: Cracking Kubernetes Network Policy This post digs into the Kubernetes NetworkPolicy model, then designs a policy enforcer based on the technical requirements and further implements it with less than 100 lines of eBPF code. Hope that after reading through this post, readers will get a deeper understanding on how network policies are enforced in the underlying.
• engineering.mercari.com: Managing Network Policies for namespaces isolation on a multi-tenant Kubernetes cluster This post outlines how to implement an abstraction over network policies in a multi-tenant Kubernetes cluster instead of directly exposing raw YAML-based manifests for better usability and verifiability

Cilium

• cilium.io 🌟 eBPF-based Networking, Observability, and Security
• cilium.io: NetworkPolicy Editor: Create, Visualize, and Share Kubernetes NetworkPolicies 🌟
• editor.cilium.io 🌟 Learn how to create Network Policies for Kubernetes using an interactive playground
• buoyant.io: Kubernetes network policies with Cilium and Linkerd
• itnext.io: Installing Cilium on Kubernetes in a fast and efficient way
• cilium.io: CNI Benchmark: Understanding Cilium Network Performance
• cockroachlabs.com: How to use Cluster Mesh for Multi-Region Kubernetes Pod Communication
  • Thanks to services provided by AWS, GCP, and Azure it’s become relatively easy to develop applications that span multiple regions. This is great because slow apps kill businesses. There is one common problem with these applications: they are not supported by multi-region database architecture.
  • CockroachDB is built to solve that problem and we’re doing it in production for many applications today. But that’s not what this blog is about. In this blog, I will provide a solution for the problem of getting Kubernetes pods to talk to each other in multi-region deployments.
• cilium.io: Cilium 1.10: WireGuard, BGP Support, Egress IP Gateway, New Cilium CLI, XDP Load Balancer, Alibaba Cloud Integration and more Traditional workloads have a fixed and unique IP that can be recognized by a firewall. Traffic coming from a containerized application will come from many different IPs. How can you fix that? Cilium allows users to specify an egress NAT policy

Kubernetes Network Policy Samples

• ahmetb/kubernetes-network-policy-recipes 🌟 Example recipes for Kubernetes Network Policies that you can just copy paste. This repository contains various use cases of Kubernetes Network Policies and sample YAML files to leverage in your setup. If you ever wondered how to drop/restrict traffic to applications running on Kubernetes, this is for you

Kubernetes Ingress Specification

• Supporting the Evolving Ingress Specification in Kubernetes 1.18
• medium: Ingress service types in Kubernetes 🌟

Xposer Kubernetes Controller To Manage Ingresses

• Xposer 🌟 A Kubernetes controller to manage (create/update/delete) Kubernetes Ingresses based on the Service
  • Problem: We would like to watch for services running in our cluster; and create Ingresses and generate TLS certificates automatically (optional)
  • Solution: Xposer can watch for all the services running in our cluster; Creates, Updates, Deletes Ingresses and uses certmanager to generate TLS certificates automatically based on some annotations.

Software-Defined IP Address Management (IPAM)

• IP Address Management (IPAM)
• fusionlayer.com: Software-Defined IP Address Management (IPAM)
  • Cloud computing and service automation are changing the way in which applications and data are being delivered and consumed. The existing 30-year-old networking model is failing to keep up with the automated service architectures and the Internet of Things (IoT) based on end-to-end automation.
  • To facilitate the migration to cloud-era computing, service providers and data centers must add networking into the automated service workflows. This requires agility and elasticity that traditional networking products are not designed to provide. As IT environments of tomorrow involve a plethora of orchestrators and controllers spinning up services and applications inside shared networks, they all must be managed and provisioned by a unified solution authoritative for all network-related information.

CNI Container Networking Interface

• Kubernetes.io: Network Plugins
• rancher.com: Container Network Interface (CNI) Providers
• github.com/containernetworking 🌟
  • CNI
• dzone: How to Understand and Set Up Kubernetes Networking 🌟 Take a look at this tutorial that goes through and explains the inner workings of Kubernetes networking, including working with multiple networks.
• medium: Container Networking Interface aka CNI
• itnext.io: Benchmark results of Kubernetes network plugins (CNI) over 10Gbit/s network (Updated: August 2020)

List of existing CNI Plugins (IPAM)

• Kubernetes Networking
• Overlay Network plugins:
  • Flannel
  • Weave-net
• Routed Network Plugins:
  • AWS-VPC
  • kube-router
  • Calico
  • Canal
  • VMware-tanzu Antrea
• IPAM modules:
  • dhcp
  • host-local
• Multi CNI plugins:
  • Damn
  • Multus
  • CNI-Genie

Project Calico

• tigera.io
• Project Calico 🌟 Secure networking for the cloud native era
• medium: Calico for Kubernetes networking: the basics & examples
• thenewstack.io: Tigera’s Calico Aims to Ease Connectivity Pain with Kubernetes
• projectcalico.org: Advertising Kubernetes Service IPs with Calico and BGP
• mhmxs.blogspot.com: Autoscaling Calico Route Reflector topology in Kubernetes
• tigera.io: Enforcing Network Security Policies with GitOps – Part 1 (Calico + ArgoCD) Network policy is a key element of Kubernetes security. Network policy is expressed as a YAML configuration and works very well with GitOps. By adopting GitOps, security teams benefit in the following ways:
  • Take your policies with you. Kubernetes cluster creation from code is fairly common. It is much easier and less error-prone to push your Git-based policies to a new cluster.
  • You can monitor policy changes using information from pull requests. This will also be easy to integrate with your existing systems, instead of writing integrations from scratch. If something goes wrong, you can simply roll back to an earlier commit.
  • You can lock down who can deploy security policies. If you lock it down to only a single Git user, that will be easy to control. Everybody else can push their policy changes into Git via pull request.
  • Your GitOps tool can ensure that it will override any accidental or malicious change at runtime. This solves a major compliance concern. Git becomes the source of truth for your security policies.
  • It would be much easier to manage if no user could create a security policy from kubectl. Then you can enable de-centralized security by creating specific users for different services, and giving them rights to deploy only specific policies. Developers and DevOps teams are very comfortable with the notion of a Git pipeline.

DNS Service with CoreDNS

• medium: How to Autoscale the DNS Service in a Kubernetes Cluster
• thenewstack.io: Supercharge CoreDNS with Cluster Addons 🌟
• sysdig.com: How to monitor coreDNS 🌟 The most common problems and outages in a Kubernetes cluster come from coreDNS, so learning how to monitor coreDNS is crucial.
• ungleich.ch: Making kubernetes kube-dns/CoreDNS publicly reachable

Kubernetes Node Local DNS Cache

• NodeLocal DNSCache
• Kubernetes Node Local DNS Cache

Kubernetes Sidecars

• banzaicloud.com: Sidecar container lifecycle changes in Kubernetes 1.18 🌟
• medium: Delaying application start until sidecar is ready Taking advantage of a peculiar Kubernetes implementation detail to block containers from starting before another container starts.

Videos

Click to expand!

Tweets

Click to expand!

Kubernetes is an example of what happens when you have an indefinitely complex network stack and no troubleshooting tools in place.

— Jaana Dogan ヤナ ドガン (@rakyll) November 10, 2021

Let's see how many folks here haven't seen this thread on Kubernetes Networking.

Once again, the thread doesn't try to explain the subject matter in great detail but offers a particular learning order instead.

As usual, based on my personal experience 🔽 pic.twitter.com/pxCWJUxj5j

— Ivan Velichko (@iximiuz) November 28, 2021

🧵 How does Pod to Pod communication work in Kubernetes?

How does the traffic reach the right Pod?

Let's see 👇 pic.twitter.com/gF2eVWYL4Q

— Daniele Polencic (@danielepolencic) January 31, 2022

When your apps receive a ton of traffic, how do you scale your Ingress Controller in Kubernetes?

Here is what I do 👇 pic.twitter.com/T6aYurE7Lj

— Daniele Polencic (@danielepolencic) March 2, 2022

Should you use a single Kubernetes Ingress controller or multiple?

On Monday 8PT/5CET Andrea will make a convincing case on why multiple controllers are good for

✅ security
✅ segregating team & resources
✅ isolation

Register here (it's free) https://t.co/62oKodt7tQ pic.twitter.com/DWNy0iTYq6

— Learnk8s (@learnk8s) March 13, 2022