Solinea Customer Case on How We Architected and Run Kubernetes on OpenStack at Scale at Yahoo! JAPAN


Originally posted on

Editor’s note: today’s post is by the Infrastructure Engineering team at Yahoo! JAPAN, talking about how they run OpenStack on Kubernetes. This post has been translated and edited for context with permission — originally published on the Yahoo! JAPAN engineering blog

This post outlines how Yahoo! JAPAN, with help from Google and Solinea, built an automation tool chain for “one-click” code deployment to Kubernetes running on OpenStack. 

We’ll also cover the basic security, networking, storage, and performance needs to ensure production readiness. 

Finally, we will discuss the ecosystem tools used to build the CI/CD pipeline, Kubernetes as a deployment platform on VMs/bare metal, and an overview of Kubernetes architecture to help you architect and deploy your own clusters. 

Since our company started using OpenStack in 2012, our internal environment has changed quickly. Our initial goal of virtualizing hardware was achieved with OpenStack. However, due to the progress of cloud and container technology, we needed the capability to launch services on various platforms. This post will provide our example of taking applications running on OpenStack and porting them to Kubernetes.

Coding Lifecycle
The goal of this project is to create images for all required platforms from one application code, and deploy those images onto each platform. For example, when code is changed at the code registry, bare metal images, Docker containers and VM images are created by CI (continuous integration) tools, pushed into our image registry, then deployed to each infrastructure platform.


We use following products in our CICD pipeline:



Code registry

GitHub Enterprise

CI tools


Image registry


Bug tracking system


deploying Bare metal platform

OpenStack Ironic

deploying VM platform


deploying container platform


Image Creation. Each image creation workflow is shown in the next diagram.

VM Image Creation:



Bare Metal Image Creation:

push code to GitHub
hook to Jenkins master
Launch job at Jenkins slave 
checkout Packer repository
Run Service Job
Download base bare metal image by build script
build script execute diskimage-builder with Packer to create bare metal image
Upload new created image to Glance
Upload the image to Artifactory

Container Image Creation:


push code to GitHub
hook to Jenkins master
Launch job at Jenkins slave 
checkout Dockerfile repository
Run Service Job
Download base docker image from Artifactory
If no docker image found at Artifactory, download from Docker Hub
Execute docker build and create image 
Upload the image to Artifactory

Platform Architecture

Let’s focus on the container workflow to walk through how we use Kubernetes as a deployment platform. This platform architecture is as below.



Infrastructure Services


Container Host


Container Cluster Manager


Container Networking

Project Calico

Container Engine


Container Registry


Service Registry


Source Code Management

GitHub Enterprise

CI tool


Infrastructure Provisioning



Fluentd, Elasticsearch, Kibana


Heapster, Influxdb, Grafana

Service Monitoring


We use CentOS for Container Host (OpenStack instances) and install Docker, Kubernetes, Calico, etcd and so on. Of course, it is possible to run various container applications on Kubernetes. In fact, we run OpenStack as one of those applications. That’s right, OpenStack on Kubernetes on OpenStack. We currently have more than 30 OpenStack clusters, that quickly become hard to manage and operate. As such, we wanted to create a simple, base OpenStack cluster to provide the basic functionality needed for Kubernetes and make our OpenStack environment easier to manage.

Kubernetes Architecture

Let me explain Kubernetes architecture in some more detail. The architecture diagram is below.




OpenStack Keystone

Kubernetes Authentication and Authorization

OpenStack Cinder

External volume used from Pod (grouping of multiple containers)


Configure and validate objects like Pod or Services (definition of access to services in container) through REST API


Allocate Pods to each node


Execute Status management, manage replication controller


Run on each node as agent and manage Pod


Enable inter-Pod connection using BGP


Configure iptable NAT tables to configure IP and load balance (ClusterIP)


Distribute KVS to store Kubernetes and Calico information


Run on each node and transfer client request to etcd clusters

Tenant Isolation

To enable multi-tenant usage like OpenStack, we utilize OpenStack Keystone for authentication and authorization.


With a Kubernetes plugin, OpenStack Keystone can be used for Authentication. By Adding authURL of Keystone at startup Kubernetes API server, we can use OpenStack OS_USERNAME and OS_PASSWORD for Authentication.


We currently use the ABAC (Attribute-Based Access Control) mode of Kubernetes Authorization. We worked with a consulting company, Solinea, who helped create a utility to convert OpenStack Keystone user and tenant information to Kubernetes JSON policy file that maps Kubernetes ABAC user and namespace information to OpenStack tenants. We then specify that policy file when launching Kubernetes API Server. This utility also creates namespaces from tenant information. These configurations enable Kubernetes to authenticate with OpenStack Keystone and operate in authorized namespaces.

Volumes and Data Persistence

Kubernetes provides “Persistent Volumes” subsystem which works as persistent storage for Pods. “Persistent Volumes” is capable to support cloud-provider storage, it is possible to utilize OpenStack cinder-volume by using OpenStack as cloud provider.


Flannel and various networking exists as networking model for Kubernetes, we used Project Calico for this project. Yahoo! JAPAN recommends to build data center with pure L3 networking like redistribute ARP validation or IP CLOS networking, Project Calico matches this direction. When we apply overlay model like Flannel, we cannot access to Pod IP from outside of Kubernetes clusters. But Project Calico makes it possible. We also use Project Calico for Load Balancing we describe later.

In Project Calico, broadcast production IP by BGP working on BIRD containers (OSS routing software) launched on each nodes of Kubernetes. By default, it broadcast in cluster only. By setting peering routers outside of clusters, it makes it possible to access a Pod from outside of the clusters.


External Service Load Balancing

There are multiple choices of external service load balancers (access to services from outside of clusters) for Kubernetes such as NodePort, LoadBalancer and Ingress. We could not find solution which exactly matches our requirements. However, we found a solution that almost matches our requirements by broadcasting Cluster IP used for Internal Service Load Balancing (access to services from inside of clusters) with Project Calico BGP which enable External Load Balancing at Layer 4 from outside of clusters.


Service Discovery

Service Discovery is possible at Kubernetes by using SkyDNS addon. This is provided as cluster internal service, it is accessible in cluster like ClusterIP. By broadcasting ClusterIP by BGP, name resolution works from outside of clusters. By combination of Image creation workflow and Kubernetes, we built the following tool chain which makes it easy from code push to deployment.



In summary, by combining Image creation workflows and Kubernetes, Yahoo! JAPAN, with help from Google and Solinea, successfully built an automated tool chain which makes it easy to go from code push to deployment, while taking multi-tenancy, authn/authz, storage, networking, service discovery and other necessary factors for production deployment. We hope you found the discussion of ecosystem tools used to build the CI/CD pipeline, Kubernetes as a deployment platform on VMs/bare-metal, and the overview of Kubernetes architecture to help you architect and deploy your own clusters. Thank you to all of the people who helped with this project. –Norifumi Matsuya, Hirotaka Ichikawa, Masaharu Miyamoto and Yuta Kinoshita. This post has been translated and edited for context with permission — originally published on the Yahoo! JAPAN engineer blog where this was one in a series of posts focused on Kubernetes.