Azure Arc enabled Kubernetes for Hybrid Cloud Management — Manage Everything and Anywhere
Azure Arc enabled Kubernetes allows you to connect Kubernetes clusters to Azure for extending Azure's management capabilities like Azure Policy, Azure Monitor, and Azure Resource Graph.
With Azure Arc, customers can attach, connect and configure Kubernetes clusters inside or outside of Azure and deploy modern applications at scale. Key features include inventory and organization, governance and configuration, integrated DevOps and management capabilities, and a unified tool experience.
In this blog, we will follow a step by step approach and learn how to:
1. Connect Kubernetes clusters running inside or outside of Azure
2. GitOps - to define applications and cluster configuration in source control
3. Azure Policy for Kubernetes
4. Azure Monitor for containers
1. Connect Kubernetes clusters
Install az-cli extensions
Install the latest connectedk8s and k8sconfiguration CLI extensions using below commands:
* az extension add --name connectedk8s
* az extension add --name k8sconfiguration
Create a Resource Group
Create a Resource Group using below command in Azure portal choose East US or West Europe location.
* az group create --name AzureArcRes -l EastUS -o table
Note: At the time of writing this blog Microsoft provides Azure Arc-enabled Kubernetes in the East US and West Europe Azure regions.
Connect to the cluster with admin access and attach it with Azure Arc
We use az connectedk8s connect cli extension to attach our Kubernetes clusters to Azure Arc.
First, it will verify the connectivity to our Kubernetes clusters via kube-config (“~/.kube/config”) and deploy Azure Arc Agents for Kubernetes using Helm 3 into the “azure-arc” namespace.
For this demonstration we connect to and attach to Kubernetes clusters in AKS (Azure), EKS (AWS) and GKE (Google Cloud). Below, we step through the commands used to connect and attach to each cluster.
AWS – EKS
* aws eks --region <Region> update-kubeconfig --name <ClusterName>
* kubectl get nodes
* az connectedk8s connect --name <ClusterName> --resource-group AzureArcRes
Azure – AKS
* az aks get-credentials -g <ResourceGroupName> -n <ClusterName>
* kubectl get no
* az connectedk8s connect --name <ClusterName> --resource-group AzureArcRes
GCLOUD- GKE
* gcloud container clusters get-credentials <ClusterName> --zone <ZONE> --project <ProjectID>
* kubectl get no
* az connectedk8s connect --name <ClusterName> --resource-group AzureArcRes
Verify Connected Clusters
* az connectedk8s list -g AzureArcRes -o table
2. Using GitOps to define applications & clusters
We use the connected GKE cluster for our example to deploy a simple application.
Create a configuration to deploy an application to kubernetes cluster.
We use “k8sconfiguration” extension to link our connected cluster to an example git repository provided by SNP.
* export KUBECONFIG=~/.kube/gke-config
* az k8sconfiguration create \
--name app-config \
--cluster-name <ClusterName> --resource-group AzureArcRes \
--operator-instance-name app-config --operator-namespace cluster-config \
--repository-url https://github.com/gousiya573-snp/SourceCode/tree/master/Application \
--scope cluster --cluster-type connectedClusters
Check to see that the namespaces, deployments, and resources have been created:
* kubectl get ns --show-labels
We can see that cluster-config namespace have been created.
* kubectl get po,svc
The flux operator has been deployed to cluster-config namespace, as directed by our sourceControlConfig and application deployed successfully, we can see the pods are Running and Service LoadBalancer IP also created.
Access the EXTERNAL-IP to see the output page:
Please Note:
Supported repository-url Parameters for Public & Private repos:
* Public GitHub Repo - http://github.com/username/repo (or) git://github.com/username/repo
* Private GitHub Repo - https://github.com/username/repo (or) git@github.com:username/repo
* For the Private Repos – flux generates a SSH key and logs the public key as shown below:
Go to GitHub repo settings choose the Deploy keys option to add the above key for CI/CD.
3. Azure Policy for Kubernetes
Use Azure Policy to enforce that each Microsoft.Kubernetes/connectedclusters resource or Git-Ops enabled Microsoft.ContainerService/managedClusters resource has specific Microsoft.KubernetesConfiguration/sourceControlConfigurations applied on it.
Create Policy:
To create the policy navigate to Azure portal and Policy, in the Authoring section select the Definitions.
Click on Initiative definition to create the policy and search for gitops in the Available Definitions, click on Deploy GitOps to Kubernetes clusters policy to add.
Select the subscription in the Definition locations, Give the Policy assignment Name and Description.
Choose the Kubernetes in the existing Category list and scroll-down to fill the Configuration related details of an application.
Give any name for the Configuration resource, Operator Instance, Operator Namespace and set the Operator scope to cluster level rather than namespace, Operator type is Flux and give your application github repo url (public or private) in the Repository Url field. Now, additionally pass the Operator parameters such as “--git-branch=master --git-path=manifests --git-user=your-username –git-readonly=false” finally click on Save option and see the policy with the given name is created in the Definition.
--git-readonly=false enables the CI/CD for the repo and creates the Auto releases for the commits.
Assign Policy:
Select the above created policy definition and click on Assign option above, while creating the assignment set the scope for the assignment, scope can be Azure resource group level or subscription, enable the Policy enforcement and click on Review+create. Once the assignment is created the Policy engine will identify all connectedCluster or managedCluster resources that are located within the scope and will apply the sourceControlConfiguration to each one.
Verify a policy assignment
Go to Azure portal and click on connected Cluster resources to check the Compliant Status, Compliant: config-agent was able to successfully configure the cluster and deploy flux without error.
We can see the policy assignment that we created above, and the Compliance state should be Compliant.
4. Azure Monitor for containers
It provides rich monitoring experience for the Azure Kubernetes Service (AKS) and AKS Engine clusters. This can be enabled for one or more existing deployments of Arc enabled Kubernetes clusters using either a PowerShell or Bash script.
Enable using bash script
Download the script using below command:
* curl -o enable-monitoring.sh -L https://aka.ms/enable-monitoring-bash-script
This script configures our cluster with the monitoring add-on.
To enable monitoring, there are different commands provided based on deployment scenario.
The bash command below runs the script with default options, such as using current kube-context, create a default Log Analytics workspace, and without specifying a proxy server:
The only required parameter is the Azure Arc Cluster ResourceId
*bash enable-monitoring.sh --resource-id $azureArcClusterResourceId
where,
azureArcClusterResourceId="/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.Kubernetes/connectedClusters/<clusterName>".
We can find azureArcClusterResourceId in properties of connected clusters.
After enabled monitoring, it might take about 10 to 15 minutes to get health metrics for the cluster.
We can also scrape and analyze Prometheus metrics from our cluster.
Delete Resources
To disable monitoring:
*bash disable-monitoring.sh --resource-id $azureArcClusterResourceId
To delete a configuration:
*az k8sconfiguration delete --name '<config name>' -g '<resource group name>' --cluster-name '<cluster name>' -- cluster-type connectedClusters
To delete a connected cluster:
*az connectedk8s delete --name --resource-group AzureArcRes
Conclusion:
This blog covers the overview of the Azure Arc enabled Kubernetes with an example on how SNP helps its customers setup Kubernetes clusters with Azure Arc. To help you accelerate your Kubernetes journey with our subscription service that supports installing production-grade Kubernetes on-premise, in Microsoft Azure. Contact SNP specialists here
