CloudLabs Architecture & Infrastructure Deployments Automation: Technical Deep Dive

CloudLabs is a hands-on lab platform used to plan, provision, and manage the lifecycle of lab environments for various scenarios including technical training, workshops, industry conferences, POCs, demos, etc.  CloudLabs is used by technology companies and instructors around the globe at a scale where we provision thousand of lab environments every month.

In this article, we’ll take a dive into the underlying technologies we use to manage the labs at that scale using the power of Azure. 

This article is published as a part of the Applied Cloud Stories initiative by Microsoft. Here the link to more details about this initiative. 

CloudLabs Architecture and Underlying Technologies

The following diagram depicts the architecture and flow of information in CloudLabs at a high-level.

CloudLabs is deployed using Azure PaaS Services throughout the solution. This allows us to scale for thousands of deployments without worrying about underlying infrastructure management.

Authentication

We use Azure AD B2C for authentication. Using Azure AD B2C enables us to offer various authentication mechanisms including Azure AD, LinkedIn, Microsoft Account (Live ID), and other identities.

Since our focus is B2B, using Azure AD B2C allows us to allow companies using AzureAD as their identity provider to use single sign-on for accessing the CloudLabs administration portal. At this time, We’ve Azure AD and LinkedIn authentication enabled.

Application Hosting

We leverage various Azure PaaS technologies to host various components of CloudLabs Architecture. Architecture diagram included the solution components used at a high level, Now let’s take a look at all the azure services we’re using. This includes:

• App Service for Web Apps
• Azure API Management for Hosting API service
• Azure SQL Database Elastic Pool for backend database
• Azure Functions for on-demand applications and custom extensions
• Azure Storage Account for all static content
• Azure Key Vault for storing all the secrets and certificates
• Application Insights for telemetry analysis and overall application performance and debugging
• Azure Cognitive Search for all searching capabilities
• Traffic Manager for Global site load balancing
• Application Gateway WAF for web application security.
• Azure DNS for hosting the public DNS
• Azure Service Bus for communication between applications and messaging.
• Azure Monitor for monitoring and alerts
• Azure SignalR for real-time communication
• Azure Synapse Analytics for data analysis and warehousing.
• Azure Data Factory for data ingestion pipeline for analytics.
• Azure Container Instances for running some of our custom extensions on demand.  

CloudLabs Development and Deployment Process.

We use Azure DevOps as our DevOps platform, with fully automated testing and shipping pipelines.  Source code is stored in multiple Azure Repos.

Deployments are performed in dev and QA environments first, after through validations and testing they’re pushed to production. A single pipeline is built to manage the build and release to all the environments in a staged fashion.

The database schema upgrade is performed through scripts and deployed via the same Azure Pipelines. We follow blue-green deployments methodologies at this time. All Production deployments are manually approved whereas test/QA deployments are automated on pull request completion in the master branch.  

Security

We rely on following security capabilities from Azure helps us to keep access and data security for the CloudLabs platform.

• Access Security: We do not store any user credentials in the platform, Authentication is handled through Azure AD B2C, which allows us to protect our admin accounts through Azure AD Premium security features such as Conditional Access, Multi-factor authentication, etc.  
• Role-Based Access Control: Platform access is designed with RBAC from the ground up, enabling us to provide access on a need to know basis only.
• Application Security: All webapps and API apps are hosted on Azure App Service with HTTPS only and TLS Version 1.2 enforced. We
• Database Security:  We leverage Azure SQL Database Transparent data encryption, dynamic data masking and several other security capabilities to protect our databases. Admin access to the database is enabled following JIT access principal through AAD, which secures access with MFA protection. For various reporting and data views, we’re using SQL Row-level security to protect data access.
• Azure Application Gateway WAF V2: We leverage Azure Application gateway WAF v2 in detect mode to prevent against any attack from the internet.
• Azure Security Center and Azure Monitor: We leverage Azure Security center to monitor our overall environmental health from a security standpoint, with alerts being configured through  Azure Monitor.
• Storage accounts hosting static data are protected with Server-side encryption and regular rotation of access keys.
• Azure Key Vault is used as a secure credentials store with no access information being stored in configuration files.

Business Continuity and Disaster Recovery.

CloudLabs systems are designed with high availability as a fundamental design requirement for every component. We use multiple app services, API management instances, SQL Elastic Pools, and similar technologies at each level which ensures CloudLabs is fault-tolerant with ability to scale up or scale out in case of load spikes.

All CloudLabs components have redundant deployments in 2 additional Azure regions with SQL Databases being replicated to 3 Azure regions. End user-facing endpoints are published through Traffic Manager to ensure that in case of an outage, there’s no impact on end-users accessing the applications.

Automating Infrastructure Labs Deployments.

CloudLabs integrates with Microsoft Cloud through the various SDKs and APIs specifically Azure ARM APIs and Microsoft Graph API. All deployments to Azure are initiated with API and are tracked through its lifecycle.

Everything in CloudLabs is stored in a lab definition templates, which consists of various deployment methods, configurations choice and automation options. Some of them includes

• ARM Templates
• PowerShell and CLI Script
• Python
• Selenium
• Platform extensions
• Azure API
• Databricks Extensions
• Azure Machine Learning extensions
• Synapse Analytics Extensions

Lab definition includes various ways to tighten up the security of the lab environment including capabilities such as Custom Azure Role, Azure Policy with some of the advanced security configurations enabled by the platform.

Automating VM Based Lab Deployments with Shared Image Gallery and ARM Templates

CloudLabs provides exceptional lab experiences to provide an impactful learning experiences for our audiences. One of key requirement for this is to ensure the lab environments are ready to use upon signup with no additional setup requirement. CloudLabs packages the all pre-requisite and desired experience configurations in the VMs images which are then used to deploy thousands of lab environments by many of our customers.

We use Azure Shared Image Gallery to store those images. While it’s pretty easy to set up a Shared Image Gallery and use that to deploy VMs, things can really go out of hand when we talk about maintaining hundreds of images with thousands of deployments every month.

We use a combination of Azure DevOps, PowerShell, Shared Image Gallery, ARM templates and CloudLabs’s native capabilities to automate the entire infrastructure.

Azure Pipelines (Part of Azure DevOps) are used to automate the image updates and replication process.

Flow includes

1. Administrator uploads a new VHD image to the defined storage account or creates an Image directly in Azure.
2. Azure DevOps detects the changes and identifies if an existing version of image already exists in Shared Image Gallery.
3. Based on #2, Azure Pipelines create a new version of the image in the Shared Image Gallery or create a new image definition based on the configurations provided with the image.
4. Pipeline replicates the image to multiple Azure regions based on the configurations specified.
5. Administrator use ARM templates referring the shared image gallery and updates the lab definition in CloudLabs admin portal. Templates are built in such a way that they can be used across azure subscriptions and regions. A sample of the template is shared at the end of this article.
6. Through CloudLabs, customers can deploy thousands of such instances with just a couple of clicks.

We are currently working on integrating the Azure VM Image builder in the lab deployments scenarios to make the image building and updating process more seamless and customizable. We are excited to enhance the deployments experience with everything that the VM image builder brings to the table.

Using Nested Virtualization on Azure for On-prem Lab Infrastructure Migrations

Traditionally, Lab Infrastructure has been deployed on on-premises virtualization infrastructure running Hyper-V or similar hypervisor technology. Lab deployment would including having a set of VHDs for various VMs in the lab along with deployment script built on PowerShell to spin those VMs. These scripts would generally run for hours and are required to be constantly monitored to ensure any failures are detected and fixed in time. It hadn’t been uncommon to have capacity issues in lab environments, which typically led to many attendees sharing the same lab VMs. We are solving these challenges and eliminating the possibility of a bad experience for attendees and instructors by using nested virtualization capabilities in Azure.

With CloudLabs, we help trainers to move those on-prem lab infrastructures to Azure without rebuilding the entire lab topology and scripts.

Azure VMs running DSv2 and ESv3 VM series allows us to run Hyper-V inside the VM and use the full-blown virtualization to host VMs inside the Azure VM. There are various network topologies available such as using NAT on Hyper-V host to enable internet access to VMs, using RRAS on Hyper-V to connect the Guest VMs with other resources running in the Azure Virtual network. Depending on the lab topology, we capture the required networking configuration and automate that using Custom Script Extensions.

These Hosts are then converted into Images and deployed with CloudLabs thousands of times in an automated fashion.

How Customers are using CloudLabs

Customers are using CloudLabs to deploy thousands of labs instances every month with a lifecycle of few hours to few months. Let’s look at some of the stats we’ve so far.

• 400000+ Lab Instances deployed
• 28920+ trained attendees
• 5000+ events

While CloudLabs can be used to automate any kind of cloud infrastructure deployments, We’ve seen our customers enhancing the experience for everyone for the following use-cases.

• Internal/External Training
• Partner Enablement
• Sales Enablement
• Cloud Deployment Orchestration
• Virtual Labs
• Self-Paced Learning
• Certification preparation
• Hackathons
• POCs
• Sales Demos

Resources

1. Sample ARM Template using Shared Image Gallery – Access Now
2. Spektra Systems Github Account containing many repositories for our code samples, labs sample, guides, etc. – Access Now

Thank you. We will keep updating this article as we make changes to the platform and keep innovating with Azure.

Amit Malik

Avinash Sharma