Recently, I posted the “Shared Storage Options in Azure: Part 1 – Azure Shared Disks” blog post, the first in the 5-part series. Today I’m posting Part 2 – IaaS Storage Server. While this post will be fairly rudimentary insofar as Azure technical complexity, this is most certainly an option when considering shared storage options in Azure and one that is still fairly common with a number of configuration options. In this scenario, we will be looking at using a dedicated Virtual Machine to provide shared storage through various methods. As I write subsequent posts in this series, I will update this post with the links to each of them.
- Part 1: Azure Shared Disks
- Part 2: IaaS Storage Server
- Part 3: Azure Storage Services
- Part 4: Azure NetApp Files (coming soon)
- Part 5: Conclusion (coming soon)
Virtual Machine Configuration Options:
While it may not seem vitally important, the VM SKU you choose can impact your ability to provide storage capabilities in areas such as Disk Type, Capacity, IOPS, or Network Throughput. You can view the list of VM SKUs available on Azure at this link. As an example, I’ve clicked into the General Purpose, Dv3/Dvs3 series and you can see there are two tables that show upper limits of the SKUs in that family.
In the limits for each VM you can see there are differences between Max Cached and Temp Storage Throughput, Max Burst Cached and Temp Storage Throughput, Max uncached Disk Throughput, and Max Burst uncached Disk Throughput. All of these represent very different I/O patterns, so make sure to look carefully at the numbers.
Below are a few links to read more on disk caching and bursting:
- Disk Caching: https://docs.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance#disk-caching
- Disk Bursting: Managed disk bursting – Azure Virtual Machines | Microsoft Docs
You’ll notice when you look at VM SKUs that there is an L-Series which is “storage optimized”. This may not always be the best fit for your workload, but it does have some amazing capabilities. The outstanding feature of the L-Series VMs are the locally mapped NVMe drives which as of the time of writing this post on the L80s_v2 SKU can offer 19.2TB of storage at 3.8 Million IOPS / 20,000 MBPS.
The benefits of these VMs are the extremely low latency, and high throughput local storage, but the caveat to that specific NVMe storage is that it is ephemeral. Data on those disks does not persist a reboot. This means it’s incredibly good at serving from a local cache, tempdb files, etc. though its not storage that you can use for things like a File Server backend (without some fancy start-up scripts, please don’t do this…). You will note that the maximum uncached throughput is 80,000 IOPS / 2,000 MBPS for the VM, which is the same as all of the other high spec VMs. As I am writing this, no Azure VM allows for more than that for uncached throughput – this includes Ultra Disks (more on that later).
For more information on the LSv2 series, you can read more here: Lsv2-series – Azure Virtual Machines | Microsoft Docs
Additional Links on Azure VM Storage Design:
- Azure Premium Storage: Design for high performance – Azure Virtual Machines | Microsoft Docs
- Virtual machine and disk performance – Linux – Azure Virtual Machines | Microsoft Docs
Networking capabilities of the Virtual Machine are also important design decisions when considering shared storage, both in total throughput and latency. You’ll notice in the VM SKU charts I posted above when talking about performance there are two sections for networking, Max NICs and Expected network bandwidth Mbps. It’s important to know that these are VM SKU limitations, which may influence your design.
Expected network bandwidth is pretty straight forward, but I want to clarify that the number of Network Interfaces you mount to a VM does not change this number. For example, if your expected network bandwidth is 3200 Mbps and you have an SMB share running on that single NIC, adding a second NIC and using SMB multi-channel WILL NOT increase the total bandwidth for the VM. In that case you could expect each NIC to potentially run at 1,600 Mbps.
The last networking feature to take into consideration is Accelerated Networking. This feature allows for SR-IOV (Single Root I/O Virtualization), which by bypassing the host CPU and offloading the network traffic directly to the Network Interface can dramatically increase performance by reducing latency, jitter, and CPU utilization.
Accelerated Networking is not available on every VM though, which makes it an important design decision. It’s available on most General Purpose VMs now, but make sure to check the list of supported instance types. If you’re running a Linux VM, you’ll also need to make sure it’s a supported distribution for Accelerated Networking.
In an obvious step, the next design decision is the storage that you attach to your VM. There are two major decision types when selecting disks for you VM – disk type, and disk size.
Image Reference: https://docs.microsoft.com/en-us/azure/virtual-machines/disks-types
As the table above shows, there are three types of Managed Disks (https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview ) in Azure. At the time of writing this, Premium/Standard SSD and Standard HDD all have a limit of 32TB per disk. The performance characteristics are very different, but I also want to point out the difference in the pricing model because I see folks make this mistake very often.
|Disk Type:||Capacity Cost:||Transaction Cost:|
|Ultra SSD||Highest (Capacity/Throughput)||None|
Transaction costs can be important on a machine whose sole purpose is to function as a storage server. Make sure you look into this before a passing glance shows the price of a Standard SSD lower than a Premium SSD. For example, here is the Azure Calculator output of a 1 TB disk across all four types that averages 10 IOPS * ((10*60*60*24*30)/10,000) = 2,592 transaction units.
Sample Standard Disk Pricing:
Sample Standard SSD Pricing:
Sample Premium SSD Pricing:
Sample Ultra Disk Pricing:
The above example is just an example, but you get the idea. Pricing gets strange around Ultra Disk due to the ability to configure performance (more on that later). Though there is a calculable break-even point for disks that have transaction costs versus those that have a higher provisioned cost.
For example, if you run an E30 (1024 GB) Standard SSD at full throttle (500 IOPS) the monthly cost will be ~$336, compared to ~$135 for a P30 (1024 GB) Premium SSD, with which you get x10 the performance. The second design decision is disk capacity. While this seems like a no-brainer (provision the capacity needed, right?) it’s important to remember that with Managed Disks in Azure, the performance scales with, and is tied to, the capacity of the disk.
You’ll note in the above image the Disk Size scales proportionally with both the Provisioned IOPS and Provisioned Throughput. This is to say that if you need more performance out of your disk, you scale it up and add capacity.
The last note on capacity is this, if you need more than 32TB of storage on a single VM, you simply add another disk and use your mechanism for combining that storage (Storage Spaces, RAID, etc.). This same method can be used to further tweak your total IOPS, but make sure you take into consideration the cost of each disk, capacity, and performance before doing this – most often it’s an insignificant cost to simply scale-up to the next size disk. Last but not least, I want to briefly talk about Ultra Disks – these things are amazing!
Unlike with the other disk types, this configuration allows you to select your disk size and performance (IOPS AND Throughput) independently! I recently worked on a design where the customer needed 60,000 IOPS, but only needed a few TB of capacity, this is the perfect scenario for Ultra Disks. They were actually able to get more performance, for less cost compared to using Premium SSDs.
To conclude this section, I want to note two design constraints when selecting disks for your VM.
- The VM SKU is still limited to a certain number of IOPS, Throughput and Disk Count. Adding together the total performance of your disks, cannot exceed the maximum performance of the VM. If the VM SKU supports 10,000 IOPS and you add 3x 60,000 IOPS Ultra Disks, you will be charged for all three of those Ultra Disks at their provisioned performance tiers but will only be able to get 10,000 IOPS out of the VM.
- All of the hardware performance may still be subject to the performance of the access protocol or configuration, more on this in the next section.
Additional Reading on Storage:
- Disk Types: Select a disk type for Azure IaaS VMs – managed disks – Azure Virtual Machines | Microsoft Docs
Software Configuration and Access Protocols:
As we come to the last section of this post, we get to the area that aligns with the purpose of this blog series – shared storage. In this section I’m going to cover some of the most common configurations and access types for shared storage in IaaS. This is by no means an exhaustive list, rather what I find most common.
Scale-Out File Server (SoFS):
First up is Sale-Out File Server, this is a software configuration inside Windows Server that is typically used with SMB shares. SoFS was introduced in Windows 2012, uses Windows Failover Clustering, and is considered a “converged” storage deployment. It’s also worth noting that this can run on S2S (Storage Space Direct), which is the method I recommend using with modern Windows Server Operating Systems. Scale-Out File Server is designed to provide scale-out file shares that are continuously available for file-based server application storage. It provides the ability to share the same folder from multiple nodes of the same cluster. It can be deployed in two configuration options, for Application Data or General Purpose. See the additional reading below for the documentation on setup guidance.
- Storage Spaces Direct: Storage Spaces Direct overview | Microsoft Docs
- Scale-Out File Server: Scale-Out File Server for application data overview | Microsoft Docs
- Setup guide for 2-node SSD for RDS UPD: Deploy a two-node Storage Spaces Direct SOFS for UPD storage in Azure | Microsoft Docs
Now into the access protocols – SMB has been the go-to file services protocol on Windows for quite some time now. In modern Operating Systems, SMB v3.* is an absolutely phenomenal protocol. It allows for incredible performance using things like SMB Direct (RDMA), Increasing MTU, and SMB Multichannel which can use multiple NICs simultaneously for the same file transfer to increase throughput. It also has a list of security mechanisms such as Pre-Auth Integrity, AES Encryption, Request Signing, etc. There is more information on the SMB v3 protocol below, if you’re interested, or still think of SMB in the way we did 20 years ago – check it out. The Microsoft SQL Server team even supports SQL hosting databases on remote SMB v3 shares.
NFS has been a similar staple as a file server protocol for a long while also, and whether you’re running Windows or Linux can be used in your Azure IaaS VM for shared storage. For organizations that prefer an IaaS route compared to PaaS, I’ve seen many use this as a cornerstone configuration for their Azure Deployments. Additionally, a number of HPC (High Performance Compute) workloads, such as Azure CycleCloud (HPC orchestration) or the popular Genomics Workflow Management System, Cromwell on Azure prefer the use of NFS.
- Create NFS Ubuntu Linux Server volume – Azure Kubernetes Service | Microsoft Docs
- azure-quickstart-templates/nfs-ha-cluster-ubuntu at master · Azure/azure-quickstart-templates (github.com)
While I would not recommend the use of custom block storage on top of a VM in Azure if you have a choice, some applications do still have this requirement in which case iSCSI is also an option for shared storage in Azure.
That’s it! We’ve reached the end of Part 2. Okay, here we go with the Pros and Cons for using an IaaS Virtual Machine for your shared storage configuration on Azure.
Pros and Cons:
- More control, greater flexibility of protocols and configuration.
- Depending on the use case, potentially greater performance at a lower cost (becoming more and more unlikely).
- Ability to migrate workloads as-is and use existing storage configurations.
- Ability to use older, or more “traditional” protocols and configurations.
- Allows for the use of Shared Disks.
- Significantly more management overhead as compared to PaaS.
- More complex configurations, and cost calculations compared to PaaS.
- Higher potential for operational failure with the higher number of components.
- Broader attack surface, and more security responsibilities.
Alright, that’s it for Part 2 of this blog series – Shared Storage on IaaS Virtual Machines. Please reach out to me in the comments, on LinkedIn, or Twitter with any questions about this post, the series, or anything else!
- Part 1: Azure Shared Disks
- Part 2: IaaS Storage Server
- Part 3: Azure Storage Services
- Part 4: Azure NetApp Files (coming soon)
- Part 5: Conclusion (coming soon)
*Edit* 6/9/20: At the time of writing this blog custom roles were not available in the Azure Portal. Since that feature now exists, I’ve added how to create this role through the portal to the end of this blog.
We all know that cloud environments are different than on premises. Development environments can be even more difficult, the right cross between giving freedom to produce business value and enough security and controls to maintain a good security posture. Recently I came across a scenario where the design was that the networking components (Virtual Network, S2S VPN, Peerings, Service Endpoints, UDRs, Subnets, etc.) were all under the control of a Networking team so that the Azure environment would be an extension of their local network. From a permissions standpoint though this caused some problems. The goal would be to empower developers to build whatever they need and just attach it to the vNet when connectivity was needed, with the caveat that they shouldn’t be able to modify any of the network settings or configurations in the shared vNet. This however, turned out not to be possible with default IAM roles.
In my testing, even though you’re not “modifying” anything per se in the vNet – when attaching a network interface card to a Subnet it does require write access. Reader access will give the user this error.
After looking through the default IAM roles, there aren’t any that do what I need them to do. Alas, a custom role is needed. For reference, please checkout this docs page for custom roles – https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles.
First, I need to see what actions are available to pack into the custom role so I run the following command in Azure Powershell and get these results.
Get-AzProviderOperation “Microsoft.Network/virtualNetworks/*” | FT OperationName, Operation, Description -AutoSize
Great, now I can see what is available. It looks like the actions that I need are the /subnet/join/action and /subnet/joinViaServiceEndpoint/Action. Based on the descriptions these two will essentially give “operator” role to the vnet. The assignee will be able to use the subnet(s) but not able to modify them.
Next, you can use one of the template references in the Microsoft Docs link (https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles-powershell#create-a-custom-role-with-json-template) and modify the actions.
“Name”: “Custom – Network Operator Role”,
“Description”: “Allows for read access to Azure network and join actions for service endpoints and subnets.”,
After modifying the assignable subscription IDs, save that as a .json file and use it in the following command to import the custom role.
New-AzRoleDefinition -InputFile “C:\FileFolderLocationPath\CustomNetworkOperatorRole.json”
After a portal refresh you get the custom role available as an IAM role assignment.
There you go, after assigning this role the user was able to create VMs and attach them to the Virtual Network while still leaving control of the Network configuration to the Networking Team.
EDIT 6/9/20: Now that custom roles are available in the portal I’ve added how to create this role that way below.
Using the documentation on how to create custom IAM roles in the Azure portal you can clone any role, but I’m just going to use the “Network Contributor” role to clone.
I am going to just “start from scratch”, though you can easily use this page to modify the permissions set of a certain role that already exists. Additionally, if you have the JSON (like we generated before with powershell) you can just upload that here.
Now go ahead and add new permissions by going to the “Microsoft Network” Category and selecting the same permissions we wrote in the JSON and used with the powershell command above.
After adding those the next page will show you the permission that you’ve added, let’s double check those and make sure we have what we need.
Looks good, now we can set a scope where this role can be applied. I have this one scoped to the whole subscription, but you can use the interface to easily change that on this page.
Alright, that’s about it – let’s take a quick look at the summary pages. You can see that the portal actually generates the JSON, which is nice if you want to assign roles using Azure Policy or any other automation tooling.
Nice! Now we can see that users can use the four above actions which will allow them to interact with, and use a shared hub-style virtual network while now allowing full contributor access.
I hope I’ve made your day at least a little bit easier!
If you have any questions or suggestions for future blog posts feel free to comment below, or reach out to me via email, twitter, or LinkedIn.
Reading Time: 2 minutesBack in the classic portal with backup services it was an easy fix. Simply change the settings value of storage replication type. I’ve recently started moving my workloads to recovery serveries vaults in ARM, and noticed something peculiar. By default, the storage replication type of the vault is GRS.
If your needs require geographically redundant storage, that that’s perfectly fine. I however don’t have such needs, and trust in Microsoft’s ability to keep data generally available in a LRS replication topology. It should be just like it was in classic, as an option anyways, right? Strangely, the option to change the replication type for the storage configuration on the vault is grayed out.
Odd, right? I thought so, until I found this.
Okay, well it’s not optimal but it looks like I need to remove the backup data from the vault to change the storage replication types right? Well, I gave that a shot and no go. I had the same issue, the option was still grayed out.
I ultimately had to completely delete, and create a new recovery services vault. Once it’s initially created you can change the replication type.
Ah, finally! Then register the VM(s), run some backup jobs and voila! Confirmation that the vault is using LRS storage.
I hope this makes your day at least a little bit easier.
Reading Time: 3 minutesEDIT: If you liked this post, I’ve updated my process a little bit and written a script to automate a good chunk of this! Go check out Part 2 of this blog! http://thetechl33t.com/2014/10/03/how-to-mount-your-onedrive-as-a-local-mapped-drive-part-2/
OneDrive is an online storage system by Microsoft that is included when you have an email account such as @live.com @hotmail.com etc. I use it fairly often and I was curious if I could map it locally, turns out that I can.
First of all, you need to go to https://onedrive.com and use your Windows Live account (the same you use to access Hotmail, Messenger, Windows Live Mail or MSN) to log in and create the folders you want to use by using the New menu. You can create private and shared folders and customize the access for every one of them.
After your have created your folders and customized it to your liking, you will need to link your computer to your online ID so it can access them without asking for credentials every time.
Click on the Start Menu button and select Control Panel.
Select User Accounts and Family Safety.
Select User Accounts
Select Link Online IDs, on the left side of the window.
Click on Link Online ID.
If you haven´t installed the Windows Live ID provider, you will be taken to a website to download it. If not, click the “Add an online ID provider” link in the above photo and it will take you there.
Now you will be taken back to the Online ID providers and click on Link Online ID to sign in.
Now, to get the address where to map your OneDrive´s folders, you can open Excel, Word, PowerPoint or OneNote click on File and then on Save & Send. Then click “Save to Web” and it will populate the OneDrive folders from the OnlineID you just linked, select that folder and click “Save As”.
Double click on the folder you want to map and copy the folder´s address.
Now that you have that link, go back to “Computer” and click “Map Network Drive”.
Choose a drive letter, and paste that URL in there that was copied a few steps back.
There ya go! You’ve now got your OneDrive linked locally!
EDIT: If you liked this post, I’ve updated my process a little bit and written a script to automate a good chunk of this! Go check out Part 2 of this blog! http://thetechl33t.com/2014/10/03/how-to-mount-your-onedrive-as-a-local-mapped-drive-part-2/