Microsoft WinDays 15 conference will take place again in Umag (Croatia) from 21st till 24th of April 2015. I will be presenting again this year and my session is about a Failover Clustering in Windows Server vNext. Hope to see some of you there.
Cloud Witness is a new type of Failover Cluster quorum witness that will be available in the next version of Windows Server – now available in Technical Preview. The idea is to use the Azure Storage to place the cluster witness.
Cluster uses the Azure Blob Storage for a blob file which gets a vote and participates in quorum calculations and the same storage account can be used for multiple clusters. In the case you are worried about the expenses keep in mind that the Azure storage is billed based on data amount stored and that a witness blob file is very small – insignificant in size to be more precise.
The same storage account used by two clusters:
There are two main scenarios when you should use the Cloud Witness:
- Stretched clusters
- Clusters running within Azure VMs
When having a stretched cluster it is important to ensure that nodes in a secondary location can failover roles. Imagine that you have a 4 node failover cluster with 2 nodes at each location. Each node has one vote, but we need an extra vote to be able to calculate the quorum and determine which part of the cluster will continue to serve clients. For a stretched clusters we couldn’t use the locally shared disk – as for a local clusters -, we have to use a file share. Placing this file share was an issue and for the proper setup it has to be in the third location and it has to be highly available. Having a third location just for placing witness was demanding and very often it is placed in one of the two locations. In the case of primary location disaster and file share being placed there we would loose an entire cluster. Two nodes in the second location can not tell if they just lost connection to primary site or there was a disaster. They consist only two votes, while the other part of the cluster has three votes (2 nodes and witness) and the nodes on the second location will not failover roles from the nodes on the primary location. Solution is to place the witness in a third location – in the Azure as a highly available platform.
Clusters running within Azure VMs do not have a shared storage and therefore we can not use a shared disk as a cluster witness. So far we had to use a file share, but that required an extra VM just to place the share. Now we can use the Azure storage account as already mentioned which makes building virtual clusters in Azure less complicated.
Configuring the Cloud Witness is very simple, just follow the steps below.
Create Storage Account in Azure and get Manage Keys
Make sure you choose Locally Redundant replication since Geo Replicated can not guarantee consistency for blob file.
Storage Account Name and Access Key you will need to setup cluster quorum. If you want to regenerate the primary key for example, first update all your clusters with secondary key and vice verse.
Configure Cloud Witness
If you want to install the Reporting Services in failover cluster you must install them in separate SQL Server instance on each subsequent node. Funny thing, you can install the Reporting Services on the first cluster node normally. This “feature” was first introduced in SQL Server 2008 and it remains the same till today – check my previous posts about SQL 2008 and SQL 2012. What I needed was highly available Reporting Services and I didn’t want to manage multiple SQL Server instances side by side. There are scenarios when you should or want to use multiple instances, but not always and the use of existing instance seemed logical. Reporting Services are not cluster aware – this means that RS will be running on each cluster node all the time and they are not visible in Failover Cluster Manager.
When you run SQL Server setup to add other nodes to cluster you can not install Reporting Services – option is grayed out. If you run the setup again to add features on the second node and select the Reporting Services, the SQL Server setup rule “Existing clustered or cluster-prepared instance” will fail.
The solution is to run setup with option to skip check if Reporting Services are being installed in already clustered instance. Run the SQL Server setup, this time from Command Prompt:
Setup.exe /SkipRules=StandaloneInstall_HasClusteredOrPreparedInstanceCheck /Action=Install
After installing Reporting Services on failover cluster please keep in mind:
- Reporting Services running on an Active-Passive cluster handle requests on each cluster node on which the service is deployed.
- Report server must be configured to use SQL failover cluster virtual name to connect to the report server database. This is because it is hosted on a SQL Server that is part of a failover cluster. If not, the report server will be unable to connect to the report server database if a failover occurs.
This solution provides the highly available Reporting Services with default SQL server instance and uses already deployed hardware. It is not substitute for a true Scale-out deployment of the Reporting Services, but a way of achieving high availability (we just used the existing high availability platform).
Whenever I’m involved in a discussion about the Storage Spaces 3 major issues are always present:
- Performance related to disk layout
- Storage Pool Expansion – especially tiered one
- Parallel rebuild not working
The technology itself reached maturity, but you have to take care about few things when you deploy it. In this post I’ll try to explain what is the Storage Spaces Stripping and Mirroring and how is affecting the above mentioned issues. This is not an article that will cover everything about the Storage Spaces. If you have never worked with the technology I suggest you first visit the Storage Spaces Frequently Asked Questions (FAQ). Storage Spaces offer two types of resiliency: Mirror and Parity. There is also a third so called resiliency Simple, but it’s not resilient to disk failures at all. Here I will discuss only Mirror resiliency, since it is the resiliency you would use for production workloads like Hyper-V or SQL. There are two types of a mirror: two-way mirror and three-way mirror. First one creates one more copy of your data, while the other creates two more copies of your data. With the other you can use only 1/3 of available disk space and I have never used it in production.
Stripping and Mirroring
When you create the Storage Spaces they will try to create such a layout to achieve maximum performance by default. They will try to stripe the data across multiple disk to be able to read/write simultaneously. Stripping is defined with NumberOfColumns and Interleave where:
- NumberOfColumns represent the number of disks across which stripe is written.
- Interleave represents the amount of data written to a single column (default 256 KB).
Stripe size or stripe width, which represents the amount of data written in one pass to a Storage Space, is NumberOfColumns * Interleave. More columns (disks) the better performance you will have. If you use a GUI the maximum NumberOfColumns is 4 and Interleave is allways 256KB. When you create a Storage Spaces from PowerShell you can control the number of columns and the interleave with PowerShell cmdlet New-VirtualDisk with the NumberOfColumns and Interleave parameters.
Since we definitely need resiliency for the production workload, besides the stripping we will combine it with the mirroring. It allows us to write another copy of data so we can survive disk failure. To create a two-column two-way mirror you will need a minimum of 4 physical disks. Storage Spaces will stripe the data to two physical disks and then it will write another copy of data on the other two disks. This is very similar to RAID 0+1, but depending on the number of disks in the pool and the number of spaces created, it will use 4 disks, but not the way RAID controllers usually do. That’s why when talking about the Storage Spaces RAID terminology is not used. This type of resiliency can sustain the failure of a single disk.
Let’s sum up all the above in a few examples to make it more clear. Minimum number of disks you need is 2 * NumberOfColumns, because after the stripping we need the same size of disk set to write the copy of data – you remember.
6 disks, 3-columns 2-way mirror
With 6 physical disks to achieve the maximum performance we will use 3 columns in two-way mirror. The data will be first stripped to three disks and then copied to another three disks.
6 disks, 2-columns 2-way mirror
In this example we are not using the maximum performance since we are stripping data “only” to two disks simultaneously, but with this layout you will need less disks to expand the storage pool.
Figure 2 – 6 disks, 2-columns 2-way mirror Storage Spaces layout
Storage tiers allow us to create the Storage Spaces on top of the pool which combines SSDs and HDDs. Data is moved on the subfile level between the two tiers, based on how frequently the data is accessed. The most used, so called “hot” data is moved to the SSD tier, while major amount of data, infrequently accessed, is kept on the HDD tier. This way we can have the performance and capacity of inexpensive HDDs. When creating a Storage Space with storage tiers, the storage pool must have a sufficient number of SSDs and HDDs to support the selected storage layout. Stripping model (number of columns) has to be the same in the both tiers. Besides keeping the “hot” data on SSDs, storage tiering also creates Write Back Cache on part of SSD tier (default 1GB) which additionally improves the overall performance.
4 SSD, 6 HDD, 2-columns 2-way mirror
The maximum number of columns we can have is determined by the number of disks in fewer tier. In this example the SSD tier has 4 disks and the maximum number of columns for two-way mirror is 2. The data will be stripped to two disks and then copied to another two disks. In HDD tier we have 6 HDDs, but the data will also be stripped to two disks and then copied to another two disks. HDD tier could perform better with the higher number of columns, but it has to follow the SSD tier layout.
Figure 3 – Tiered Storage Spaces with 4 SSD, 6 HDD and 2-columns 2-way mirror layout
2 SSD, 6 HDD, 1-column 2-way mirror
In this example the SSD tier has 2 disks and maximum number of columns is 1. No data stripping can occur is such a layout. Data is written to first SSD and then another copy on the other SSD. The same behavior occurs on the HDD tier. Performance of the HDD tier is equal to performance of a single HDD.
Figure 4 – Tiered Storage Spaces with 2 SSD, 6 HDD and 1-column 2-way mirror layout
This is a feature that automatically rebuilds a Storage Spaces from the storage pool free space when a physical disk fails. Marketing says that you don’t need an extra disk for Hot spare, which is true, but you need to reserve the free space in the pool. Unprovisioned space size has to be the same or larger than a size of a single disk for this feature to work. This applies to the both tiers considering the size of disks in each tier. Keep in mind that each tier will rebuild only within that tier. Main advantage is rebuilding the data from the failed disk to multiple physical disks in the pool instead to a single hot spare. Full resiliency is achieved in a shorter time, while rebuilding a single large disk can be time consuming.
Figure 5 – Storage Spaces parallel rebuild
Storage Pool Expansion
If you want to expand a Storage Pool you have to follow the Storage Spaces layout. If you are using tiers each tier can be expanded independently. In the example below which is 2-columns 2-way mirror tiered Storage Space, you will need minimum of 4 SSDs or 4 HDDs to expand the each tier. For the layout shown on figure 4 only two disks in each layout are sufficient for the expansion.
Figure 6 – expanding Storage Spaces
There is no magic layout that fits all. Below are a few guidelines, which you may or may not choose to follow. They are based on my experience and it’s my view of how it should be done.
- The more columns the better performance (bigger stripe).
- Always follow the storage layout.
- When using tiers use at least 4 SSDs if you want stripping.
- Create a single Storage Space in a Storage Pool. Mixing Simple, Mirror, Parity or Dual Parity in the same Storage Pool is only for non-demanding workloads and the layout is hard to track.
- Unpartitioned space has to be large enough for a parallel rebuild to work.
- Test, test and test before you put anything into the production.
Cluster Operating System Rolling Upgrade is a full name for something we all have been waiting for a long time. It is a feature allowing you to have Windows Server Failover Cluster where nodes are running different versions of Windows Server OS. Everyone who performed, even only once, migration upgrade of Failover Cluster will appreciate this new feature – I know I will.
Main benefits of Rolling Upgrade are:
- Additional hardware is not required
- A new cluster is not required
- No need to stop or restart cluster
The idea is to upgrade the cluster without any downtime by moving all the roles from a Windows Server 2012 R2 node to Windows Server Technical Preview node. Technical Preview nodes added to the cluster operate in a Windows Server 2012 R2 compatibility mode, while cluster operates in so called mixed mode. In Windows Server Technical Preview upgrade scenarios support Hyper-V clusters and Scale-Out File Servers and clusters running Windows Server 2012 R2. In the demo details shown below I upgraded Scale-Out File Server cluster.
The state transitions are shown here:
The basic upgrade steps are:
- Pause node and drain roles
- Evict node from the cluster
- Install Windows Server Technical Preview (In-place upgrade is not supported – I wouldn’t recommend it anyway)
- Join node back to the cluster
- Reinstall cluster role (Hyper-V, SOFS, SQL etc.)
- Repeat above steps for the remaining nodes
- Upgrade ClusterFunctionalLevel with PowerShell
After all the nodes have been updated you still have the chance to rollback, especially if your apps experience any problem. Windows Server Technical Preview nodes can be removed from the cluster and Windows Server 2012 R2 nodes can be added back to the cluster in this mode. Once the Update-ClusterFunctionalLevel PowerShell cmdlet is run on the cluster you can’t go back. There is no technical limit for the cluster to run in mixed mode or in Windows Server 2012 R2ClusterFunctionalLevel, but Microsoft intend to provide support for maximum of 4 weeks for such a cluster. It is recommended that all the nodes and the ClusterFunctionalLevel is upgraded as soon as possible. Please keep in mind that you should do all the administration, either with FCM or PowerShell, from the node running Windows Server Technical Preview.
To test the upgrade I created virtual Scale-Out File Server with Windows Server 2012 R2 and then upgraded it to Windows Server Technical Preview. For this demo I used 5 VMs: Domain Controller, iSCSI Target, 3 Cluster Nodes. I created SOFS as two node cluster, then I added the 3rd node running Windows Server Technical Preview and upgraded first two nodes. Also, I created 3 CSVs to see if Automatic Rebalancing will move one CSV (distribute CSVs on each node) after I add 3rd node running Technical Preview to the cluster.
Virtual lab diagram:
SOFS with 2 nodes, 3 CSVs and a Share on each CSV:
3rd node with Windows Server Technical Preview is added to the cluster:
Automatic Rebalancing kicked in and distributed CSVs evenly including the CU-NODE3 which is running Windows Server Technical Preview. At this point we have a cluster with nodes running different version of Windows, but participating equally.
After upgrading first two nodes ClusterFunctionalLevel is still Windows Server 2012 R2:
Upgraded ClusterFunctionalLevel to Windows Server Technical Preview:
This is definitely the awesome feature in Windows Server, an upgrade without any downtime. Microsoft is really going to spoil us, now I would like to see the same functionality within SQL Server running in Failover Cluster. Imagine upgrading Windows/SQL failover cluster to the new OS and DB version without any downtime.