Introduction
Enterprises capable of meeting demands for 24/7 availability, whether On-Premise or Private Cloud, must be prepared to handle different loads and different failures with minimal or no downtime. To avoid unscheduled downtimes, high availability databases are used. These databases are configured in such a way that single points of failure are eliminated and the databases are optimized to ensure that the end user does not experience an interruption in service or a degradation in user experience when hardware or networks fail.
As compared to a relational database, High Availability database architecture design is generally easy, less complex and ensures high availability in a multiple node and cluster environment. Based on a masterless architecture that uses clustering, with multiple servers grouped together, any server within a cluster can respond to read or write requests. Data replication is achieved across all servers in the cluster to provide system redundancy and to minimize downtime possibility.
In this article, we will be setting up a Highly Available Environment, which includes High Availability Database Servers and High Availability Application Servers.
The article is divided into the following sections:
1. Highly Available Environment for SQL (DB) Servers
i. Windows Failover Cluster Feature Installation
ii. Windows Failover Clustering Configuration for SQL Server 2016 AlwaysOn Availability Groups
iii. Enable SQL Server 2016 AlwaysOn Availability Groups Feature
iv. Create and Configure SQL Server 2016 AlwaysOn Availability Groups
2: Highly Available Environment for VIDIZMO (Application) Servers
i. Installation Prerequisites
ii. Installation of VIDIZMO on Application Servers (Repeat same steps on each Application Server and Each Application Server will be activated with “Unique” License Key. Ask VIDIZMO Sales representative to provide these.)
3: Post Installation Tasks
i. Add Databases to High Availability Group
ii. CDN and Upload Path
iii. Cache Server List
2. Server Requirements
Following are the minimum requirements for installing VIDIZMO in an On-Premises/Private Cloud environment:
2.1 Database Servers:
Following are the minimum requirements for installing an SQL Server:
- Database Server:
- Windows Server 2016 Standard Edition VM/Physical Server with a minimum of 16GB RAM, 4 Core CPU, 256 GB HDD with 150 Mbps Throughput
- Microsoft SQL Server 2017 Standard or Enterprise Edition
- SQL Server Management Studio (will be installed separately in case of SQL Server 2016)
- Open Firewall Ports between application and database servers: 80, 443, 11211, 1433
- All machines should be on the same Local or Virtual Network.
2.2 Application Servers:
Windows Server 2016 Standard Edition VM/Physical Server with a minimum of 16GB RAM, 4 Core CPU, 128 GB HDD with 150 Mbps Throughput.
- IIS 10 or greater
- Microsoft .Net Framework 4.6
- .Net Core Hosting Bundle 2.1.6 or later
Open Firewall Ports between application and database servers: 80, 443, 11211, 1433:
- Open Firewall Ports 80 and 443 between corporate firewall and following domains. This is required for VIDIZMO Software License Activation, monitoring, receiving ongoing software updates and upgrades, and VIDIZMO player to function properly.
- *.enterprisetube.com
- *.vidizmo.com
- *.vo.msecnd.net
- Open Firewall Port 1935 between your live encoder and streaming server, for live streaming purposes. If you are using VIDIZMO ECDN, this port should be open on all ECDN nodes. For cloud-based streaming depending on which of the following streaming providers you choose for its streaming services, following streaming server domains should be open for access via this port:
- *.windows.net (Azure Media Services)
- *.akamaihd.net (Wowza Streaming Cloud)
- All machines should be on the same Local or Virtual Network.
- Administrative access to the VM/server on which VIDIZMO Application will be installed.
2.3 Encoding Servers:
Windows Server 2012 R2/2016 Standard Edition VM/Physical Server with a minimum of 8GB RAM, 4 Core CPU, 256 GB HDD with 150 Mbps Throughput Microsoft Office 2013/2016 Professional
- Microsoft Office 2013/2016 Professional
- Microsoft .Net Framework 4.5.2
- Microsoft Windows Management Framework 5.1
- Open Firewall Ports between application and database servers: 80, 443, 11211, 1433
- All machines should be on the same Local or Virtual Network.
- Administrative access to the VM/Server (Application Servers) on which VIDIZMO Application will be installed.
3. Installation Prerequisites
Prerequisites for installation and configuration of VIDIZMO in an On-Premise/Private Cloud environment are:
- Domain Name for your Media Portal: This is the domain name users will type in to access your media portal. The domain name should map to a public (or private in case external access is not intended) IP of VIDIZMO Application Server.
- SSL Certificate for Selected Domain Name (Optional): In case use of HTTPS is required for secure communication between users’ browser and VIDIZMO application.
- SMTP Service Credentials: For VIDIZMO Application to send out email notifications to administrators and users triggered in different scenarios such as registration requests, content upload, content sharing etc. This can be a service account (recommended) or user account that can be obtained from your corporate mail server administrator. Alternatively, it can also be setup with a 3rd party SMTP service such as sendgrid.com, Amazon SES etc.
4. Highly Available Environment for SQL (DB) Servers
Installation
To install SQL Server to set up your Database Server, see: Install SQL Server from the Installation Wizard. Following this Installation Wizard, under Step 11 for Feature Selection, you need to select the following features:
- Database Engine Services
- Full-Text and Semantic Extractions for Search
- SQL Client Connectivity SDK
Highly Available SQL Servers
SQL Server 2012 AlwaysOn Availability Groups bring SQL Server high availability and disaster recovery to a whole new level by allowing multiple copies of the database be highly available and potentially using them for read-only workloads and offloading management tasks such as backups. AlwaysOn Availability Groups allow you to fail over a group of databases as a single entity. This is very useful for applications that access multiple databases in a single SQL Server instance.
This step-by-step has been created to help you get started in creating a SQL Server 2012 AlwaysOn Availability Group for your mission-critical databases.
Prerequisites
- Windows Server Failover Cluster (WSFC). AlwaysOn Availability Groups rely on the Windows Server Failover Cluster for failure detection and management of the Availability Group replicas. Windows Server 2008 and higher provide the option to use a file share witness as a quorum configuration. Therefore, you DO NOT need sharing storage to create a Windows Server Failover Cluster for AlwaysOn Availability Groups as earlier versions of Windows.
- Download SQL Server 2016 Enterprise Edition. AlwaysOn Availability Group is an Enterprise Edition feature. Before deciding to implement this feature, take stock of your SQL Server licenses to make sure you have enough to get you covered.
- Same SQL Server collation for all replicas. If you want to configure AlwaysOn Availability Groups for your databases, they should all be running the same collation on all of the SQL Server instances acting as replicas.
- Two to Five SQL Server Instances acting as replicas. SQL Server instances that will be used as a standby for high availability and/or disaster recovery are called replicas. Unlike database mirroring where you can only have one extra copy of the database, AlwaysOn Availability Groups allow you to have up to five copies of the database running on five replicas – three of which can be configured for synchronous-commit mode and two in asynchronous-commit mode.
4.1 Windows Failover Cluster Feature Installation
Since AlwaysOn Availability Groups require a Windows Server Failover Cluster, we first need to add the Windows Failover Cluster Feature to all the machines running the SQL Server instances that we will configure as replicas. For the operating system, we will be using Windows Server 2012 R2. To add the Failover Clustering feature:
1. Open the Server Manager console and select Add roles and features. This will launch the Add Roles Features Wizard
2. Click Next until you reach the Select Features dialog box. Select the Failover Clustering checkbox. When prompted with the Add features that are required for Failover Clustering dialog box, click Add Features. Click Next.
3. Click Install to install the Failover Clustering feature.
4.2 Windows Failover Clustering Configuration for SQL Server 2016 AlwaysOn Availability Groups
Steps
To configure Windows Failover Clustering, you must be logged in with a Domain User Account. A Domain User Account is required because you need access to Active Directory Domain Services.
1. Launch Failover Cluster Manager from within the Server Manager console.
2. Within Failover Cluster Manager, click the Validate Configuration link in the right pane under "Actions".
3. In the Validate a Configuration Wizard dialog box, click Next.
4. In the Select Servers or a Cluster dialog box, add the server hostnames of the SQL Server instances that you want to configure as replicas in your Availability Group. Click Next.
5. In the Testing Options dialog box, make sure that the option Run all tests (recommended) is selected. Click Next.
6. In the Confirmation dialog box, click Next.
7. In the Summary dialog box, click Finish to create the Windows Failover Cluster.
NOTE: The Failover Cluster Validation Wizard is expected to return several Warning messages, especially if you will not be using shared storage. As mentioned earlier, there is no need to use shared storage to create the Windows Server Failover Cluster that we will use for our Availability Group. Just be aware of these Warning messages as we will configure a file share witness for our cluster quorum configuration. However, if you see any Error messages, you need to fix those first prior to creating the Windows Server Failover Cluster.
8. As the Validate Configuration Wizard completes, a new Wizard "Create Cluster Wizard" starts.
In the Access Point for Administering the Cluster dialog box, enter the Cluster name of your Windows Server Failover Cluster. Click on the "Next" button to continue.
9. In the Confirmation dialog box, click Next. This will create the Windows Failover Cluster using the servers as nodes of the cluster, add DNS and Active Directory entries for the Cluster Host Name.
10. The Summary dialog box should display the successful configuration details.
11. To configure the Cluster Quorum Configuration to use a file share:
i. Right-click on the cluster name.
ii. Select More Actions and
iii. Click Configure Cluster Quorum Settings…
We will be configuring a file share witness for our cluster quorum setting. By default, the wizard will configure the cluster to use Node Majority.
Click Next to continue.
12. In the Select Quorum Configuration page, select the Add or change the quorum witness option. Click Next.
In the Select Quorum Witness page, select the Configure a file share witness (recommended for special configuration) option. Click Next.
13. In the Configure File Share Witness page, type path of the file share that you want to use in the File Share Path text box. Click Next.
14. In the Confirmation page, click Next.
15. In the Summary page, click Finish.
4.3 Enable SQL Server 2016 AlwaysOn Availability Groups Feature
Once the Windows Server Failover Cluster has been created, we can now proceed with enabling the AlwaysOn Availability Groups feature in SQL Server 2012.
This needs to be done on all of the SQL Server instances that you will configure as replicas in your Availability Group.
To enable the SQL Server 2012 AlwaysOn Availability Groups feature,
1. Open SQL Server Configuration Manager. Double-click the SQLServer (MSSQLSERVER) service to open the Properties dialog box.
2. In the Properties dialog box, select the AlwaysOn High Availability tab. Check the Enable AlwaysOn Availability Groups check box. This will prompt you to restart the SQL Server service. Click OK.
4.4 Create and Configure SQL Server 2016 AlwaysOn Availability Groups
Availability Groups can be created on existing databases or even a temporary one in preparation for application installation. If you intend to create an Availability Group for a new SharePoint 2013 farm, you will need to create a temporary database. This is so that the SharePoint 2013 farm will use the AlwaysOn Availability Group when creating the farm configuration and the admin content databases. After the SharePoint 2013 farm has been created, this database can be removed from the Availability Group configuration and deleted from the instance.
To create and configure a SQL Server 2012 AlwaysOn Availability Group follow the steps below:
1. Open SQL Server Management Studio. Connect to the SQL Server instance.
2. In Object Explorer, expand the AlwaysOn High Availability folder. Right-click on the Availability Groups folder and select the New Availability Group Wizard… option. This will launch the New Availability Group Wizard.
3. In the Introduction page, click Next.
4. In the Specify Availability Group Name page, enter the name of the Availability Group in the Availability group name field.
Click on the "Next" button to continue.
5. In the Select Databases page, select the checkbox beside the database that you want to include in your Availability Group.
The databases have to be in Full Recovery model and the database must be backed up by selecting the Backup Type as "Full" prior to joining them in the Availability group.
Click on the "Next" button to continue.
6. In the Specify Replicas page, under the Replicas tab, click the Add Replicas button and connect to the other SQL Server instances that you joined as nodes in your Windows Server Failover Cluster.
Here, configure the following options
- Automatic Failover (Up to 3): Checked
- Synchronous Commit (Up to 3): Checked
- Readable Secondary: No
7. In the Endpoints tab, verify that the port number value is 5022.
8. In the Listener tab, select the Create an Availability Group Listener option and enter the following details:
- Listener DNS Name:Create a New ‘Host A’ record with suitable name and assign a Static Available IP. This name will be used in your application connection. A Listener is required for an application to connect with Availability Group in SQL.
- Port:1433
- Port:1433
9. Click the Add… button to provide the IP address. In the Add IP Address dialog box, enter your preferred virtual IP address in the IPv4 Address field.
Click OK and then click on the "Next " button to continue.
10. In the Select Initial Data Synchronization page,
i. Select Initial Data Synchronization.
ii. Select the Full option.
iii. Provide a shared folder that is accessible by the replicas and that the SQL Server service account used by both replicas has Write permissions.
This is just a temporary file share to store the database backups that will be used to initialize the databases in an Availability group. If you are dealing with large databases, it is recommended that you manually initialize the databases prior to configuring them as your network bandwidth may not be able to accommodate the size of the database backups.
Click on the "Next" button to continue.
11. On the Validation page, verify that all validation checks return successful results. Click Next to continue.
12. On the Summary page, verify all configuration settings and click Finish. This will create and configure the AlwaysOn Availability Group and join the databases.
13. On the Results page, verify that all tasks have been completed successfully.
With the completion of this step, SQL Server 2012 AlwaysOn Availability Groups gets created.
You can check all the values defined in the Availability Group from the Object Explorer:
You can now use the Availability Groups listener name in your application connection string. Keep in mind that you need to manually add new databases in the Availability Group even though your application has already been using the listener name.
Be sure to monitor the replicas in your Availability Groups to be alerted when new databases are created.
Possible Issue:
When you configure SQL Server AlwaysOn Available Group from the Management Studio it may fail while joining Databases to the availability group on the secondary replica. In that case, it may give the following error:
Following Error is displayed on Availability Group Dashboard
The connection to the primary replica is not active. The command cannot be processed.
Error 1
{
Joining database on secondary replica resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)
——————————
ADDITIONAL INFORMATION:
Failed to join the database ‘AG’ to the availability group ‘AG1’ on the availability replica ‘NODE2’. (Microsoft.SqlServer.Smo)
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
——————————
The connection to the primary replica is not active. The command cannot be processed. (Microsoft SQL Server, Error: 35250)
}
Error 2
{
TITLE: Microsoft SQL Server Management Studio
——————————
Failed to join the instance ‘NODE2’ to the availability group ‘AG1’. (Microsoft.SqlServer.Management.SDK.TaskForms)
For help, click:
http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=11.0.2100.60+((SQL11_RTM).120210-1917+)&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&LinkId=20476
——————————
ADDITIONAL INFORMATION:
Failed to join local availability replica to availability group ‘AG1’. The operation encountered SQL Server error 41106 and has been rolled back. Check the SQL Server error log for more details. When the cause of the error has been resolved, retry the ALTER AVAILABILITY GROUP JOIN command. (Microsoft SQL Server, Error: 41158)
For help, click:
}
When you configure AG availability group using alter database command given below or synchronization might fail with 35250 error, you may get the error mentioned below.
This command is used to Add Database to the Availability Group.
ALTER DATABASE [AG] SET HADR AVAILABILITY GROUP = [Group name];
Error 1
Msg 35250, Level 16, State 7, Line 1
The connection to the primary replica is not active. The command cannot be processed.
To resolve the above errors:
1. Ensure AlwaysOn endpoints ([Hadr_endpoint]) are not blocked by the firewall (Default port 5022).
2. If the login account of SQL Server is “Nt service\” or local system account, then ensure that the System Account (YourAdDomainname\systemname$) of each replica is added to other replicas.
e.g.
on DB01 run the following Query
CREATE LOGIN [YOURADDOMAIN\DB02$] FROM WINDOWS
on DB02 run the following Query
CREATE LOGIN [YOURADDOMAIN\DB01$] FROM WINDOWS
3. Grant connect on always on endpoints created on each replica for startup account of other replica servers (Grant connect on endpoints even if startup account of other replicas are added as sysadmins).
e.g.
on DB01 run the following Query
GRANT CONNECT ON ENDPOINT::[Hadr_endpoint] TO [YOURADDOMAIN\DB02$]
on DB02 run the following Query
GRANT CONNECT ON ENDPOINT::[Hadr_endpoint] TO [YOURADDOMAIN\DB01$]
5. Highly Available Environment for VIDIZMO Application Servers
When setting up Application Servers for VIDIZMO, follow the instructions provided in the article How To Prepare A New Application Server For VIDIZMO.
The installation instructions are the same for Highly Available Environments with the following exceptions:
1. Under Section 4 Install SQL Reporting Services > Step 7 Report Server Database Configuration Wizard: Provide the name of the "Listener" you configured for High Availability DB environment in Section 1 Step 8 of this article, instead of "DB Server" name.
2. Under Section 3 Installation of VIDIZMO on Application Server > Step 18 VIDIZMO Installation Wizard: Provide the name of the "Listener" instead of "DB Server" when selecting a Database for VIDIZMO to use.
3. Under Section 3 Installation of VIDIZMO on Application Server > Step 32 VIDIZMO Software Activation: Prior to executing this step, edit the local host file of the Application Server and map loopback address (127.0.0.1) with the Domain Name of your VIDIZMO Media Portal as selected in Step 16. This step needs to be repeated for all VIDIZMO Application Servers where VIDIZMO software is installed and activated.
Why do we need to edit localhost file?
To setup a Highly Available Environment, multiple instances of VIDIZMO Application need to be installed, with each Application activated using a unique VIDIZMO Software License Key.
VIDIZMO Licenses are bound with the Domain Name so when the activation takes place, VIDIZMO central licensing servers validate the license with the Application Server. In High Available environments the domain name is mapped to the Load Balancer instead of the of the Application Server, which causes this validation process to fail since the requests are being load balanced and not guaranteed to be forwarded to the intended Application Server. For a license to be activated successfully, the Media Portal Domain Name needs to resolve to the Application Server on which the VIDIZMO software has been installed. This is achieved by editing the localhost file and mapping the VIDIZMO Media Portal Domain Name so that all requests generated by the Application Server for that domain name always map to itself.
For License Key information, please contact VIDIZMO sales on sales@vidizmo.com.
6. Post Installation Tasks
6.1 Add Databases to Availability Group
There are Multiple Replicas (Database Servers/Nodes) in SQL AlwaysOn High Availability Group. One replica (server/node) is set to ”Primary”. After configuration of Reporting Services and Installing VIDIZMO, following Databases are created in DB Server but only as the “Primary”. In this case, we will configure:
- Primary Replica: DB01
- Secondary Replica: DB02
- Reporting Services Databases: VIDIZMO Databases
- ReportServer Logging: VIDIZMODB
- ReportServerTempDB System Information: VIDIZMO Notification System
1. Open “SQL Management Studio” on any SQL Server and Connect to both DB01 and DB02. The databases are created only in Primary Replica which is “DB01” (image " i below).
In a High Available environment, all the Databases must be synchronized on all the DB Servers/Nodes.
2. Before adding Databases to other Servers/Nodes
i. Make sure “Recovery Model” of each Database is set to “Full”.
To verify this right click on each Database > Properties > Options > Recovery Model
ii. Take “Full Backup” of each Database
Right click on each Database > Tasks > Backup > Backup Type: Full > OK
3. Now in order to Add Databases to other Nodes, in SQL Management Studio look for “AlwaysOn High Availability“ and expand to view details by navigating to:
AlwaysOn High Availability > Availability Groups > [Current Availability Group] > Availability Databases
Right click on “Availability Databases” and click “Add Database”
4. On select Database select, all Databases Mentioned Earlier click “Next”
5. On Data Synchronization Preference:
i. Select the "Full" option
ii. Browse to the network share path and provide it here. Click “Next” to continue.
6. On the Next screen connect to the other Replicas, click “Next”.
7. In the last step, it will Restore and join all the Databases to the other Replicas. Click “Close”.
8. Make sure that all the Databases are in “Up” state in
AlwaysOn High Availability > Availability Groups > [Current Availability Group] > Availability Databases
9. Also, make sure that all the Databases are in “Synchronized” state on all the Servers/Nodes.
6.2 CDN and Upload Path
It is recommended to use the same Network Location from where the Content would be served. For this purpose create a “Network Share Location” that will be used as CDN and Upload Path for Vidizmo Application. In this case, the Share Path is \\FILESERVER\new\.
The following steps will be performed on the Application Server:
Steps
1. Log in to the Application Server and open “Internet Information Services (IIS)” and run it as administrator.
2. Locate “CDN” under VIDIZMO (WebApp) sites in IIS
3. Right click “CDN” and click “Explore”. This will open currently configured folder behind the CDN application.
4. Select and “Copy” the following:
i. “vidizmo-configuration”: File Folder
ii. “clientaccesspolicy”: XML Document
iii. “crossdomain”: XML Document
iv. “Web.config”: CONFIG File
5. “Paste” the Copied file into the Network share location created in Step#1
6. Next, right click on “CDN” again and click “Remove”. When prompted select “Yes”
7. After removing the CDN folder, right click VIDIZMO and select “Add Application”
8. Provide “CDN” in Alias and paste the same “Network Share Path”, in “Physical Path” field. Make sure to terminate the folder path with ‘\’.
9. Click on “Connect as” and provide user credentials of the user having “Full Control” permissions over the network path. Make sure to provide full username including the domain name. In this case, the username is “wowzauser@softechww.com”
10. Next, select “Application Pool” and select “CDN” from the list and click on the "OK" button
11. Click “OK” to save the updated Settings.
12. After updating settings, expand the newly added “CDN” application in IIS and verify “UploadService” still exists in IIS.
13. Next, under IIS, click “Application Pools”
14. Right click “UploadService” under CDN and select “Advanced Settings”.
i. Select “Identity” under “Process Model”.
ii. Change Identity value from “NetworkService” to the Domain user having “Full Control” on network share folder (again in this case the user is wowzauser.softechww.com) and click on the “OK” button to continue.
With this step, IIS configuration is complete. Next VIDIZMO Application configuration will be updated
15. Next, go to VIDIZMO portal and login using credentials having “Administrative” level user rights
16. After you log in, go to ‘Web Admin’ > ‘System Configuration’ option.
17. Select “Domains and URLs” and look for the “UploadPath” configuration.
18. Provide the same Network Share Path in “UploadPath” field and press “Update” button in line and then “Apply Changes & Reload” button on the top.
19. After this step, VIDIZMO application will restart and reload the updated Configuration.
With this last step, VIDIZMO application is configured to use the defined Network Path to store content. This configuration is applied across entire VIDIZMO instances and is used by all Channels under the Account.
6.3 Cache Server List
Since there are more than one Application Servers you need to update the Cache Server list in the application configuration (System Configuration) because if the Cache Server list is not updated, user login session will expire each time the Application Server switches from one Server to another.
1. Go to VIDIZMO portal and login using credentials having “Administrative” level user rights
2. Login, and then go to ‘Web Admin’ > ‘System Configuration’ option
3. Select “Cache” and look for “CacheServerList” configuration, by default, it is set to the 127.0.0.1
4. Replace the default IP 127.0.01 and enter the IPs of all Application Servers as comma separated entries and press the “Update” button in line. When done, click on “Apply Changes & Reload” button on the top.
After this step, VIDIZMO application will restart and reload the updated Configuration.
The Highly Available Environment is now ready to use.