Using XCM to update HA-replicated CSGs with Xona v5.5.0 and higher

Table of Contents

• Background
• Update Notes & Recommendations
  • General
  • HA Replication
  • Xona Fabric 
  • Note: Conventions Used
  • Recommendation: Maintenance Web Connection
• Procedure Assumptions & Prerequisites
• Procedure Risks & Mitigations
• The Update Procedure
• The Switch Procedure

Affected Environments

• Platform: [x] Hardware (DIN Rail, 1U Server) | [x] Virtual Image (VMware/Hyper-V) | [x] AWS AMI | [x] Azure VM
• Component: HA Replication mode CSGs joined to XCM

Introduction

Xona v5.5.0 introduced new CSG management features for XCM, including the ability to push staged Xona updates and install them directly from the XCM interface. One limitation to this feature is that it does not work with CSGs configured in High Availability (HA) Replication mode. To work around this limitation, this KB article provides details about procedures that can be used when XCM-joined CSGs are operating in HA Replication mode.

Overview

The procedures involve updating the CSGs while switching the HA replication modes of the Primary and Replica CSGs. The Xona administrator will need a way to directly access the CSG web interfaces to control the HA replication settings. Manual steps will be taken to temporarily disable HA replication during the update, change the Xona Fabric status, and re-enable HA replication after the update completes.

Please review all sections of this KB article. The Update Notes & Recommendations are useful for ensuring a smooth update process. The Procedure Risks & Mitigations section is essential for understanding potential problems that can arise and how to avoid them. The Procedure Assumptions & Prerequisites section contains key items that should be validated before proceeding with the update procedures.

Update Notes & Recommendations

When updating your Xona CSGs, please keep in mind the following notes and recommendations.

General

Applicable to all Xona updates, regardless of eiter HA Replication or Xona Fabric status.

1. It is considered best practice to setup and use a test environment to become familiar with this procedure.
2. Reference the Release Notes documentation page. Cite the documentation page as a supporting document if a change request is required per your organization. If needed, supplement with the Security & Compliance documentation section.
3. Coordinate to schedule a maintenance period. Notify your stakeholders and users, both before and after the update.
4. Updates are not risk-free, but many risks can be mitigated. A common risk mitigation is an out-of-band management network, VPN, other SRA technology, or availability of on-site assistance from another Xona administrator.
5. Check the system Logs for any unusual activity after updates. Errors may be a cause for concern, especially if they are associated with unexpected behavior. If you encounter a problem after your update, then please contact support@xonasystem.com.
6. Refer to the System Updates section in the main Xona documentation for more info.

HA Replication

Notes and recommendations specific to HA Replication:

1. Document the network architecture & ensure connectivity is verified for both HA Primary and HA Replica CSG network paths.
2. Document the permissions configuration for users, groups, and resources (system-level permissions, connections, file buckets, etc.)
3. Take a backup of the HA Primary before and after the update. Ensure the backup is uniquely named so that it is not confused with other backups.
4. If possible, prevent non-Admin users from logging in during the update procedure. If you have a small number of local users, then consider temporarily disabling them. If you have more users logging in through either Active Directory or SSO, then consider temporarily disabling the connector. Remember to re-enable these after the update is completed.

Xona Fabric

Notes and recommendations specific to Xona Fabric:

1. The XCM will upload the update to the CSG. If the update file upload is interrupted, then it will be retried until it successfully completes.
2. Time between when the update installations begins and ends should be no more than 30 minutes, typically less with newer hardware versions.

Note: Conventions Used

The procedures in this KB article use the following conventions:

1. The XCM lists CSG-A, CSG-B, CSG-C, etc. for CSGs.

2. CSG-A1 is the hostname of the current HA Primary and CSG-A2 is the hostname of the current HA Replica.

Recommendation: Maintenance Web Connection

For Xona deployments where administrators do not have direct access to the CSG, it can be beneficial to set configure a connection for maintenance purposes. By default, CSGs come with a CSG Setup SSH connection that is important for maintenance purposes. In addition to this connection, administrators may want to setup a Web connection that provides access to the CSG web interface. Not all CSG functions can be managed from the XCM, including Xona Fabric and High Availability Replication settings. This limitation can be overcome with a dedicated Web connection.

A maintenance Web connection should have the following parameters:

• Application URL - The URL should point to your HA Primary CSG.
• Allowed Domains - Resolvable IPs or domain names of the HA Primary and HA Replica CSGs. If the CSGs use FQDNs with a subdomain (CSG-A1.example.internal, CSG-A2.example.internal), then a wildcard can be used instead (*.example.internal).
• Allow Multiple Sessions, Enable Clipboard, Enable Downloads.

In addition to the above connection configuration, it is very beneficial to have the the CSGs and XCM trust each other's HTTPS certificate. If a common CA was used to sign all the certificates, please add that CA's signing certificate to Trusted Certificates under Settings > Security > Certificates. If using self-signed certificates, then each certificate can be uploaded as Trusted Certificates. This will ensure browser warnings are not shown in the maintenance web connection.

Procedure Assumptions & Prerequisites

1. Both the HA Primary and HA Replica CSGs have verified stable network communication paths:
   1. Xona Fabric to the XCM.
   2. HA replication in either direction.
   3. HTTPS access to each other's web interface.
   4. NTP for synchronization to network time servers.
2. Both the HA Primary and HA Replica CSGs have rebooted in the past and not had subsequent connectivity issues.
   1. It is important to establish this fact before beginning the update procedure because connectivity problems after a reboot are a strong indicator of a dual-gateway misconfiguration. For more information, please see On-Premises Appliances: Resolving Dual Gateway Misconfigurations on Multi-NIC CSGs.

3. The HA Primary and HA Replica CSGs are interchangeable, i.e. it does not matter which one is the Primary/Replica as long as they’re able to work.

   1. If the HA Primary is preferred, then first complete The Update Procedure before moving on to The Switch Procedure.

4. The CSG local admin account is used for steps involving CSGs, to avoid potential problems with AD/SSO integrations during the upgrade process.

5. The Xona administrator has some means of accessing both HA Primary CSG and HA Replica CSG web interfaces, such as:

   1. Out-of-band management network, VPN, or SRA technology.
   2. RDP connection to a jump box on the CSG trusted network.
   3. Web connection configured on the CSG to allow for access to the web interface.
   4. On-site assistance from another Xona administrator.

6. The Xona Fabric permissions setup for resources on CSGs is documented and understood. I.e., the groups assigned to specific connection, file bucket, and schedule permissions are known in detail by the Xona administrators.

Procedure Risks & Mitigations

1. Loss of Xona Fabric on the Gateway

   1. Risk: Failure to correctly implement Procedure Assumptions & Prerequisites: #1 or follow certain ports of either The Update Procedure / The Switch Procedure will result in the XCM users, groups, and their associated permissions, being deleted from the CSG. Depending on the environment, this risk can be compounded with Risks & Mitigations: #3 Loss of Connectivity.
   2. Mitigations: If the CSG has not been removed from the perspective of the XCM, and if the XCM's data still remains on the HA Replica CSG (e.g., replication was disabled before the CSG left the fabric), then the data will be preserved if the HA Replica CSG is promoted to the role of HA Primary. If not, then the Xona administrator must be prepared to re-join CSG-A to the Xona Fabric and reconfigure permissions.
2. Conflicting HA Primary CSGs
   1. Risk: If The Update Procedure: Step 8 is not immediately followed up with Step 9, or The Switch Procedure: Step 2 is not immediately followed up with Step 3, then both CSG-A1 and CSG-A2 could simultaneously attempt to communicate  with the XCM as HA Primary CSGs. This will make the XCM connection to the CSGs unstable, disrupt access to connections on the CSGs, and lead to synchronization conflicts with the XCM.
   2. Mitigation: Ensure you have a way to configure one of the CSGs as an HA Replica. You may need an out-of-band management network, VPN, other SRA technology, or on-site assistance from another Xona administrator.
3. Loss of Connectivity
   1. Risk: If the Xona admin relies on a connection profile within Xona to access the CSG web interfaces, then it can become unavailable. A temporary gap in availability is part of the procedure, but if performed incorrectly, then manual intervention will be required.
   2. Mitigations: You may need an out-of-band management network, VPN, other SRA technology, or on-site assistance from another Xona administrator.
4. Failed Upgrade
   1. Risk: Very rarely do upgrades of Xona fail, but in the event that one does fail on your HA Primary CSG, then remote access to the site will be lost.
   2. Mitigation: If you have an out-of-band management network or support from an on-site Xona administrator available to help, then work to promote the HA Replica to an HA Primary. After the HA role has been switched and connectivity restored, please contact support@xonasystems.com to escalate a support case for the failed upgrade on your HA CSG. Postpone the upgrade process on the other HA CSG until further direction from the Xona Systems support team.

The Update Procedure

Steps necessary to update the Xona version of an HA pair that is joined to Xona Fabric.

1. Access the web interface for the XCM as a user with Admin permissions.

2. At the XCM, upload the update file for staging.

3. Access the web interface for CSG-A1 (current HA Primary) as a user with Admin permissions.

4. At CSG-A1  navigate to the Settings > Appliance > High Availability page, then click on Disable as Primary to disable HA replication.

   1. CSG-A2 (current HA Replica) remains in Replica state and doesn’t attempt to independently join the Xona fabric.

5. At the XCM, select the target version for CSG-A to stage the update file, wait for the update to upload, begin the installation, and wait until installation is completed. 

   1. The XCM will show the CSG status as Disconnected while the update is being installed. 

6. At CSG-A1, login to verify the upgrade was successful.

7. Access the web interface for CSG-A2 (current HA Replica) as a user with Admin permissions.

8. At CSG-A2, navigate to Settings > Appliance > High Availability page, click on Unlink HA Replica Pair to have it become the HA Primary.

   1. Immediately move to step 9! A reboot of CSG-A2 will be be initiated and it is important to complete step 9 while the reboot is occurring, otherwise the Xona Fabric connection will become unstable.
   2. CSG-A2 will become the HA Primary and connect to the Xona Fabric like CSG-A1 was before HA replication had been disabled.

9. At CSG-A1, navigate to Settings > Fabric page, click on Leave Fabric, and confirm.

   1. DO NOT remove CSG-A from the XCM.

10. At the XCM, select the target version for CSG-A to stage the update file, wait for the update to upload, begin the installation, and wait until installation is completed (again).

    1. The XCM will show the CSG status as Disconnected while the update is being installed. 

11. At CSG-A2, login to verify the upgrade succeeded and no unexpected errors appear in the audit log.

12. At CSG-A2, navigate to Settings > Appliance > High Availability page, configure it to become the HA Primary.
13. At CSG-A1, navigate to Settings > Appliance > High Availability page, configure it to become the HA Replica of CSG-A2.

14. Verify the HA replication status on CSG-A1 and CSG-A2.

15. This is the end of The Update Procedure. If you require CSG-A1 to be the HA Primary and CSG-A2 to be the HA Replica, then move on to The Switch Procedure.

The Switch Procedure

Steps necessary to switch the roles of an HA pair that is joined to Xona Fabric.

1. At CSG-A2 (current HA Primary), navigate to the Settings > Appliance > High Availability page, then click on Disable as Primary to disable HA replication.

   1. CSG-A1 (current HA Replica) remains in Replica state and doesn’t attempt to independently join the Xona fabric.

2. At CSG-A1, navigate to Settings > Appliance > High Availability page, click on Unlink HA Replica Pair to have it become the HA Primary.

   1. Immediately move to step 3! A reboot of CSG-A1 will be be initiated and it is important to complete step 9 while the reboot is occurring, otherwise the Xona Fabric connection will become unstable.

   2. CSG-A1 will become the HA Primary and connect to the Xona Fabric like CSG-A2did before HA replication had been disabled.

3. At CSG-A2, navigate to Settings > Fabric page, click on Leave Fabric, and confirm.

   1. DO NOT remove CSG-A from the XCM.
4. At CSG-A1, navigate to Settings > Appliance > High Availability page, configure it to become the HA Primary.
5. At CSG-A2, navigate to Settings > Appliance > High Availability page, configure it to become the HA Replica of CSG-A1.

6. Verify the HA replication status on CSG-A1 and CSG-A2.

7. This is the end of The Switch Procedure.

Support

If you have questions about this procedure or encounter problems during the upgrade process, then please reach out to support@xonasystems.com.