Firewall Failover
Incident Report for PGA of America
Postmortem

The secondary HA partner (named Lead) for the security appliances that service the Peak 10 data center in Kentucky was acting sporadically when the primary (named Tenor) crashed due to a overload condition.

To resolve the issue, we rebooted Lead and force started the virtual server that lives on the firewalls from Tenor on the box level. This was accomplished using the NG Firewall Console accessing the box from HQ at both the local and public IP for the firewall as the outage caused the VPN tunnel to HQ to act sporadically. Barracuda network support was engaged for assistance and guided the troubleshooting to the virtual server being offline as a first step. If that is down the system will not work properly. After getting everything online, we shut down Lead because or it’s sporadic nature. The datacenter is running on a single firewall at this time.

Posted Jul 25, 2019 - 07:41 EDT

Resolved
This incident has been resolved. All systems are reporting as functional at this time.
Posted Jul 17, 2019 - 15:01 EDT
Update
Repairs for this issue are ongoing. System connectivity continues to be affected at this time. The next update will be posted by 3:00 PM EST.
Posted Jul 17, 2019 - 14:00 EDT
Update
We are continuing to work on a fix for this issue. The next update will be posted by 2:00 PM EST.
Posted Jul 17, 2019 - 13:30 EDT
Identified
A firewall issue has been reported that affects connectivity to several applications including SMS/CCMS and VPN access. The cause has been identified and updates are in progress.
Posted Jul 17, 2019 - 12:58 EDT
This incident affected: CCMS/SMS, APIs (API - CF+JS), and Member Tools (PGA.org).