Incident Notification: [Tuesday, April 19 - Wednesday, April 20, 2022] Hoylu Systems Down - Critical Impact

[Resolved] Hoylu Incident Alert[Wednesday, April 20, 2022, 3:49 a.m. (ET)] Hoylu Systems Continue to be Down - Critical Impact

On Tuesday, April 19, 2022, at approximately 8:45 a.m. (ET)/2:45 p.m. (CET), our DevOps systems experienced an unexpected issue causing a systemwide outage.

In working closely with our partners, throughout the day and night, we were able to resolve the issue at 3:49 a.m. (ET)/9:49 a.m. (CET) on Wednesday, April 20, 2022. The outage lasted just under 19 hours.

ALL systems are now up and working as expected.

During this outage, no user data was compromised, and all systems remained secure. Our teams are continuing to investigate to better understand the overall impact on client Hoylu Workspaces, but we do anticipate clients who were actively working in their Hoylu Workspaces at the start of the outage, may have experienced data loss.

Any open tickets regarding this issue will be responded to through our Customer Success team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 5:00 p.m. (ET)] Hoylu Systems Continue to be Down - Critical Impact

As of 5:00 p.m. (ET) our systems remain down and Engineers from both Hoylu and Azure are actively working on restoring our platforms.

This remains a critical issue and is being handled with the highest priority.

We no longer can guarantee the previously anticipated resolution of 7:30 p.m. (ET). Our teams continue to work collaboratively with Azure and we still anticipate a final resolution today (Tuesday, April 19, 2022).

We will provide updates on our System Status page as they become available. Our next updated communication will be after an implemented resolution or at 8:30 a.m. (ET) tomorrow – which ever comes first.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 3:30 p.m. (ET)] Hoylu Systems Continue to be Down - Critical Impact

As of 3:30 p.m. (ET), Hoylu systems remain down. Our Engineering team continues to work with our Azure team to resolve the issue.

This is a critical issue and is being handled with the highest priority.

We still anticipate our systems to be back to a normal state by 7:30 p.m. (ET).

Our next status update will be at 5 p.m. (ET) via our System Status page.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 2:00 p.m. (ET)] Hoylu Systems are Down - Critical Impact

Systems remain down. Our Engineering team continues to work with our Azure team to resolve the issue.

Work is underway and expected resolution and validation testing on the fix will take approximately 4-8 hours. We still anticipate our systems to be back to normal state by 7:30 p.m. (ET).

We will provide updates on our System Status page as they become available.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 12:30 p.m. (ET)] Hoylu Systems are Down - Critical Impact

Our Engineering team continues to work with our Azure team to resolve the issue.

Work is underway and expected resolution and validation testing on the fix will take approximately 4-8 hours. We anticipate our systems back up to normal state by 7:30 p.m. (ET).

We will provide updates on our System Status page as they become available.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 10:51 a.m. (ET)] Hoylu Systems are Down - Critical Impact

Our Engineering team continues to work with our Azure team to resolve the issue.

It is expected the issue will take approximately 4-8 hours to complete and bring our systems to a normal state. Currently, all systems remain down.

We will provide updates on our System Status page as they become available.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 10:21 a.m. (ET)] Hoylu Systems are Down - Critical Impact

Our Engineering team continues to work with our Azure team to resolve the issue.

We are continuing to work on the resolution and an estimated time for systems to return to normal state. We will provide updated on our System Status page as they become available.

All systems remain down.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert [Update] [Tuesday, April 19, 2022, 09:41 a.m. (ET)] Hoylu Systems are Down - Critical Impact

We have identified the root cause of the outage and determining the estimated time to resolution.

Currently, ALL systems remain down. 

We are continuing to investigate the situation and determine a root cause. We will provide updates on our System Status page as they become available.

Any open tickets regarding this issue will be responded to through our Customer Success Team.

 

Incident Alert Notification [Tuesday, April 19, 2022, 09:01 a.m. (ET)] Hoylu Systems are Down - Critical Impact

We are aware of an issue with our Hoylu production environments. Currently, ALL Systems are down. 

We are continuing to investigate the situation and determine a root cause. We will provide updates on our System Status page as they become available.


Any open tickets regarding this issue will be responded to through our Customer Success Team.