Resolved -
This incident has been resolved.
Jan 13, 08:44 EST
Update -
We continue to see strong performance of the updated database infrastructure today; we will continue to monitor for any anomalies.
Jan 9, 14:14 EST
Update -
Overnight, our team took additional steps to re-configure the underlying AWS database hosting platform to increase total capacity and better tune the environment to the real-world data we collected. Initial metrics indicate a tenfold reduction of resource usage at rest and a marked reduction at load; we will continue to monitor all processes and services today to gain additional insights into the overall resource use, but are confident that these changes will provide better short- and long-term stability to the hosting environment.
Jan 9, 08:42 EST
Update -
We continue to monitor, all systems appear to be functioning nominally at this time.
Jan 8, 16:18 EST
Update -
We are continuing to monitor the situation.
Jan 8, 15:25 EST
Update -
We are reviewing abnormal activity that may be affecting system performance.
Jan 8, 15:07 EST
Update -
We are currently monitoring a spike in usage and will update infrastructure as needed.
Jan 8, 12:52 EST
Update -
As we continue to monitor today, all systems are functioning nominally; we will continue to monitor throughtout the day.
Jan 8, 11:24 EST
Update -
As we continue to monitor this morning, all systems are functioning nominally; we will continue to monitor throughtout the day.
Jan 8, 09:13 EST
Update -
Initial testing of the hot fix in the Production environment appears nominal; we will continue to monitor overnight and throughout the day tomorrow. Mitigation plans are in place if concerns should arise tomorrow.
Jan 8, 01:22 EST
Update -
The hotfix has been applied and appears to have resolved the concern; we will continue to monitor overnight and throughout the day tomorrow.
Jan 8, 00:49 EST
Update -
We will be making a no-down-time hotfix within the next hour.
Jan 7, 23:33 EST
Update -
We are currently testing through an update in our Development environment, if successful we will release the update to the Production system as a no-down-time hotfix.
Jan 7, 21:43 EST
Update -
We are continuing to monitor the status of all services and work towards a permanent resolution.
Jan 7, 21:19 EST
Update -
We are continuing to work to mitigate any ongoing sluggishness and other concerns.
Jan 7, 20:10 EST
Update -
We have now located an offending piece of code that as a result of the failover process was demanding additional resources; this process has been manually terminated and we will now begin to work on a permanent resolution to prevent the code from running again. Services should now return to close to normal operation.
Jan 7, 18:39 EST
Update -
We are continuing to monitor and adjust the database configuration; at this time, processes may have delayed completion, but will complete.
Jan 7, 17:58 EST
Update -
We are seeing services continue to return to normal.
Jan 7, 15:32 EST
Update -
We continue to monitor and shift infrastructure as needed; please ensure that you log out and back in to CampusCloud. As services come back online there may be slower processing times.
Jan 7, 15:22 EST
Monitoring -
Please log out and back in to CampusCloud.
Jan 7, 14:53 EST
Update -
The failover is complete, and all services are re-connecting now.
Jan 7, 14:52 EST
Update -
We are again going to failover the database to another location.
Jan 7, 14:40 EST
Update -
We continue to monitor the AWS database environment for high load and will make adjustments as needed; we continue to see improvement in processes overall.
Jan 7, 14:31 EST
Update -
We have taken additional failover mitigation steps and are seeing services return to normal operation; we will continue to investigate and monitor the situation.
Jan 7, 14:11 EST
Update -
We are continuing to investigate.
Jan 7, 14:05 EST
Update -
Currently it appears that the AWS outage may be caused by an internet service provider backbone limiting the scope to certain stores; we are seeing successful transactions process. We are continuing to investigate, but all resero-controlled systems appear fully functional.
Jan 7, 13:41 EST
Investigating -
We are seeing additional concerns from AWS and are investigating now.
Jan 7, 13:31 EST
Monitoring -
The database failover has been completed and we are monitoring for additional concerns.
Jan 7, 12:45 EST
Identified -
Our AWS database hosting environment is experiencing an outage; we are currently migrating to the backup environment.
Jan 7, 12:32 EST