AWS Outage

Incident Report for resero CampusCloud & SmartLockers

Resolved

This incident has been resolved.
Posted Jan 13, 2026 - 08:44 EST

Update

We continue to see strong performance of the updated database infrastructure today; we will continue to monitor for any anomalies.
Posted Jan 09, 2026 - 14:14 EST

Update

Overnight, our team took additional steps to re-configure the underlying AWS database hosting platform to increase total capacity and better tune the environment to the real-world data we collected. Initial metrics indicate a tenfold reduction of resource usage at rest and a marked reduction at load; we will continue to monitor all processes and services today to gain additional insights into the overall resource use, but are confident that these changes will provide better short- and long-term stability to the hosting environment.
Posted Jan 09, 2026 - 08:42 EST

Update

We continue to monitor, all systems appear to be functioning nominally at this time.
Posted Jan 08, 2026 - 16:18 EST

Update

We are continuing to monitor the situation.
Posted Jan 08, 2026 - 15:25 EST

Update

We are reviewing abnormal activity that may be affecting system performance.
Posted Jan 08, 2026 - 15:07 EST

Update

We are currently monitoring a spike in usage and will update infrastructure as needed.
Posted Jan 08, 2026 - 12:52 EST

Update

As we continue to monitor today, all systems are functioning nominally; we will continue to monitor throughtout the day.
Posted Jan 08, 2026 - 11:24 EST

Update

As we continue to monitor this morning, all systems are functioning nominally; we will continue to monitor throughtout the day.
Posted Jan 08, 2026 - 09:13 EST

Update

Initial testing of the hot fix in the Production environment appears nominal; we will continue to monitor overnight and throughout the day tomorrow. Mitigation plans are in place if concerns should arise tomorrow.
Posted Jan 08, 2026 - 01:22 EST

Update

The hotfix has been applied and appears to have resolved the concern; we will continue to monitor overnight and throughout the day tomorrow.
Posted Jan 08, 2026 - 00:49 EST

Update

We will be making a no-down-time hotfix within the next hour.
Posted Jan 07, 2026 - 23:33 EST

Update

We are currently testing through an update in our Development environment, if successful we will release the update to the Production system as a no-down-time hotfix.
Posted Jan 07, 2026 - 21:43 EST

Update

We are continuing to monitor the status of all services and work towards a permanent resolution.
Posted Jan 07, 2026 - 21:19 EST

Update

We are continuing to work to mitigate any ongoing sluggishness and other concerns.
Posted Jan 07, 2026 - 20:10 EST

Update

We have now located an offending piece of code that as a result of the failover process was demanding additional resources; this process has been manually terminated and we will now begin to work on a permanent resolution to prevent the code from running again. Services should now return to close to normal operation.
Posted Jan 07, 2026 - 18:39 EST

Update

We are continuing to monitor and adjust the database configuration; at this time, processes may have delayed completion, but will complete.
Posted Jan 07, 2026 - 17:58 EST

Update

We are seeing services continue to return to normal.
Posted Jan 07, 2026 - 15:32 EST

Update

We continue to monitor and shift infrastructure as needed; please ensure that you log out and back in to CampusCloud. As services come back online there may be slower processing times.
Posted Jan 07, 2026 - 15:22 EST

Monitoring

Please log out and back in to CampusCloud.
Posted Jan 07, 2026 - 14:53 EST

Update

The failover is complete, and all services are re-connecting now.
Posted Jan 07, 2026 - 14:52 EST

Update

We are again going to failover the database to another location.
Posted Jan 07, 2026 - 14:40 EST

Update

We continue to monitor the AWS database environment for high load and will make adjustments as needed; we continue to see improvement in processes overall.
Posted Jan 07, 2026 - 14:31 EST

Update

We have taken additional failover mitigation steps and are seeing services return to normal operation; we will continue to investigate and monitor the situation.
Posted Jan 07, 2026 - 14:11 EST

Update

We are continuing to investigate.
Posted Jan 07, 2026 - 14:05 EST

Update

Currently it appears that the AWS outage may be caused by an internet service provider backbone limiting the scope to certain stores; we are seeing successful transactions process. We are continuing to investigate, but all resero-controlled systems appear fully functional.
Posted Jan 07, 2026 - 13:41 EST

Investigating

We are seeing additional concerns from AWS and are investigating now.
Posted Jan 07, 2026 - 13:31 EST

Monitoring

The database failover has been completed and we are monitoring for additional concerns.
Posted Jan 07, 2026 - 12:45 EST

Identified

Our AWS database hosting environment is experiencing an outage; we are currently migrating to the backup environment.
Posted Jan 07, 2026 - 12:32 EST
This incident affected: CampusCloud Core Services (eCommerce Platform, AWS West Coast Environment, AWS East Coast Environment).