Storage services -other than Freezer- are operational.
Recovery of login services is progressing well, however this work is ongoing. The compute nodes are not online yet, and batch jobs are not running.
We expect to provide an update at approximately 10:00am tomorrow.
Thank you for your patience thus far.
Posted Jul 30, 2026 - 17:05 NZST
Update
HPC storage services appear to be in good health, although final checks are ongoing.
In parallel, we are working to get login services up and running however, at this stage we do not expect to have compute nodes online today.
Freezer and tape-based storage is expected to remain offline for longer, while we perform drive and media consistency checks. We will provide further information on this as we know more.
We plan to provide another update at approximately 17:00 today.
Posted Jul 30, 2026 - 15:00 NZST
Update
We have powered on essential hardware to begin bringing systems online. We are actively working on ramping up services starting with HPC storage and validating the status of storage. We are bringing up the minimum number of virtual instances to manage the rest of the fleet once storage is online.
We are keeping a very close eye on the environmental metrics at the data centre to prevent any additional fluctuations that might impact our tape media.
We plan to provide another update at approximately 15:00 today.
Posted Jul 30, 2026 - 12:55 NZST
Update
Tamaki Data Centre (TDC) has confirmed that the cooling system is back online and they are performing environmental checks etc.
We need to do a slow and controlled resumption of business as usual and perform health checks on the systems as we go.
We plan to provide another update about the progress and status by 13:00 today.
Posted Jul 30, 2026 - 10:05 NZST
Identified
The current status is that we have the two working chillers at the data centre - TDC and a team is working on restoring the full chiller capacity.
Posted Jul 30, 2026 - 10:00 NZST
Update
Tamaki Data Centre has identified issues affecting two of the site's three chillers, resulting in reduced cooling capacity.
Teams are actively investigating the root cause and monitoring site temperatures and cooling performance. Our priority is the safe restoration of cooling services and maintaining the integrity of the platform.
Once temperatures have stabilised and cooling capacity has been validated, we will commence our recovery and startup plan. We do not plan to proceed with any start up until at least tomorrow morning. Further updates will be provided as the investigation progresses.
Posted Jul 29, 2026 - 17:16 NZST
Update
Cooling at the Tamaki Data Centre has not yet been restored. All services and hardware remains down. We currently don't have any ETA.
Posted Jul 29, 2026 - 16:25 NZST
Update
We are continuing to investigate this issue.
Posted Jul 29, 2026 - 15:10 NZST
Update
This incident has continued to escalate so we are shutting down the HPC to avoid hardware damage.
Posted Jul 29, 2026 - 14:57 NZST
Update
We are continuing to investigate this issue.
Posted Jul 29, 2026 - 13:24 NZST
Update
For jobs already running on impacted nodes, users will see some jobs ended early with state NODE_FAIL and then requeued.
New jobs can still be submitted to the queue, but they will not start.
Posted Jul 29, 2026 - 13:23 NZST
Update
We are stopping new jobs from starting while the investigation continues. Submitted jobs will remain in the queue and start once we have resolved the incident.
Posted Jul 29, 2026 - 13:21 NZST
Investigating
We have had some nodes go down due to a serious environmental (cooling) issue at the data centre. We are working to mitigate impacts.
Posted Jul 29, 2026 - 13:15 NZST
This incident affects: Apply for Access, Data Transfer, Submit new HPC Jobs, Jobs running on HPC, NeSI OnDemand, HPC Storage, User Support System, Flexible High Performance Cloud, Long-term Storage (Freezer), Login to Mahuika and Flexible High Performance Cloud Services (Virtual Compute Service, Bare Metal Compute Service, FlexiHPC Dashboard (web interface), FlexiHPC CLI interface, Public API of the FlexiHPC Service).