Slow I/O Performance
Update - There has been a period of IO stalls this morning as we've dealt with some storage hardware failures. That issue is now resolved, however space reclamation continues in the backend and is having a detrimental impact on read performance. We are working with WEKA support to look at mitigation options. Apologies for the performance impact - if your jobs are impacted and need a runtime limit extension then please reach out to support.
Nov 24, 2025 - 12:44 NZDT
Identified - Our storage system is currently very full and this is forcing the backend object storage to undertake some urgent administration by way of defragmentation. This increased load is having a detrimental affect on I/O performance, especially read I/O, and this is likely to continue for some days. We are urgently looking at ways to mitigate this. Researchers could help alleviate this in the short term by cleaning up any unwanted files and data as soon as possible please.
Nov 21, 2025 - 12:29 NZDT

About This Site

This page shares the system status of REANNZ's advanced computing platform and storage services.
To view the status of REANNZ's network services, visit: https://reannz.status.io

Uptime over the past 90 days. View historical uptime.
Apply for Access Operational
Data Transfer Operational
Submit new HPC Jobs Operational
Jobs running on HPC Operational
NeSI OnDemand Operational
90 days ago
99.83 % uptime
Today
HPC Storage Degraded Performance
User Support System Operational
Support Documentation Operational
Flexible High Performance Cloud Operational
Long-term Storage (Freezer) Operational
90 days ago
99.97 % uptime
Today
Flexible High Performance Cloud Services Operational
90 days ago
99.99 % uptime
Today
Virtual Compute Service Operational
Bare Metal Compute Service Operational
FlexiHPC Dashboard (web interface) Operational
90 days ago
100.0 % uptime
Today
FlexiHPC CLI interface Operational
90 days ago
100.0 % uptime
Today
Public API of the FlexiHPC Service Operational
90 days ago
99.99 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Scheduled Maintenance

Automatic cleaning launched for scratch filesystem (/nesi/nobackup) - Announcement Nov 20, 2025 09:00 - Nov 21, 2025 09:00 NZDT

Update - The scratch autocleaner is now underway, doing its first round of file deletions. The next deletion run will start on Dec 17th,, once that is completed we will suspend the autocleaner over the Christmas holiday period. Then it will start again with a scan on Jan 14th, followed by the file deletions two weeks later. From there it will run every two weeks. If you have any issues please contact support @nesi.org.nz.
Dec 05, 2025 - 10:43 NZDT
Scheduled - As announced earlier, we’ve re-activated an automatic cleaning process for temporary data stored on our scratch filesystem (/nesi/nobackup). Project members with files scheduled for deletion were notified by email (Subject to their my.nesi.org.nz Notification preferences. To confirm or adjust what notifications you receive, follow these instructions: https://docs.nesi.org.nz/Getting_Started/my-nesi-org-nz/Managing_notification_preferences/)
Files will be deleted on Wednesday 03 December (two weeks from yesterday's email notification). For more details on how the auto-deletion process works, visit:
https://docs.nesi.org.nz/Storage/File_Systems_and_Quotas/Automatic_cleaning_of_nobackup_file_system/
If you have files identified as candidates for deletion that you need to keep beyond the scheduled expiry date, you can move them to your project directory or to Freezer, our long-term storage service. However, if you plan to move more than 2 TB of data or if you need to increase your project directory quota, email support@nesi.org.nz so that we can discuss your storage needs and assist you.
Nov 20, 2025 - 08:44 NZDT
Dec 16, 2025
Freezer authentication failing
Resolved - This incident has been resolved.
Dec 16, 08:44 NZDT
Investigating - Freezer sessions are unable to authenticate, we are actively working to bring it online.
Dec 16, 08:15 NZDT
Cumulus switch upgrades
Completed - The upgrade went smoothly without causing any disruption till about 12:25am. At this point there was a ~20 min period of breakage in the internal network. Brief disruptions occurred several more times throughout the night as config issues were worked through. We now have redundant connectivity up again and one of two switches upgraded. The maintenance will be completed this evening.
Dec 16, 08:43 NZDT
Update - Initial controlled shutdown of the first switch went smoothly thanks to configuration fixes made after the last aborted attempt. The network OS update was also applied successfully.
However, bring-up of redundant services on the first upgraded and reconfigured switch is causing some unforeseen issues. This has resulted in several short periods of external and internal connectivity loss that may have adversely affected some services.
Vendor support engineers are working to investigate and resolve these issues. A further update will be given before 9am.
Dec 16, 02:42 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 15, 21:00 NZDT
Update - We will be undergoing scheduled maintenance during this time.
Dec 15, 16:30 NZDT
Scheduled - The border switches upgrade has been rescheduled to Mon Dec 15th from 9pm. Any ssh and external connections to the cluster and storage may get broken during this maintenance. Slurm jobs will be unaffected. The login nodes may experience some short term disruption.
Dec 15, 09:50 NZDT
Dec 15, 2025
Major incident - Weka Filesystem Not Available
Resolved - This incident has been resolved.
Dec 15, 16:29 NZDT
Monitoring - WEKA support have managed to restore service and the original storage cluster version upgrade is now continuing in the background. Access to Mahuika/HPC3 services is now restored and most running jobs appear to have survived the storage outage. Some jobs may have completed in a failed state in Slurm - users should review outputs before rerunning failed jobs.

We expect continued intermittent performance issues while the upgrade completes and are monitoring this closely.

Apologies for any disruption this caused to your work this afternoon!
Dec 9, 16:58 NZDT
Update - The storage system is slowly being recovered. There may be some filesystem access available but performance will be very degraded. We hope to be fully resolved in about an hour, but it could take longer
Dec 9, 16:34 NZDT
Update - We are continuing to work with our storage vendors to recover the filesystems. But we do not yet have an ETA.
Dec 9, 14:51 NZDT
Identified - Vendor support is actively engaged and working on this now.

We expect all IO will be hanging at the moment, so access to the systems and OnDemand will also be impacted. Currently running jobs will likely block when attempting IO and may continue once service is restored.
Dec 9, 13:42 NZDT
Investigating - During an upgrade of the shared filesystem we have encountered an issue. We are working on fixing this issue and will post updates here with an ETA ASAP
Dec 9, 13:35 NZDT
my.nesi.org.nz system update
Completed - The scheduled maintenance has been completed.
Dec 15, 13:16 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 15, 13:00 NZDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Dec 15, 12:10 NZDT
REANNZ Globus nodes maintenance
Completed - The scheduled maintenance has been completed.
Dec 15, 13:12 NZDT
Verifying - Verification is currently underway for the maintenance items.
Dec 15, 13:11 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 15, 11:00 NZDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Dec 12, 08:47 NZDT
Dec 14, 2025

No incidents reported.

Dec 13, 2025

No incidents reported.

Dec 12, 2025

No incidents reported.

Dec 11, 2025
my.nesi.org.nz system update
Completed - The scheduled maintenance has been completed.
Dec 11, 16:24 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 11, 15:30 NZDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Dec 5, 12:15 NZDT
Dec 10, 2025
Problem submitting HPC jobs
Resolved - Jobs have not been launching for the past 12 hours. That problem is now fixed.
Dec 10, 10:46 NZDT
Monitoring - We have identified the problem and Slurm jobs are now starting correctly
Dec 10, 10:45 NZDT
Update - We are continuing to investigate this issue.
Dec 10, 10:34 NZDT
Update - We are continuing to investigate this issue.
Dec 10, 10:33 NZDT
Update - Jobs can be submitted but will not start
Dec 10, 10:32 NZDT
Investigating - We are currently investigating this issue.
Dec 10, 10:11 NZDT
Cumulus switch upgrades
Completed - We've identified the reason behind the lack of failover and restored services. A further preparatory change is needed before we can complete the original upgrade, so this will be rescheduled.
Dec 10, 01:07 NZDT
Update - We've encountered an unexpected lack of failover behaviour in our software-defined networking stack during the first stage of this upgrade while attempting an "orderly" failover and shutdown. There is currently intermittent connectivity to/from services and the Internet while we debug further and assess whether the change can continue.
Dec 9, 22:30 NZDT
Update - Scheduled maintenance is still in progress. We will provide updates as necessary.
Dec 9, 22:09 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 9, 18:00 NZDT
Scheduled - The border switches maintenance has been rescheduled to Dec 9th from 1800hrs. Any ssh and external connections to the cluster and OnDemand may get broken during this maintenance. Slurm jobs will be unaffected.
Dec 3, 13:56 NZDT
Dec 9, 2025
WEKA upgrade
Completed - The scheduled maintenance has been completed.
Dec 9, 22:08 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 9, 12:00 NZDT
Scheduled - We will be performing a live upgrade of the backend WEKA storage cluster to the latest LTS version. This will resolve some known issues and give us greater insight to address current issues with performance consistency / intermittent slow reads. The system is expected to remain online through the upgrade, though there may be some reduction in performance at various times as nodes are rebooted.
Dec 8, 11:45 NZDT
WEKA filesystem and compute core changes - Announcement
Completed - The scheduled maintenance has been completed.
Dec 9, 22:07 NZDT
Scheduled - In order to get our new WEKA filesystems up to their best possible performance we need to dedicate some cores on each compute node for exclusive use by WEKA. So on Milan nodes Slurm jobs will only be able to use 126 cores per node rather than 128, and on Genoa nodes 166 rather than 168.
This change has already begun, having already been applied to all of the Milan nodes and the majority of the Genoa nodes. We expect to be doing the last of the Genoa nodes on September 8th.
Sep 4, 12:10 NZST
Dec 8, 2025

No incidents reported.

Dec 7, 2025

No incidents reported.

Dec 6, 2025

No incidents reported.

Dec 5, 2025

No incidents reported.

Dec 4, 2025
my.nesi.org.nz system update
Completed - The scheduled maintenance has been completed.
Dec 4, 17:30 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 4, 15:30 NZDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Dec 2, 14:53 NZDT
Dec 3, 2025
Cumulus switch upgrades
Completed - This maintenance was postponed further, the status updates posted since yesterday are erroneous. Apologies for the confusion. We will make a new post for the rescheduled dates.
Dec 3, 13:51 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 2, 18:00 NZDT
Update - This maintenance has now been postponed to Dec 2nd at 1800hrs
Nov 18, 08:50 NZDT
Scheduled - The border switches will have an upgrade applied overnight on Nov 18th from 6pm. Any ssh and external connections to the cluster and OnDemand may get broken during this maintenance. Slurm jobs will be unaffected.
Nov 7, 11:13 NZDT
Dec 2, 2025
Freezer access (to s3 buckets) impacted by known software issue
Resolved - This incident has been resolved.
Dec 2, 14:53 NZDT
Monitoring - A software bug is causing frequent restarts of the underlying infrastructure behind Freezer. An upgrade to resolve this is planned for Monday morning.
Nov 28, 13:27 NZDT
Investigating - Freezer was down briefly from Friday 2025/11/28 12:40PM until Friday 2025/11/28 12:46PM. Please check if your transfers have been interrupted. For more information see https://docs.nesi.org.nz/Storage/Long_Term_Storage/Freezer_Guide/#synchronise-data
Nov 28, 13:26 NZDT
← Incident History Powered by Atlassian Statuspage