Network Status

19th Aug 2011 20:10

Network Connectivity Issue.

Today, at 13:01 monitoring detected network connectivity degradation (& failure) across all servers. This resulted in all servers becoming unresponsive to external requests.

Engineers attended to the issue and determined that a Denial of Service attack was occuring from with in the Data centre's own network, however the attack was not directed at any locally hosted servers, but congested the network with abnormally high levels of traffic, causing intermittent packet loss and unavailability of all requests.

Once the origin of the traffic was tracked down and blocked, normal service for all servers was restored at 13:34.

The offending servers (2 of them - not part of Oxigeno Online's own network), were quarrantined and investigated. The evidence collected showed that known vulnerabilities of out of date applications were exploited and used to launch the attack. The Network provider has contacted the website owners and remedial work is being carried out. Monitoring will continue over the weekend to ensure a repeat doesn't reoccur.

Affected Systems: All

Downtime: approx. 30 mins

We apologise for the inconvienance this interruption to services has caused and are working to ensure that a similar situation does not reoccur.

(Please feel free to follow @oxigenoonline on twitter for regular updates)

18 May 2011 8:22

2 server outages were experienced overnight.

Load Balanced Servers - At approx. 2:15am one node that hosts half of the load balanced web servers became unresponsive. The issue has been identified as a bug that has been fixed in a kernel patch that has yet to be applied.

A work around has been implemented until the patch can be applied - planned for the next maintenance window - date to be confirmed. The servers were rebooted and were brought back online at approx. 3:50am.

This outage did not affect services to the sites as other other load balanced web servers were uninterrupted.

Affected Systems: Half of the Load Balanced web servers - No sites affected

Downtime: none

Production Hosting Servers

At approx. 4:02am a similar issue was experienced on the production hosting servers. The same measures were taken to deal with the issue and services were returned at approx. 5:04am. This outage affected all sites hosted on the production servers and all production vps's excluding the load balanced environment - resulting in them becoming unreachable. 

Affected Systems: All production hosting websites and vps's

Downtime: approx. 1 hour and 2 mins

We apologise for the inconvienance this interruption to services has caused and are working to ensure that a similar situation does not reoccur.

 


 

30 June 2010 9:17

SSH Warnings began earlier this morning @ ~ 1:35. Investigations identified an attack on the server. Remedial action was undertaken to restrict SSH access to a list of known IP Addresses only.

Affected Systems: Load Balancer

Downtime: None - system remained functional & operational


11 June 2010 14:25

An outage has just been restored following Auckland and potentially Nation wide network and DNS connectivity issue.

The outage was first detected at ~ 13:25 and connectivity was recovered ~ 14:10 today.

The Network problems source has been reported to be from an upstream provider - Orcon - and has been confirm to have been resolved.

Affected Systems: All


18 May 2010 13:01

The Oxigeno Online Load Balancer, that is responsible for spreading incoming traffic across the various dedicated Web servers that serve the Load Balanced Hosting environments experienced an outage which was caused by an incorrect configuration change, which resulted in the server becoming unresponsive.

This affected all the websites hosted within the Load Balanced environment.

Total downtime 13:01 -> 13:35

Additional processes have been implemented to ensure that this issue does not repeat itself.


6 May 2010

We undertook system level upgrades overnight which may have resulted in minor service unavailability (< 2mins). This services involved upgrading the software packages installed on the servers and in some cases results in a service being stopped as part of the upgrade process.

This work forms part of our ongoing security patching and software upgrading processes which occurs monthly.

Affected Systems: All

Oxigeno Online

Support Team Twitter Feed

@OnlineOxigeno

follow me
  • 04:10 - 30 October 2013
    Emergency maintenance will be performed on the Network overnight. No downtime is expected.
  • 02:10 - 30 October 2013
    Services have been restored. We will provide details once known. Appears was temporary issue experienced with Network traffic within DC.
  • 02:10 - 30 October 2013
    We are experiencing Network issues currently. We are investigating cause and will advise shortly
  • 08:04 - 20 April 2012
    We are seeing external monitoring alerts which have been confirmed to be false positives. This appears to be a Internet backbone issue only
  • 05:01 - 13 January 2012
    All systems are go. We are investigating the cause of the temporary inaccessibility. Total disruption ~ 20 mins. Affecting load balncd sites