System Status Report
This page allows our customers to keep informed about any issues currently affecting our network.
If an issue is listed here, you can rest assured that we are working to resolve it. If you have a problem not listed here please contact us, as we may not be aware of the issue, or it may only affect your site.
You may want to bookmark this page for future reference.
21 Jul 2010
Networking IssueTraffic should now be routed around the affected hardware.
26 Feb 2010
LHC Data Centre Annual MaintenanceThe maintenance will involve the comprehensive testing of the building generators and will entail switching to the auxiliary generator supplies for a period of several hours.
Testing and maintenance will also be carried out on the HV transformers and associated HV and LV switchgear, as well as uninterruptable power supplies (UPSs) and breakers.
The tests will be undertaken with all facilities engineers on-site as well as representatives from all contractors responsible for the various subsystems at the LHC facility.
9 Feb 2010
File System Check23:58 Update - The server is now back on-line after a successful file system check.
00:21 Update - A fresh hot spare drive has been put in the system to restore full redundancy.
3 Feb 2010
Server MaintenanceThis work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.
29 Sep 2009
Website Access ProblemsApologies for any inconvenience caused.
26 Aug 2009
Server MaintenanceThis work may also result in 5 minutes of downtime per server.
11 Aug 2009
Intermittent Telephone ProblemsThe telecoms provider is working to resolve this issue.
We apologise for any inconvenience caused.
Our telecoms provider has reported that this problem was caused by a configuration issue with a piece of equipment on the BT network that was causing intermittent call failures. The problem has been resolved.
Apologies if you have been cut off or had trouble calling in the last couple of hours.
29 Apr 2009
Exchange SMTP Connector IssuesThe cause of the problem has been identified and the bug reported to the software vendor in question. Please contact us for a temporary work around if you think you are experiencing this problem.
24 Apr 2009
Service OutageUPDATE 15:05 - This issue has now been resolved.
13 Apr 2009
Server MaintenanceThis work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.
5 Mar 2009
Server MaintenanceThis work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.
2 Mar 2009
Planned Maintenance WindowAs part of ongoing preventative maintenance work at the London Hosting Centre, UPS maintenance work has been scheduled as per above. During this maintenance work, the redundant UPS systems will be isolated and put into bypass mode. The actual time in bypass will be around 5 hours.
There is no planned interruption to service, however during this period, any disruption of the utility power supply may cause knock-on issues, as redundancy on the UPS system will be impacted and we will be running on raw mains. EDF are aware of the intended works and have confirmed that they have no service affecting maintenance themselves.
6 Feb 2009
Planned Maintenance WindowThis planned maintenance window is to implement the second stage of core network upgrades. The backup router (Juniper M7i) will be switched for a Juniper MX240. Firmware upgrades will also be performed on the core switching equipment.
Clients may experience up to 5 minutes of packet loss and high latency.
During this window there will be reduced levels of redundancy within the core network.
2 Dec 2008
Planned Maintenance WindowThis planned maintenance window is to implement the first stage of core network upgrades. The existing core Juniper M7i routers will be swapped for Juniper MX240s.
Each of the two core routers will be swapped out one at a time. As a router is removed from the core, its counterpart will take over primary routing and gateway functions.
Customers may experience short periods of high latency or packet loss as traffic is diverted between routers during the maintenance.
4 Nov 2008
EDF HV Power Grid Maintenance WorkAs the utility power is likely to be very unreliable over the next 24 hours, the decision has been made to switch the entire LHC facility over to stand-by generator power in the next few minutes, and remain as such until the all clear has been given by EDF.
Update: EDF have completed their remedial works and we have successfully switched to UPS backed utility power.
This issue was resolved without affecting any customer services.
3 Sep 2008
Website Access Problems19 Aug 2008
Server MaintenanceThis work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.
25 Jul 2008
Website Access ProblemsWe are performing a file system check on the server duple, which should be online within 15 minutes.
UPDATE: Completed and rebooting.
25 Jun 2008
Uninterruptible Power Supply IssueEarlier this evening, the power infrastructure at the data centre detected that the utility supply was outside of the acceptable tolerances, so automatically started up the diesel backup generators. The UPS systems on each floor of the LHC briefly went to battery power to assume the load whilst the backup generators started.
The redundant UPS equipment on the 2nd floor detected an internal failure, and as such did not allow incoming power to be transferred from the generators to the mains power. At the moment, the UPS systems are in "bypass" mode, with power being supplied by the standby diesel generators.
Update 21:45: It has now been confirmed that we are back on mains high voltage utility power, but the UPS systems are still in bypass mode. Engineers are analysing the data from the management controllers and are deciding on a course of action. The engineers hope to restore battery backup protection shortly.
Update 26th June: Replacement parts for the two faulty units (filters and choke) have been installed and the change over to full UPS protection for the 2nd floor of the LHC went ahead without incident.
Engineers are still carefully monitoring the situation, however we now consider this issue closed unless we issue a further update.
This issue was resolved without affecting any customer services.