We are receiving reports of network performance problems for traffic reaching us via LINX. It has been reported that there is a problem at LINX, so a fair amount of UK traffic is being affected.

Traffic should now be routed around the affected hardware.

Maintenance window: 2010-03-13 06:30 to 2010-03-13 18:30 GMT

The maintenance will involve the comprehensive testing of the building generators and will entail switching to the auxiliary generator supplies for a period of several hours.

Testing and maintenance will also be carried out on the HV transformers and associated HV and LV switchgear, as well as uninterruptable power supplies (UPSs) and breakers.

The tests will be undertaken with all facilities engineers on-site as well as representatives from all contractors responsible for the various subsystems at the LHC facility.

23:45 - The server weldon will be rebooted shortly for a file system check. The raid controller reported a cache issue after a drive failure at 23:10PM. The RAID array has been fully rebuilt, with no downtime. We would, however, like to check the file system to be on the safe side and will be taking the machine off-line shortly to perform a check.

23:58 Update - The server is now back on-line after a successful file system check.

00:21 Update - A fresh hot spare drive has been put in the system to restore full redundancy.

We shall be performing a number of updates to all of our servers this week. During the course of the updates, the control panel will be unavailable. All other services will continue to run normally.

This work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.

[08.45 AM GMT] The server 'gable' is currently offline. We are working to resolve an issue with this server. We are running a file system check on the machine and hope to have it back online within 15 minutes

Apologies for any inconvenience caused.

We shall be performing a number of updates this evening. During the course of the updates, the control panel will be unavailable. All other services will continue to run normally.

This work may also result in 5 minutes of downtime per server.

We are currently (11:20AM) experiencing intermittent inbound telephone line problems, so you may have difficulty contacting us by phone. Should you wish to contact us regarding an issue, please submit your name and number via our contact form and you will be called back straight away.

The telecoms provider is working to resolve this issue.

We apologise for any inconvenience caused.

Our telecoms provider has reported that this problem was caused by a configuration issue with a piece of equipment on the BT network that was causing intermittent call failures. The problem has been resolved.

Apologies if you have been cut off or had trouble calling in the last couple of hours.

A small number of customers have reported problems sending emails when running their own MS Exchange servers, which then relay mail via our severs using an SMTP connector. This issue won't affect you if you are not using this set up (which isn't common).

The cause of the problem has been identified and the bug reported to the software vendor in question. Please contact us for a temporary work around if you think you are experiencing this problem.

15:00: We are working to resolve an issue with the server gable. We are running a file system check on the machine and hope to have it back online within 10 minutes

UPDATE 15:05 - This issue has now been resolved.

We shall be performing a number of updates to all of our servers today. During the course of the updates, the control panel will be unavailable. All other services will continue to run normally.

This work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.

We shall be performing a number of updates to all of our servers this week. During the course of the updates, the control panel will be unavailable. All other services will continue to run normally.

This work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.

Maintenance window: 2009-03-07 0830 to 2009-03-07 1800 GMT

As part of ongoing preventative maintenance work at the London Hosting Centre, UPS maintenance work has been scheduled as per above. During this maintenance work, the redundant UPS systems will be isolated and put into bypass mode. The actual time in bypass will be around 5 hours.

There is no planned interruption to service, however during this period, any disruption of the utility power supply may cause knock-on issues, as redundancy on the UPS system will be impacted and we will be running on raw mains. EDF are aware of the intended works and have confirmed that they have no service affecting maintenance themselves.

Maintenance window: 2009-02-14 00:01 to 2009-02-14 05:00 GMT

This planned maintenance window is to implement the second stage of core network upgrades. The backup router (Juniper M7i) will be switched for a Juniper MX240. Firmware upgrades will also be performed on the core switching equipment.

Clients may experience up to 5 minutes of packet loss and high latency.

During this window there will be reduced levels of redundancy within the core network.

Maintenance Window: 2008-12-12 23:00 to 2008-12-13 04:00 GMT

This planned maintenance window is to implement the first stage of core network upgrades. The existing core Juniper M7i routers will be swapped for Juniper MX240s.

Each of the two core routers will be swapped out one at a time. As a router is removed from the core, its counterpart will take over primary routing and gateway functions.

Customers may experience short periods of high latency or packet loss as traffic is diverted between routers during the maintenance.

This morning at 08:30 we briefly lost utility power twice at the London Hosting Centre facility. Our UPS and backup diesel generator systems worked as intended to ensure that there was no loss in service for our customers.

As the utility power is likely to be very unreliable over the next 24 hours, the decision has been made to switch the entire LHC facility over to stand-by generator power in the next few minutes, and remain as such until the all clear has been given by EDF.

Update: EDF have completed their remedial works and we have successfully switched to UPS backed utility power.

This issue was resolved without affecting any customer services.

[04.44 AM GMT] The server 'gable' is currently offline. We are running tests on the RAM and expect to be completed within 30 minutes. Apologies for the inconvenience caused.

We shall be performing a number of updates to a number of our servers this evening. During the course of the updates, the control panel will be unavailable. All other services will continue to run normally.

This work may also result in 5 minutes of downtime per server. However, this will be performed outside of normal working hours.

A few customers are unable to access their websites at present. We are currently investigating this and will attempt to rectify it as soon as possible. We would like to express our apologies for the inconvenience caused.

We are performing a file system check on the server duple, which should be online within 15 minutes.

UPDATE: Completed and rebooting.

Please note that the intention of this entry is to alert customers to a potential issue. The issue may well be resolved without any customer affecting problems, however in these circumstances, we issue a notification that redundancy is impacted as soon as possible so customers are kept up to date.

Earlier this evening, the power infrastructure at the data centre detected that the utility supply was outside of the acceptable tolerances, so automatically started up the diesel backup generators. The UPS systems on each floor of the LHC briefly went to battery power to assume the load whilst the backup generators started.

The redundant UPS equipment on the 2nd floor detected an internal failure, and as such did not allow incoming power to be transferred from the generators to the mains power. At the moment, the UPS systems are in "bypass" mode, with power being supplied by the standby diesel generators.

Update 21:45: It has now been confirmed that we are back on mains high voltage utility power, but the UPS systems are still in bypass mode. Engineers are analysing the data from the management controllers and are deciding on a course of action. The engineers hope to restore battery backup protection shortly.

Update 26th June: Replacement parts for the two faulty units (filters and choke) have been installed and the change over to full UPS protection for the 2nd floor of the LHC went ahead without incident.

Engineers are still carefully monitoring the situation, however we now consider this issue closed unless we issue a further update.

This issue was resolved without affecting any customer services.