Linode Status - Incident History

http://status.linode.com/incidents/hpkfjbfjdfmc

Jan 6, 02:31 UTC
Identified - The issue has been identified and a fix is being implemented.

Jan 6, 02:11 UTC
Monitoring - We're continuing to work to restore access to the Linode Manager, Website and API.

Jan 5, 23:23 UTC
Identified - Further attacks have resumed on our Linode Manager, Website, and API infrastructure. Our engineers are taking steps to further protect our infrastructure and mitigate the attacks. Thank you once again for your patience.

Jan 5, 22:56 UTC
Monitoring - We believe the DoS attack has been mitigated, restoring access to the Linode Manager, Website, and API. We are monitoring for additional attacks at this time.

Jan 5, 21:53 UTC
Identified - We have identified a denial of service attack targeting the Linode Manager, Website and API which is resulting in inaccessibility for many users. Our Administrators are working to mitigate the attack and restore full functionality at this time. We thank you for your patience.

http://status.linode.com/incidents/kldhjpjnfnkj

Jan 6, 02:15 UTC
Resolved - There have been no further attacks, and the Linode Blog is accessible at this time.

Jan 5, 22:57 UTC
Monitoring - We believe the DoS attack targeting the Linode Blog has been mitigated and connectivity fully restored. We are monitoring for additional attacks at this time.

Jan 5, 19:56 UTC
Identified - We are currently experiencing denial of service attacks that are targeting our Blog.

http://status.linode.com/incidents/ghdlhfnfngnh

Jan 5, 19:44 UTC
In progress - Security Notification and Linode Manager Password Reset

Jan 5, 19:44 UTC
Scheduled - Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.

A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.

This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.

The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We've retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.

You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be.

The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.

We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode.

http://status.linode.com/incidents/npnpbplcjgkm

Jan 5, 08:12 UTC
Monitoring - Since the attacks on our name server infrastructure have not stopped, we are expecting to have to leave AXFR functionality disabled for the foreseeable future. We recommend customers who depend on this feature to explore short-term alternatives.

Jan 4, 20:04 UTC
Update - Our engineers are continuing to harden our DNS systems from the ongoing attacks. We will provide an update once additional progress has been made.

Jan 4, 18:36 UTC
Identified - The denial of service attacks against our DNS infrastructure have resumed at this time.

Jan 4, 10:39 UTC
Update - There have been no additional DNS attacks detected throughout the night. Our DNS systems remain operational with AXFR disabled. We will continue to closely monitor the situation.

Jan 4, 05:47 UTC
Update - Our staff is continuing to monitor for further attacks. At this time our DNS systems are continuing to function normally, AXFR remains disabled.

Jan 4, 02:14 UTC
Update - Our engineers have taken steps to further harden our DNS systems against further attacks. It should be noted, however, that AXFR functionality has been disabled for the moment due to these measures. We will resolve this once we are able to implement additional levels of mitigation. We are continuing to monitor for any further disruptions. Thank you for your patience.

Jan 4, 01:32 UTC
Monitoring - We have been able to mitigate DDoS's toward a quorum of nameservers, restoring DNS service. We will monitor for further attacks and update accordingly.

Jan 4, 00:37 UTC
Identified - The denial of service attacks against our DNS infrastructure have resumed at this time.

Jan 4, 00:04 UTC
Monitoring - The attacks against our DNS infrastructure appear to have subsided at this time. We are currently monitoring for any additional attacks.

Jan 3, 21:02 UTC
Identified - We are currently experiencing denial of service attacks that are targeting our DNS infrastructure.

http://status.linode.com/incidents/dpdldmhgjbhl

Jan 5, 12:29 UTC
Resolved - Linode Manager, Website, and API are operating normally. At this time we are setting the issue to resolved.

Jan 5, 06:39 UTC
Monitoring - At this time, access to the Linode Manager, Website and API has been restored. Our engineers are currently monitoring the situation and working to prevent any further disruptions.

Jan 5, 04:11 UTC
Identified - At this time we have identified a DoS attack targeting Linode Manager, Website and API. Our Administrators are engaged on mitigating the issue.

http://status.linode.com/incidents/mhlblgngjnmn

Jan 1, 22:22 UTC
Resolved - The Linode Manager, Website, and API remain fully operational at this time. Our engineers are hard at work enacting measures to prevent any further service interruptions. Thank you once again for your patience. If you are still experiencing issues with any of these systems, please contact our support department for assistance.

Jan 1, 21:00 UTC
Monitoring - At this time, access to the Linode Manager, Website and API has been restored. Our engineers are currently monitoring the situation and working to prevent any further disruptions.

Jan 1, 16:30 UTC
Identified - We're currently working to mitigate a DoS attack that is causing connectivity issues with the Linode Manager, Website and API. We apologize for any inconvenience this might cause and thank you for your patience.

http://status.linode.com/incidents/cbbcjnhhpkgm

Jan 4, 01:31 UTC
Resolved - We have seen stability in Atlanta for enough time to consider this outage resolved. We apologize for the immense disruption to customers during this two-day outage, and we will be following up in the next week with a full post-mortem and plans for the future.

Jan 3, 23:33 UTC
Monitoring - Connectivity has been restored to the Atlanta datacenter. Users may encounter intermittent connectivity issues as we continue to monitor for additional attacks.

Jan 3, 18:52 UTC
Identified - We are currently working on residual issues from the ongoing DoS attacks that have caused ongoing connectivity problems on a subset of our hosts in Atlanta. Thank you very much for your patience.

Jan 3, 15:24 UTC
Monitoring - We believe that we have closed all of the attack vectors that can lead to a DDoS taking down the entire Atlanta datacenter. We are monitoring to confirm this.

Jan 3, 12:17 UTC
Update - The Atlanta data center is still experiencing network connectivity issues due to the DDoS attack. Our Network Engineers have been working throughout the night to restore connectivity.

Jan 3, 07:43 UTC
Update - We are still in the process of mitigating the DoS attack in our Atlanta data center. Our networking team is working with our upstream provider to restore normal connectivity to the data center.

Jan 3, 04:11 UTC
Update - Our Networking team is still actively engaged in restoring network connectivity. We'll continue to keep this status post updated with the latest information.

Jan 3, 02:06 UTC
Update - There has been some progress in restoring network connectivity to our Atlanta location, but our engineers are still hard at work to fully restore functionality to this datacenter. Thank you very much for your patience.

Jan 3, 01:03 UTC
Update - We're still actively working with our upstream provider to get the transit link hardened and getting connectivity restored. We will continue to keep providing updates until the issue is resolved.

Jan 2, 23:01 UTC
Update - The process of hardening our transit link and restoring connectivity in Atlanta is ongoing. We'll continue to provide updates as we work to resolve this issue.

Jan 2, 22:01 UTC
Update - Our networking team is still working to harden our dedicated transit link in Atlanta against further attacks. Once this process has been completed, connectivity should be restored. We'll continue to provide updates as things progress.

Jan 2, 21:15 UTC
Update - Our Network Engineers are still working with the transit provider mentioned below to apply the necessary DDoS mitigation hardening to facilitate restored connectivity to our infrastructure in Atlanta. We will keep you updated as these efforts continue.

Jan 2, 19:56 UTC
Update - Our network operations and systems teams have been working non-stop for the last ~36 hours toward a resolution of the Atlanta outage. We have acquired a dedicated transit link that is now directly connected to the Linode network, and we are waiting for this transit provider to apply DDoS mitigation hardening, after which we believe that Atlanta should be restored to full service. We will keep you updated as things progress.

Alex Forster
Network Engineer, Linode

Jan 2, 19:01 UTC
Update - Our Network Engineers are still actively working with our upstream provider in Atlanta to restore normal network connectivity.

Jan 2, 13:45 UTC
Update - Our Network Engineers are still working with our upstream provider on mitigating this latest DoS attack. We'll post an update here when we believe connectivity has normalized.

Jan 2, 07:01 UTC
Update - Around two hours after bringing things back online, attacks on Atlanta have started again which are affecting the entire datacenter. At this time we are being taken back offline to attempt further mitigation.

Jan 2, 05:24 UTC
Update - We have received word that all of our Atlanta IP space is now being announced. We are still fine tuning our geographic blocking, so people in the affected regions may see intermittent loss of connectivity over the next hour or two.

Jan 2, 04:42 UTC
Update - Zayo has begun bringing our IP ranges online.

For the short term, we will be using BGP communities to attempt to block Asia Pacific, the Middle East, South America, and others, hopefully leaving us only with traffic from North America and Western Europe. Blocking geographic regions this way is the only way to make sure that large botnets won't be able to launch further attacks.

We hope to have all of our Atlanta IP space advertised in the next several hours.

Alex Forster
Network Engineer, Linode

Jan 2, 02:36 UTC
Update - Our upstream provider has informed us that they plan to restore network connectivity to our systems gradually over an undisclosed period of time. There is no ETA at the moment, but our engineers and leadership team members are working closely with theirs to accelerate this process as much as possible. We will provide another update as soon as more information becomes available.

Jan 1, 23:02 UTC
Update - Our upstream provider reports that they've made some progress on mitigating the numerous DoS on their infrastructure, but due to the varying nature of the target, scope, and size of the attacks, network connectivity has not yet been restored. We are working closely with their team to remain informed on their progress, and will update this page as soon as there are any further developments. Once again, we sincerely thank you all for your patience.

Jan 1, 20:22 UTC
Update - The DoS attack in Atlanta is ongoing, and is still causing major network disruptions in our Atlanta location. We are continuing to work with our upstream provider to restore network functionality as quickly as possible. Thank you again for your patience.

Jan 1, 17:40 UTC
Update - The attack in Atlanta is still ongoing and causing significant connectivity issues. We're closely working with our upstream provider to ensure that connectivity is restored as soon as possible. We apologize for any inconvenience these issues are causing and we thank you for your patience.

Jan 1, 15:29 UTC
Identified - We're currently engaged with our upstream provider in mitigating a large, distributed DoS attack targeting our infrastructure in Atlanta. Users may experience packet loss and issues with connectivity as we work to resolve this. We thank you for your patience.

http://status.linode.com/incidents/cmhfcjnjlmhd

Jan 6, 02:34 UTC
Resolved - Network performance in Atlanta has been restored.

Jan 5, 00:06 UTC
Identified - We are aware of decreased network performance in our Atlanta location. We are continuing work to restore this location to full functionality. Thank you very much for your patience.

http://status.linode.com/incidents/nlfnbdlhkjkb

Jan 1, 16:03 UTC
Resolved - This incident has been resolved and connectivity appears to be stable in Singapore.

Jan 1, 15:36 UTC
Monitoring - The attack has been mitigated and connectivity in Singapore is returning to normal. We are continuing to monitor for additional attacks at this time.

Jan 1, 14:23 UTC
Identified - We've identified a large DoS attack targeting our infrastructure in Singapore and we're currently engaged in mitigating it. Users may experience significant packet loss and connectivity issues. We thank you for your patience and apologize for any inconvenience this might cause.

http://status.linode.com/incidents/jmbnghnfjghj

Jan 1, 16:03 UTC
Resolved - This incident has been resolved and connectivity appears to be stable in Frankfurt.

Jan 1, 15:39 UTC
Monitoring - The attack has been mitigated and connectivity in Frankfurt is returning to normal. We are continuing to monitor for additional attacks at this time.

Jan 1, 14:21 UTC
Identified - We are experiencing significant packet loss and connectivity issues as a result of a DoS attack targeting our infrastructure in Frankfurt. We're currently engaged in mitigating the attack and will update this post once we have further information. Thank you for your patience.