LucidLink

Join us next week at #MPTS2024 to learn more about remote #collaboration between creatives and animators 🎨 Hear more from Daniel Budin about the first remotely produced film to win an Oscar - #TheBoytheMoletheFoxtheHorse and Georgi Chapman (Simon's Cat) about working with other notable creatives remotely from anywhere. Find out more 👇

Come say hi at #MPTS and learn how we can help you reinvent your creative workflows 🎈 Book a meeting 👇

Hi Everyone, As promised, we are sharing the root cause analysis publicly, which includes details on what led to the outage, the steps taken to reconnect service for all customers and the mitigation actions for the future. We are extremely sorry for the disruption the outage had on our customers. While we are proud that our zero-knowledge encryption model performed as intended and our customer data remained secure throughout this malicious attack, the entire company has learned and grown from this experience. As a company made of both enterprise storage technologists and former content creators, our superpower has always been understanding the workflows and experiences of content creators. Because of this, we deeply felt the pain, anxiety, and frustration of our customers’ downtime. Please know, we remain: - Committed to ensuring that the security of our infrastructure remains appropriate and limiting the potential that another outage like this is experienced again. - Devoted to our customers and continuing with your trust in the system that has been a part of your everyday workflows. - Intent on providing ongoing and transparent communication regarding relevant findings and outage resolution. - Dedicated to supporting the talented creatives that have been in our DNA since the beginning. https://lnkd.in/g7yv3HkC

Now that service has been restored for all customers, we are working on the root cause analysis and will distribute this document publicly once completed early next week. It bears repeating that this was not a ransomware attack, and no customer data was compromised. Although users couldn’t access their data during the outage, no personal or corporate data was leaked or otherwise affected due to LucidLink’s Zero-Knowledge security model. Given the severity and maliciousness of the attack, LucidLink has engaged with law enforcement, legal, and external cyber forensic experts. Thank you for your patience. We are committed to helping all of our customers address any remaining disruption from this outage.

We are happy to announce that the manual intervention to restore all filespace metadata services for the few remaining accounts is complete. All customers regained access to their filespaces, and we became fully operational as of mid-day UTC. We continue to monitor the situation. If you have any further issues, please create a ticket in our support system to allow us to support you in the best way possible. The past couple of days have been pretty crazy. It’s the first time we’ve been in this situation since founding the company 8 years ago. We have learned a lot and grown from the experience. We are proud that our Zero-Knowledge architecture performed as intended, preventing any data breaches. We also want to acknowledge the downtime experienced by our customers during this outage. We fully understand that you rely on our technology and take this seriously. Now that service has been restored for all customers, we will transition efforts to root cause analysis. As stated, we will share our findings once we have all the facts. The support and encouragement we received from our customers has been humbling. It makes the efforts of the past 48 hours meaningful and reaffirms our commitment to openness and transparency.

We are happy to announce the restoration of filespace metadata service is now complete, and access to the vast majority of filespaces has been fully restored. As we expected, a small number of metadata instances failed to initialize properly and will require manual intervention. This is a time-consuming process, and our team will continue working until all customers are fully restored. If you are one of the customers who still has a filespace in a disconnected state, please create a ticket in our support system. We will proactively create tickets on behalf of affected accounts as we identify those without open tickets. Moving forward, we will switch our communications from general mass updates to individual tracking in our support ticket system. We understand that your organizations require additional in-depth accounting of what took place and why, as well as the steps being taken to mitigate the possibility of future occurrences. Now that we have brought the majority of our customers back online and reinforced our systems, we will continue implementing measures and will provide a report for you and your management. We do not yet have a timeline for this, but we are committed to sharing our findings. Thank you to everyone who has supported, encouraged, and provided moral support these past days. It is the energy of our customers that fuels the passion of our entire team.

EDIT: Issues have been resolved and services restored We are about to launch the final and largest batch of filespace metadata service restorations. We have further increased the build infrastructure to expedite the process, which we estimate with a high degree of confidence to take 4 hours to complete. This is a rolling restoration, so filespaces will continuously become available during this period. Once again, users may need to “connect” or “disconnect” and “connect” from the Lucid App for the connection to resolve. Remember that if you have had access to your files during the outage, be sure to copy them to a local disk before completing the above. We know there will be some edge cases where the service instance will require manual intervention, which we must address once the batch run has finished. Our teams will work with the individual customers owning these filespaces as soon as they are identified. Thank you again for your patience and for hanging in there with us. We will provide an update when the next batch is completed.

EDIT: Issues have been resolved and services restored Here is the latest update on restoring filespace access for all of our customers. We now have about 20% of affected filespaces back online, and the restoration velocity is increasing. Many of the early restores required manual intervention, taking longer than anticipated to get to this point. We’re now increasing the restoration batch size and expect to be through the process within the next 6 to 8 hours. Again, this will be done on a rolling basis, and the filespaces will be available as soon as each restoration is completed. Note that you may need to manually click “connect” in your Lucid App or “disconnect” and then “connect” for your filespace to resolve as quickly as possible. Remember that if you have had access to your files during the outage, be sure to copy them to a local disk before completing the above. Thank you for your patience and continued support.

EDIT: Issues have been resolved and services restored We rebuilt and scaled the discovery service to accommodate the additional load of the disconnected filespaces trying to reconnect to the offline metadata service. Based on these updates, a number of our customers have already regained access to their filespaces. You can confirm your connection status through the LucidLink App, which will show a status as “connected.” We have confirmed that the backup restoration process is working as expected. The first restored filespaces are back online. As we complete more restorations, we’ll have a better estimate of how long the entire process will take to restore all of our customers affected. We realize this is the most important point for you, and as soon as we have more visibility, we’ll update you. You may have seen a new error message saying, “AuthProxy: Call to the LucidLink filespaces service timed out.” This indicates your client is now getting through to discovery but timing out because your filespace's metadata service is offline. Once the metadata service is restored for your filespace, you will be able to reconnect. We’re receiving a number of questions regarding the scenario where customers remained connected to their filespace and continued to work during the outage. We have created a Knowledge Base article hosted on our support site detailing various scenarios - https://lnkd.in/d6WPEZqY Because there are variations depending on circumstances not readily apparent, we recommend that customers save any work done during the outage to a local disk to ensure your work is saved.

EDIT: Issues have been resolved and services restored I wanted to provide another update on the latest status for restoring filespace access for all of our customers. First, I want to reiterate that no file data prior to the last backup before the outage was lost and no personal or corporate information was leaked. However, the metadata service for each filespace was damaged during this attack, which is causing the connection outage. We are fixing the metadata service which will restore access to your data. Each filespace's metadata service is restored individually. We’ve identified a number of filespaces with metadata service unaffected, but are unable to connect to the discovery service due to the metadata infrastructure attack. We have allocated additional resources to focus on that subset and will manually reconnect these filespaces to the discovery service, bringing them online. This should happen within the next hour, by 16:00 UTC. In addition, we are now in step two of the process, restoring the metadata state for all individual filespaces, and are on track to restore access to filespaces in the next 4-6 hours. We will do this on a rolling basis to get customers back online faster. During this process, we will not be able to notify customers individually when their filespace is back online. Customers will be able to monitor their connection status on the LucidLink app. Thank you for your patience and continued support. We will provide our next update around 17:00 UTC. Regards, Peter Thompson CEO, LucidLink