Kaseya delays patch fixing zero-day attack as issues hit SaaS rollout

Kaseya encountered an issue while restoring the software-as-a-service version of its IT management platform, and said that would delay the rollout of the patch for the self-hosted version. The beleaguered IT services provider has been providing regular updates ever since the company discovered a cyberattack against its software last Friday.

“We have not yet been able to resolve the issue. The R&D and operations teams worked through the night and will continue to work until we have unblocked the release,” Kaseya said in its latest update Wednesday morning.

After discovering that attackers had compromised Kaseya VSA and were delivering ransomware to networks being managed by the tool, Kaseya brought down the SaaS version of Kaseya VSA and instructed customers to shut down the on-premises servers to prevent further attacks. After determining that attackers were not targeting the SaaS platform, Kaseya started the process to restore SaaS VSA — and also began “configuring an additional layer” for security — on Tuesday afternoon. The layer “greatly reduces the attack surface of Kaseya VSA overall,” Kaseya said.

Under the original timeline, the patch for the on-premises version of Kaseya VSA would have been available within 24 hours of completing the SaaS deployment. Customers who run Kaseya VSA locally on their own servers would receive a set of recommendations on how to increase their security posture before restarting the VSA. With the delay in SaaS rollout, it is looking more likely the efforts to mitigate against the attacks will continue into the weekend. For victim enterprises hit by ransomware, efforts to recover the data — either through backups or by negotiating the ransom — are ongoing and separate from the work to get Kaseya VSA back up and running.

“Our On-Premises patch timeline is 24 hours (or less) from the restoration of SaaS services,” Kaseya said in its rolling advisory. “We are focused on shrinking this time frame to the minimal possible – but if there are any issues found during the spin-up of SaaS, we want to fix them before bringing our on-premises customers up.”

IT teams use Kaseya VSA to manage the infrastructure — which includes activities such as network management, system updates, and backups. The fact that the ransomware attack is exploiting an IT management tool complicates recovery, Matt Tait, the Chief Operating Officer of Corellium, wrote on Lawfare. The first step to remediate malware is disabling the delivery mechanism. When the malware is using the organization’s software delivery infrastructure, that means disabling the very tool used to deploy fixes. Regaining control of servers and restoring data from backups becomes much more difficult when the management tool is part of the problem.

Identifying victims

While the company initially said fewer than 40 customers were affected, that figure has now been revised to “fewer than 60.” Since Kaseya VSA is popularly used by managed service providers to monitor customer IT infrastructure, the attack goes beyond these direct victims because customers for these IT providers are also affected. Kaseya estimated “fewer than 1,500 downstream businesses” have been impacted. There are reports that Swedish grocer Coop was forced to close 800 of its stores for more than two days because its cash register software supplier was impacted by the attack.

Security company Sophos said its evidence shows 70 managed service providers and 350 downstream customers have been affected. Most of the victims were in the United States and Canada — 145 victims in the U.S. and 77 in Canada — but victims were found in Germany, Australia, the United Kingdom, and other regions, the company said.

Huntress Labs has been tracking approximately 30 MSPs across the U.S., Australia, European Union, and Latin America, affecting 1,000 downstream customers.

“We expect the full scope of victim organizations to be higher than what’s being reported by any individual security company,” Sophos CISO and vice president Ross McKerchar told VentureBeat. “The attack didn’t discriminate by geography or business type that we can tell at this time.”

Enterprises should use the company’s Compromise Detection Tool (available as a Box download) to identify whether indicators of compromise, data encryption or the REvil ransom note are present in the network.

The Cybersecurity and Infrastructure Security Agency and the Federal Bureau of Investigation released guidance for managed service providers and their customers who have been affected. Security teams should enable and enforce multi-factor authentication on every single account that is under the control of the organization to make it harder for attackers to seize control of the accounts. Another thing to do is to limit communication with remote monitoring and management (RMM) capabilities to known IP address pairs, CISA recommended in its guidance. Administrative interfaces of RMM should also be placed behind a virtual private network or a firewall on a dedicated administrative network.

Supply chain attack

Early reports suggested attackers had modified the code for Kaseya VSA and that was how ransomware was being pushed to victims, but Kaseya said it found no evidence of its code being maliciously modified. Rather, it appears that attackers had found and exploited several vulnerabilities in the software.

“The attackers were able to exploit zero-day vulnerabilities in the VSA product to bypass authentication and run arbitrary command execution,” Kaseya explained in its incident analysis. “This allowed the attackers to leverage the standard VSA product functionality to deploy ransomware to endpoints. There is no evidence that Kaseya’s VSA codebase has been maliciously modified.”

The attack against Kaseya looks less like a supply chain attack in the sense of what happened to Solar Winds late last year, and more like a malware campaign that triggers zero-day vulnerabilities in the software to execute malicious code. A supply chain compromise is “indiscriminate,” Tait said, noting that everyone installing the update would get the malware. In this case, the attacker has to trigger the zero-day vulnerability on each victim server.

However, it is still arguably a supply chain attack because the attackers targeted third-party suppliers — in this case, the MSPs — to breach the networks of customer organizations.

Kaseya knew about at least one of the flaws (CVE-2021-30116) used in the attack, as it had been reported by the entity Dutch Institute for Vulnerability Disclosure (DIVD). However, this wasn’t a case of Kaseya neglecting to fix the issue. “[Kaseya] has shown that they were willing to put in the maximum effort and initiative into this case both to get this issue fixed and their customers patched,”DIVD said in its advisory.

“Unfortunately, we were beaten by REvil in the final sprint, as they could exploit the vulnerabilities before customers could even patch.”