Review: Threat hunting turns the tables on attackers

Advanced Persistent Threats are able to slip past even the most cutting-edge security defenses thanks in large part to a diabolically clever strategy. The threat actors behind successful APTs research the employees, practices and defenses of the organizations they want to attack. They may try to breach the defenses hundreds or thousands of times, then learn from their mistakes, modify their behavior, and finally find a way to get in undetected.

Once a network is breached, most APTs go into a stealth mode. They move slowly, laterally compromising other systems and inching toward their goals. Post-mortems from successful attacks often show that the time an APT breached a system to the time it was detected could be anywhere from six months to a year or more. And, they are often only detected after making that final big move where there is a huge exfiltration of critical data.

But what if you could turn the tables on APTs? Instead of focusing on your perimeter defenses, what if you assumed that APTs were already hiding in your network and you launched software specifically designed to hunt down these active, but hidden threats before they can do real damage?

For this review, we tested threat hunting systems from Sqrrl, Endgame and Infocyte. Each program was tested in a large demo environment seeded with realistic APTs which had bypassed perimeter defenses and were hiding somewhere within the network of virtualized clients and servers. We also snuck active threats past perimeter defenses to see how these threat hunting programs detected, caught and killed the current breed of apex predators of the threat landscape.

We found that in order to deploy these products successfully, security professionals must change the way they normally think. These threat hunting tools are not the passive observers that we’re accustomed to, simply reacting to alerts triggered in the SIEM. Instead, these are aggressive hunters who prowl their own networks looking to prey on APTs and undetected malware.

Here are the individual reviews (also see screen shots of each product):

Sqrrl Data

The Threat Hunting Platform from Sqrrl Data was created by several ex-employees of the National Security Agency in 2012. Sqrrl integrates into any network and collects data from the SIEM as well as other sources, such as outside threat data feeds. It is normally installed as software but can be run in a virtualized or even a cloud environment.

Sqrrl does not install agents on endpoints, but can provide more information to hunters by incorporating data from existing endpoint protection programs.

The installation takes no more than a couple of hours for most deployments. Sqrrl offers one or two days of training as part of the installation process, though not a lot of time is needed due to the graphical and intuitive nature of the software. We became fairly proficient hunters and were able to track down leads and uncover hidden threats after only a few hours of instruction.

At the beginning of every day, security analysts are greeted with an overall control panel showing various indicators and suspicious behaviors along with their relative severity. Sqrrl needs about seven days examining user behavior before it can accurately predict the suspicious behavior component, and its machine learning ability makes it even more accurate over time.

It’s critical to note that the behaviors which bubble up to the Sqrrl dashboard are not ones that have triggered any type of SIEM alert. Anything over the threshold of potentially malicious behavior set by the network’s active security programs is handled by security personnel however they normally would do that. What is left are the odd little things that may, or may not, be an indicator of compromise which has slipped through the cracks.

Hunters can then use their expertise to investigate behaviors like beaconing, lateral movement, data staging, unusual usage patterns and exfiltration to create a hypothesis and potentially uncover a breach. It’s possible that hunters can also verify valid activities and clear them from further consideration.

+ ALSO: In depth: What does APT really mean? +

Since we knew that most APTs rely on privilege elevation as part of their pattern, we launched an investigation, or hunt, based on a single odd event captured by Sqrrl where an administrator logged into a system labeled C586. The strange thing was that the admin had never touched that system before, but since they logged on using valid credentials the first time they tried, no alerts were triggered. Sqrrl flagged the behavior, and thus we began our investigation.

The great thing about Sqrrl is that everything is displayed visually. We didn’t have to pore through the 85 pages of related log files, although they were available if we wanted, to find out what other systems had connected or were somehow involved with C586. We sent in a query from the drop-down menu and discovered a chain going back through four other systems with lateral movement ties to the one under investigation.

From there, we looked at beaconing behavior and discovered that the next system in the chain had beaconed out at some point over the previous month. Because beaconing behavior is one way that APTs reach back to their hosts, this was suspicious even though the IP was not one indicated as dangerous by threat intelligence feeds, and thus had not triggered any alarms.

Pushing our hunch, we searched for that IP address and were surprised to discover that two other systems in the same chain had also beaconed out to the same location. Now the picture was becoming more clear.

It also seemed like the fourth system in the chain, which had several denied access attempts recorded, was not actually part of the attack though it had connected with others that were. The failed access attempts were either the legitimate mistakes of a user forgetting their password, or perhaps deliberate camouflage from the attackers attempting to trigger an alarm to get security personnel looking in the wrong place.

Back to the three systems with beaconing behavior. We queried and found a rogue PHP process active on all three. Looking over time, it was clear that each system beaconed out and used that process only long enough to capture a new system before going dark. The attack chain finally stopped after it accessed C586, but didn’t install anything on it and did no beaconing from there.

A little while later, the administrator logged into C586 successfully even though they had never done so before. But using Sqrrl, we were able to discover why, and had a very good idea that those credentials were compromised, even though C586 was totally clean and triggered no alarms.

With a successful hunt completed, we could generate a report so that the network could be protected. The administrator’s compromised credentials could be rescinded as well as the login passwords for the compromised users. The IP of the beacon could be blocked and the PHP expunged so that the attackers will have wasted all that time only to be stopped short of their actual goal. And their tactics and techniques could be fed back into both Sqrrl and the network SIEM to catch them if they tried again using the same method.

The Sqrrl Threat Hunting Platform is a great tool to aid those hunting hidden threats inside their network. It works for users with any skill level, but more experienced analysts will be able to create better theories about attacks and thus likely have more successful hunts. Pricing for Sqrrl is based on the number of hunters who need to use the system and the amount of internal traffic data that needs to be analyzed. A system with a single hunter on a modest sized network would start at $25,000. Given that the average successful breach can cost half a million dollars or more in direct and indirect costs, sponsoring a hunter and equipping them with Sqrrl seems like a good preemptive investment.

Infocyte Hunt

Unlike the more traditional model of a lone hunter stalking their prey, Infocyte Hunt has added vast amounts of automation to the point where an entire network can be hunted in about a day. It’s more like hunting from a helicopter with a machine gun.

Founded by former Air Force officers in 2014, Hunt was designed to replace the sometimes months-long, labor intensive hunting process that some government agencies were using at the time. Hunt is completely centered on network endpoints and has no need for additional sensors. The main console, which is traditionally installed as a virtual machine, but is lightweight enough to exist on a laptop, sends out agents to all endpoints. However, the agents only exist for about 90 seconds on each endpoint and are dissolved afterwards. Hunt works natively with Linux and Windows endpoints plus most payment processing terminals. A Mac version is in the works.

Pushing out an agent takes up about 1 megabyte of network bandwidth while the return response is about 1.2 megs. The software defaults to sending out 60 at a time, and agents are smart enough to wait if the network is too busy, sending their report back when traffic clears. Using this method, Hunt is able to scan about 25,000 endpoints a day if the network is that large. Our test network had a modest 50 clients, so the total process took about a minute.

The main console controls the agent deployment and response process as well as the reporting dashboards, but heavy lifting is done in the Infocyte cloud. That includes hash and DNS lookups as well as comparing results with outside threat feeds and even sandboxing. Government agencies or companies that prefer to keep everything inside their networks can opt for a much larger on-premises configuration. In addition to the lookups, unknown executables can be submitted to Infocyte for analysis, where the staff maintains a threat lab to help identify zero-day type attacks. Human operators need to choose to submit those for analysis help, so again, data will only leave a network if it’s authorized to do so.

+ RELATED: Infocyte HUNT sets out to answer the question, “Have we been hacked?” +

To begin our investigation, we first had the console send out the dissolvable endpoints to our network. A report quickly came back because our test network was so small. From there, we could see that several endpoints could not be scanned. One of those had recently changed its login credentials. We could then log into it by hand and make sure it got the agent from the Hunt console. Another was disconnected from the network, so there was nothing we could do about that other than setting Hunt to catch it when it was back online. A couple of clients were VR machines that had been decommissioned but whose images remained in Active Directory. Those could be eliminated from future consideration.

The default scan looks at everything within the detection capabilities of Hunt including processes, modules, drivers, memory scanning, account information, network connections and hooks. Scans can also be tailored to specific items. If you are explicitly hunting malware disguised as a driver for example, you could just run that part of the scan. However, because the dissolvable agents are so quick, you don’t really save too much time paring them down, so the full scan is probably best most of the time.

With active endpoint scanning, Hunt could almost be deployed as a more traditional security tool, especially for organizations that have not invested heavily in endpoint protection. However, while Hunt can find traditional threats, its value as a threat hunting tool is that it is designed to catch advanced malware that would otherwise avoid detection.

As an example, in our testing we found an instance where Firefox.exe was listed as probably bad on one client machine. This was quite puzzling so we dove into that part of the report, which was easy to do using a good graphical interface. Drilling down to the first level, we found that everything with Firefox seemed fine. Hunt runs all endpoint programs through 21 anti-virus programs and provides a report back on their findings. In this case, all of them said that the file was fine, although Hunt was still not convinced. Drilling down further, the hash for the Firefox file was correct, so it was the actual Firefox program provided by the company.

We started to think that Hunt was providing us with a false positive, until we went a little deeper. It turns out that a module installed inside that version of Firefox turned out to be a bitcoin miner. Hunt not only caught this during the sandboxing process, but also allowed us to see every module that was part of the core program. That enabled us to identify a threat that would have escaped almost every other type of endpoint protection.

Another strength of Hunt is its ability to do true memory mapping, so malware that only exists within memory, even if it uses stealth technology or tries to stay under the radar, is quickly identified. We found a memory injection type of attack against Explorer on another machine. Hunt can take all types of memory code and convert it into executable files which can then have their characteristics checked in a safe environment. We have not previously seen this type of mapping process that allowed Hunt to drill so far down, and so accurately, into the system memory of connected endpoints.