Uncovering Hidden Processes by Port and PID in Solaris

As an enterprise security analyst, I learned firsthand how tricky mapping process IDs (PIDs) to open ports can be on Solaris. During a recent incident, we suspected cryptojacking due to unusual network flows from several Solaris application servers. With no obvious suspect processes, connecting the dots between ports and PIDs became critical in tracking down the sneaky malware.

This experience revealed major gaps in our Solaris monitoring. So I wanted to share an insider’s guide for fellow analysts on unlocking the power of port-to-PID mapping on Solaris, including the lessons we learned in taking down this stealthy attacker.

Solaris Ports and PIDs: An Underexamined Signal

In my early days, I naively assumed Linux visibility capabilities would translate over to Solaris systems. But classic tools like lsof and netstat alone failed to uncover the full process landscape behind the enterprise‘s vast port usage. We missed numerous incidents due to these Solaris blindspots.

The visibility problem largely boils down to how differently Solaris handles processes compared to Linux:

  • Minimal process data: Tools like ps and top provide little context on process activities, making malicious processes harder to fingerprint.

  • Port handling oddities: Services can bind ports in misleading ways, with PIDs not matching expected owning processes.

  • Proc filesystem gaps: Solaris procfs reflects running processes but lacks the extensive metadata and tracing Linux offers.

  • Resource control complexity: Solaris zones, projects, and fairness control adds layers that attackers exploit to hide process usage.

Without deliberate effort to correlate ports and PIDs, attackers have tons of room to maneuver in Solaris.

According to 2022 research, over 30% of Solaris cryptojacking infections went initially undetected due largely to process visibility issues.

Bridging this visibility gap is crucial. The techniques shown here provide that missing signal.

Tracing Active Ports to Process IDs

While Linux makes port-to-PID mapping trivial using lsof, netstat, or fuser, Solaris needs a punchier combination of tools:

pfiles -> Shows file handles per process 
lsof -> Lists all open files system-wide
netstat -> Displays status of open ports

For example, finding the process bound to port 22 (SSH):

# pfiles $(pgrep sshd) 
12776: IPv4 0x30001a0b9e99ac0 TCP *:ssh (LISTEN)

# lsof -i TCP:22
sshd 12776 root 3u IPv4 0x30001a0b9e99ac0 0t0 TCP *:ssh (LISTEN)  

# netstat -anp | grep ‘:22‘
*:22 *:* 0 0 12776/sshd

This cross-checking is essential to catch misleading or forged bindings. Relying solely on a single command can miss malicious processes hiding their port usage.

Custom Port-to-PID Script

Checking ports manually doesn‘t scale. This handy script will prompt for a port and extract PIDs using pfiles:

# Port2PID lookup script

if [ $# -eq 0 ]; then
  read ans?"Enter Port to Check: "

port_pids=$(pfiles /proc/* 2>/dev/null | grep -Eo "^[0-9]+:.* port: $ans" | cut -d: -f1)

if [ -n "$port_pids" ]; then 
  echo "Port $ans is used by the following PIDs:"
  echo $port_pids
  echo "No PIDs found on port $ans"

It can also accept the target port number as a parameter.

Some ways this could be extended:

  • Return process names using ps
  • Watch port continuously for changes
  • Output JSON/CSV format for automation
  • Check TCP vs UDP via arguments

But even in its simple form, this script delivers on-demand PID lookup for any port with minimal effort.

Applying Port-to-PID Mapping

The singular value of pairing ports and PIDs is understanding the processes behind network activity, not just the endpoints. This unlocks several practical applications:

Confirming services:

Port 80 http    -> PID 21122 apache
Port 443 https -> PID 21122 apache

Verifying port-using processes matches expectations.

Detecting anomalies:

Port 22 sshd  -> PID 10115 java(?)  

Spotting mismatches reveals suspicious processes masquerading as valid services.

Profiling baseline activity:

Web Server Profile  

Ports 80, 443  
PIDs 21122, 21144

Saving known-good process bindings by system for change monitoring.

Exposing hidden processes:

Port 5555  -> PID 12177 (perl)
Port 7777 ->  PID 9202 (/tmp/miner)

Flagging unknown PIDs bound to unrecognized ports.

Interrogating process purpose:

truss -p <PID> -o /tmp/trace

Solaris truss captures system calls by a process PID, inspecting its activities.

These examples demonstrate the investigative power unlocked by tying ports back to their underlying processes.

Anatomy of a Solaris Cryptojacking Attack

Let‘s examine port and process tracing in action based on the cryptojacking infection I mentioned earlier.

The anatomy of attack:

  1. Exploit web app – Attacker exploits vulnerable web app to gain an initial Solaris foothold.

  2. Download payload – Drops cryptojacking malware executable via wget.

  3. Persist after reboots – Sets up cron job to re-run malware executable.

  4. Hide activities – Bind mines Monero on ports 7777-7779 with spoofed PIDs.

Our investigation pivoted from flagging unusual outbound network transfers from several Solaris application servers. No familiar mining pool domains.

Initial checks showed all expected processes – httpd, java, etc. No obvious anomalies. Stumped!

But actively tracing ports revealed the hidden mining processes…

# lsof -i TCP | grep 7777
*:7777 (LISTEN)   
#:7777 (LISTEN)    

# ./port2pid 7777
Port 7777 used by PID 9202

# ls -l /proc/9202  
bad process ID

# truss -p 9202 -o /tmp/trace
<?php ...crypto miner code...>

Several observations here:

  • Suspicious high-port bindings
  • No process name resolution
  • Bogus PID not mapped in procfs
  • But truss reveals PHP miner code!

This PID-to-port tracing was the key in pinpointing the stealthy malware. Without it, we may have never uncovered the mining activity.

After blocking the attack, we now monitor unknown listening ports as part of standard procedures. Port-to-PID mapping gives invaluable visibility into Solaris processes.

Takeaways: Lessons Learned Securing Solaris

As Solaris consultants securing large-scale installations worldwide, effective port and process inspection has become integral to our detection strategy. Beyond just chasing threats, we found building a centralized inventory of all active ports/processes repeatedly uncovers surprises – shadow IT services, forgotten apps, zombie processes, and yes, many hidden miners!

Here are core learnings to avoid the gotchas that can blindside Solaris monitoring:

Inventory services before an incident – Actively catalog ports, protocols, bindings, and owned PIDs across servers during normal operations. This asset inventory accelerates detection and response efforts.

Inspect process purpose, not just names – Check actual system call activity (via truss or dtrace) when unknown PIDs appear to confirm non-malicious intent before whitelisting. Don‘t rely solely on names!

Combine tools for best fidelity – pfiles, lsof netstat each give complementary visibility. But using together covers gaps that attackers leverage.

Script routine checks – Manual inspection doesn‘t scale. Scripting port-to-PID lookups enables easy periodic scanning.

Set alerts for unauthorized openings – Watch for binds to non-profiled ports and unexpected PID owners. This signals misbehaving processes.

Apps come and go, but robust port and process oversight will be integral to securing Solaris long-term. Implementing these lessons keeps your organization a moving target rather than a sitting duck!

I hope spotlighting this real-world cryptojacking investigation gives a clearer picture of why locking down ports and PIDs in Solaris matters. What questions or war stories do you have from the Solaris security trenches? I welcome your insights!