WRITTEN BY THOMAS RAEF
This article was contributed by Thomas Raef, the founder of WeWatchYourWebsite.com. Thomas runs a website protection, monitoring and malware removal service and is one of a very, very short list of people who we recommend for this service.
So you got a notification from your server provider that your server is attacking other websites, usually a brute force login attack, and if you don’t get it resolved, they’ll have to shut down your server.
Panic sets in.
Which site on that server???
They don’t know. The people who reported your server can’t tell which site, or even which file was used to attack their server. The log files on your server don’t show outgoing traffic. Only what’s been accessed. That’s why they’re called access logs. 😉
Now, it’s up to you to determine what site, if you have multiple sites on that server. If it’s only one website, you need to find out how it’s happening. What infection is being used to brute force other websites?
You’ve got a limited amount of time before your server provider shuts down your server. What to do???
FIRST, THE REQUIREMENTS
You must be comfortable with SSH and the Linux command line.
We’re going to use the Linux Auditing System. This functionality provides kernel-resident logging of system calls and user space tools to collect and view the logs. You have control over configuring what is logged.
When you’re notified that your server is attacking other sites, you need to find out how, from where, and when.
auditd
SSH into your server then type:
auditd
If you get back your command line, with no other text, then auditd is already installed.
If instead, you get back something like this:
Then, auditd is not installed.
To install it, type in:
apt install auditd
Time to Get Started
Now, to get into the nuts and bolts of finding how your server is being abused by hackers, type this in:
auditctl -a exit,always -S connect -k watching
What this is doing is adding a rule for auditd. The rule is looking for any outbound connections and then logs them. The nice thing about auditd is that it includes so much information. The -k watching at the end of the above command line is setting a key for us to search by. You can change that to any single word you want.
Then look in /var/log/audit/audit.log
You might see something like this:
type=SYSCALL msg=audit(1615055175.648:9511984): arch=c000003e syscall=42 success=no exit=-101 a0=5f a1=7f0de8002010 a2=1c a3=4 items=0 ppid=1 pid=21732 auid=4294967295 uid=1001 gid=2002 euid=1001 suid=1001 fsuid=1001 egid=2002 sgid=2002 fsgid=2002 tty=(none) ses=4294967295 comm="gutenblock-64" exe=2F7661722F7777772F74657374736974652F7075626C69635F68746D6C2F77702D636F6E74656E742F677574656E626C6F636B2D3634202864656C6574656429 key="watching"
To search through the logs you can type:
ausearch -k watching
That will list all the log entries with the key=”watching” in it.
In the above log entry you’ll see the pid of the process. This helps us “kill” it, but also provides further information. The comm field is the command-line name of the command used to invoke the process. In this example it’s “gutenblock-64”.
The exe field is the goldmine. It’s an ASCII hex encoded string with the exact path of the executable, when it was executed.
Bingo!
The above converts to:
/var/www/testsite.com/public_html/wp-content/gutenblock-64 (deleted)
It shows (deleted) because the filesystem knows it was already deleted.
In this particular case (an actual case, but the string was changed), the hackers were uploading a file, running it, so it was running in memory, then deleting the file.
Tricky huh?
We now have the path, which would tell us what website (testsite.com) is being abused by the hackers. But we still need to investigate further.
Like I stated earlier, the auditd log entry shows us the pid of the offending process. In this case it was 21732.
To verify the pid to comm, type this (change the 21732 to the pid showing in your auditd log):
ps -p 21732 -o comm=
Next, let’s see what user actually ran this command. Type this at the command prompt:
ls -l /proc/21732/fd
You’ll see this type of information:
lr-x------ 1 Mike7 Mike7 64 Mar 7 11:43 0 -> /dev/null
l-wx------ 1 Mike7 Mike7 64 Mar 7 11:43 1 -> /dev/null
lrwx------ 1 Mike7 Mike7 64 Mar 7 11:43 10 -> 'socket:[132580889]'
lrwx------ 1 Mike7 Mike7 64 Mar 7 11:43 100 -> 'socket:[132582695]'
...
And a whole bunch of similar lines.
So we see that the system user Mike7 (fictitious) is the culprit. We can verify this because the auditd logs also provide the fsuid. In this case it was fsuid=1001.
Look in the /etc/passwd to see which user is 1001.
Mike7:x:1001:2002::/home/Mike7:/bin/bash
If you have set-up separate system users for each website, then you know which website is being abused to attack other websites.
fsuid=1001, then look in /etc/passwd to see which user is 1001:
Marg3:x:1001:2002::/home/Marg3:/bin/bash
For further information, let’s see who is being attacked.
Type:
lsof -u Mike7
You’ll see something like this:
xxx.xxx.xxx.xxx.vultr.com:60298->lhcp3047.webapps.net:https (SYN_SENT)
gutenbloc 27626 Mike7 52u IPv4 132535731 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:35752->ip-166-62-108-229.ip.secureserver.net:http (SYN_SENT)
gutenbloc 27626 Mike7 53u IPv4 132542778 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:55124->hosted.by.liquidnetlimited.com:https (ESTABLISHED)
gutenbloc 27626 Mike7 54u IPv4 132538383 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:55196->ekiaioesiq.c09.mtsvc.net:http (SYN_SENT)
gutenbloc 27626 Mike7 55u IPv4 132539972 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:63150->ip-107-180-26-69.ip.secureserver.net:http (SYN_SENT)
gutenbloc 27626 Mike7 56u IPv4 132537091 0t0 TCP
The xxx.xxx.xxx.xxx will be the IP address of your server. The information after the “->” is the site being attacked.
How to remediate this???
The million dollar question.
Okay, it won’t cost you that much, unless you got it like that! 😉
First let’s kill that process:
kill -9 21732
Now, for the forensic analysis we need to go to the access logs.
In this case, the access logs were in the /var/log/nginx/ folder.
Search for any log entry that contains the string /wp-content for the site discovered above. Remember the full path of the executable was:
/var/www/testsite/public_html/wp-content
You would look in:
/var/log/nginx/testsite.com.access.log
In this case, on the date in question, March 7, we found these log entries that contained the /wp-content path:
[07/Mar/2021:03:33:01 +0000] 173.254.223.217 0.012 - testsite.com "GET /wp-content/plugins/jetsticky-for-elementor/includes/lib/class-tgm-plugin-activation-siS0b6.php HTTP/1.1" 200 2291 0.011 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4o44.122 Safari/537.36"
[07/Mar/2021:03:33:02 +0000] 173.254.223.217 0.024 - testsite.com "POST /wp-content/plugins/jetsticky-for-elementor/includes/lib/class-tgm-plugin-activation-siS0b6.php HTTP/1.1" 200 2352 0.022 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4o44.122 Safari/537.36"
[07/Mar/2021:03:33:02 +0000] 173.254.223.217 0.552 - testsite.com "GET /wp-content/file_get.php HTTP/1.1" 200 15 0.555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4o44.122 Safari/537.36"When we look in the /wp-content folder, there is no file named file_get.php.
This confirms the fact that the hackers would use a backdoor previously uploaded (class-tgm-plugin-activation-siS0b6.php), use it to upload the file_get.php, run it, then delete it, leaving the process running in memory. This has been witnessed a lot recently.
How the hackers uploaded the backdoor (class-tgm-plugin-activation-siS0b6.php) is for another KB.
You’ve learned how to find the website being abused by hackers to attack with brute force logins other websites, how to find the process being used, the user that was compromised, what sites are being attacked, how to kill the process, and how to find the method used by the hackers.
So, you can now delete that backdoor, check for others and let your server providers tech support know you’ve found the cause and remediated it.
IF YOU ENJOYED THIS ARTICLE...
Thomas has contributed multiple articles to the vCanopy knowledge base, all of which are listed below: