ProxyJump is safer than SSH agent forwarding

An SSH jump server is a proxy standing between clients and the rest of the SSH fleet. Jump hosts minimize threats by forcing all SSH traffic to go through a single hardened location and minimizing an individual node’s SSH endpoints to the outside world. (Read more: “How to set up an SSH jump server.”)

One way to configure a multi-hop setup is by storing a private key for the destination server on your jump server. Do not do this. A jump server is usually a multi-user environment, meaning any single party with elevated privileges could compromise any private key. A solution to this security threat is enabling agent forwarding. Given how common this method is, it may surprise you to learn this is not recommended. To understand why, let’s dig a bit deeper.

[ Also on InfoWorld: Make life easy with ssh_config ]

How does agent forwarding work?

ssh-agent is a key manager that exists as a separate program from SSH. (Read more: “How to manage SSH keys.”) It holds private keys and certificates used for authentication in memory. It does not write to disk or export keys. Instead, the agent’s forwarding feature allows our local agent to reach through an existing SSH connection and authenticate on a remote server through an environment variable.

Basically, as client-side SSH receives key challenges, the agent will forward these challenges upstream to our local machine, where the challenge response will be constructed via a locally stored private key and forwarded back downstream to the destination server for authentication. (Read more: “SSH handshake explained.”)

Behind the scenes, ssh-agent binds to a Unix domain socket to communicate with other programs ($SSH_AUTH_SOCK environment variable). The problem is that anyone with the root permissions anywhere in the chain can use the created socket to hijack our local ssh-agent. Even though socket files are well protected by the OS, a root user can impersonate another user and point the SSH client to their own malicious agent. In essence, forwarding using an agent is the same as sharing a private key with anyone that has root on a machine throughout the chain.

In fact, the man page regarding ForwardAgent reads:

Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent’s Unix-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.

Use ProxyJump instead

To navigate through jump servers, we actually don’t need agent forwarding. A modern approach is to use ProxyJump or its command line equivalent -J. (Read more: “SSH configuration: ssh_config.”)

Host myserver
      HostName myserver.example.com
      User virag
      IdentityFile /users/virag/keys/ed25519
      ProxyJump jump
Host jump
      HostName jump.example.com
      User default   

Instead of forwarding the key-challenge response via agent, ProxyJump forwards the stdin and stdout of our local client to the destination host. This way, we do not run ssh on jump.example.com; sshd connects directly to myserver.example.com and gives control of that connection to our local client.

As an added benefit, the jump server cannot see any traffic traveling through it due to it being encrypted within the SSH tunnel. The ability to set up a jump server without letting direct SSH access onto it is an essential component of safe and proper SSH setup.

ProxyJump for multiple hops

Let’s simulate a more complicated scenario. We are attempting to access a critical resource deep in our corporate network from home. We must first pass through an external bastion host with a dynamic IP, an internal jump host, and finally to the resource. Each server must authenticate against a unique local key on our machine. (Read more: “Setting up an SSH bastion host.”)

Teleport

Once again, our local config file will contain everything we need to execute ssh myserver.

Host myserver
      HostName myserver.example.com
      User virag
      IdentityFile /users/virag/keys/myserver-cert.pub
      ProxyJump jump
Host bastion
      #Used because HostName is unreliable as IP address changes frequently
      HostKeyAlias bastion.example
      User external 
Host jump
      HostName jump.example.com
      User internal 
      IdentityFile /users/virag/keys/jump-cert.pub
      ProxyJump bastion

Now imagine we have to manage a couple hundred environments across multiple cloud providers all over the country with OpenSSH configured in-house. (You may scoff at this, but we’ve heard these stories from customers.) It is impossible to rely solely on runtime commands while claiming to uphold a credible degree of security.

At this scale, effectively managing a fleet requires a conscious architecting of subnetworks, DNS, proxy chains, keys, file structures, and so on that follows predictable patterns and can be transcribed into ~/.ssh/ssh_config. Either that, or using Teleport.

Virag Mody joined Teleport in January of 2020, after co-founding a software code auditing company for Ethereum applications. He continues to learn about trending technologies and produces high quality written and video content. In his free time, Virag enjoys rock climbing, video games, and walking his dog.

—

New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.