Fixing SSH Key Issues in Linux

By Bo MorganLinux Basics

 

SSH keys are one of the most common ways engineers securely access Linux servers. They allow you to log into systems without typing passwords and are widely used in IT operations, automation, and infrastructure management.

However, beginners often run into problems when working with SSH keys. A common error looks like this:

Permission denied (publickey)

Or sometimes SSH simply refuses to connect even though everything appears to be configured correctly.

The good news is that most SSH key problems come down to just a few simple checks. Once you understand where to look, troubleshooting becomes very straightforward.

This guide walks through a practical, step by step process for identifying and fixing SSH key authentication problems.

Step 1: Confirm the SSH Key Exists

Before troubleshooting anything complicated, the first thing to check is whether an SSH key actually exists on your machine.

SSH keys are stored inside a hidden directory in your home folder called .ssh.

To see what files are there, run:

ls ~/.ssh

Example output:

id_rsa 
id_rsa.pub 
known_hosts

Two files are especially important.

The private key:

id_rsa

The public key:

id_rsa.pub

The private key stays on your computer and should never be shared.

The public key is copied to the server you want to access.

When SSH connects, the server checks whether your public key is listed as an authorized key. If it is, the server allows the login without asking for a password.

Real World IT Example

In professional environments, SSH keys are the standard method engineers use to access Linux servers.

Automation tools such as Ansible, CI/CD pipelines, and deployment systems also rely on SSH keys for authentication.

If these key files do not exist, SSH key authentication cannot work.

Step 2: Generate a New SSH Key (If Needed)

If your .ssh directory does not contain a key pair, you can generate one using the built-in Linux tool ssh-keygen.

Run the following command:

ssh-keygen

You will see output like this:

Generating public/private rsa key pair. 
Enter file in which to save the key (/home/user/.ssh/id_rsa):

Press Enter to accept the default location.

Next you will be asked to create a passphrase.

Enter passphrase (empty for no passphrase):

For simplicity, you can leave this empty and press Enter again.

Final output will look like this:

Your identification has been saved in /home/user/.ssh/id_rsa 
Your public key has been saved in /home/user/.ssh/id_rsa.pub

Now your system contains a private key and a public key.

Your private key remains on your computer.

Your public key is the file you distribute to servers that should allow you access.

Real World IT Example

When engineers set up a new workstation, generating an SSH key is usually one of the first configuration steps.

After generating the key, the public key is copied to any servers they need to manage.

Step 3: Copy the Public Key to the Server

Next, the server needs a copy of your public key.

The easiest way to do this is with the ssh-copy-id command.

ssh-copy-id user@server-ip

Example output:

Number of key(s) added: 1 
Now try logging into the machine with: 
ssh user@server-ip

This command automatically places your public key into the following file on the server:

~/.ssh/authorized_keys

This file lists the keys that are allowed to authenticate to the system.

Real World IT Example

When a new engineer joins a team, their public key is typically added to company servers.

Once the key is present in the authorized_keys file, the engineer can log in without using a password.

Step 4: Test the SSH Connection

Now we can test the connection.

Run:

ssh user@server-ip

Expected output might look like:

Welcome to Ubuntu 24.04

If everything is configured correctly, SSH will log you in without asking for a password.

This works because the server confirmed that your public key matches your private key.

Step 5: Use Verbose Mode if SSH Fails

If the connection still fails, the next step is to run SSH in verbose mode.

Verbose mode shows detailed information about the authentication process.

ssh -v user@server-ip

Example output:

Offering public key: /home/user/.ssh/id_rsa 
Server accepts key 
Authentication succeeded

If authentication fails, you might see:

Permission denied (publickey)

Verbose mode shows exactly what SSH is trying to do during login. This makes it much easier to identify the source of the problem.

Real World IT Example

Experienced Linux administrators rely heavily on verbose mode when troubleshooting SSH problems because it reveals what is happening behind the scenes.

Step 6: Check File Permissions

Incorrect file permissions are one of the most common SSH key issues.

SSH requires strict permissions on key files to prevent unauthorized access.

Check your permissions using:

ls -la ~/.ssh

Example output:

drwx------ .ssh 
-rw------- id_rsa 
-rw-r--r-- id_rsa.pub

SSH expects these permission levels.

Directory:

chmod 700 ~/.ssh

Private key:

chmod 600 ~/.ssh/id_rsa

Public key:

chmod 644 ~/.ssh/id_rsa.pub

If the permissions are too open, SSH will refuse to use the key.

Real World IT Example

When developers copy SSH keys from backups or other systems, file permissions sometimes change. Correcting permissions often resolves the issue immediately.

Step 7: Verify the Server Has Your Public Key

If everything looks correct locally, the next step is to check the server.

Log into the server using a password and inspect the authorized keys file:

cat ~/.ssh/authorized_keys

Example output:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC...

Your public key must appear in this file. If it is missing, the server cannot authenticate you.

Real World IT Example

In larger environments, this file is often managed automatically using configuration tools that add or remove keys for engineers as needed.

Common Beginner Mistakes

Here are three frequent mistakes beginners run into when working with SSH keys.

Using the Wrong Username

If you connect using the wrong user account, SSH will check the wrong home directory.

Example:

ssh root@server

But the key actually belongs to:

ssh ubuntu@server

Always confirm you are connecting with the correct user.

Incorrect Permissions

If permissions are too open, SSH refuses to use the key.

You may see errors such as:

Bad permissions on private key

Correct the permissions using chmod.

Copying the Wrong Key

Only the public key should be copied to the server.

Never share your private key.

The file you share will always end with:

.pub

A Practical Troubleshooting Workflow

Imagine a developer reports they cannot access a production server and receives this error:

Permission denied (publickey)

A practical troubleshooting workflow would look like this.

First, test the connection using verbose mode:

ssh -v dev@server

Next, verify the key exists locally:

ls ~/.ssh

Then check file permissions:

ls -la ~/.ssh

If everything appears correct locally, log into the server and inspect the authorized keys:

cat ~/.ssh/authorized_keys

In most cases, the root cause becomes clear during one of these checks.

This systematic approach solves the majority of SSH key authentication problems.

Optional Next Step: Using an SSH Agent

Once you become comfortable with SSH keys, a helpful improvement is using an SSH agent.

An SSH agent stores your key in memory so you do not need to repeatedly enter passphrases.

Start the agent:

eval "$(ssh-agent -s)"

Then add your key:

ssh-add ~/.ssh/id_rsa

This small improvement makes working with SSH throughout the day much smoother.

Conclusion

SSH key problems can feel confusing when you first encounter them. In reality, most issues come down to a few simple checks.

Confirm the key exists.

Verify permissions are correct.

Ensure the public key is present on the server.

Once you work through this process a few times, troubleshooting SSH authentication becomes routine.

Linux systems reward methodical troubleshooting, and SSH is a perfect example of that principle.