Guaranteeing scripts don't get hung on remote ssh

Date March 31, 2011

This is probably remedial for a lot of you, but it never occurred to me before today.

I've got a lot of shell scripts that run in cron or manually, and a significant portion of those need to connect to a remote server via ssh, either to rsync or to issue commands, or whatever.

The way I do this is to set up key-based authentication between the machines...but what if, for whatever reason, that doesn't work? Have you ever ran a script, and after 5 minutes, you realize it hasn't finished, because it's still sitting there yelling at you:

username@yourhost's password:

That's no fun...mostly because you're probably going to end up typing the password another dozen times (or more!) for each time that it has to connect to a remote host to which you haven't copied the keys.

Today, I learned and/or realized that you can specify options from 'man ssh_config' on the command line to ssh. That means you can avoid the mess above and more gracefully handle these exceptions. Instead of just

ssh user@remotehost command

Do this:

ssh -o PreferredAuthentications=publickey user@remotehost command

This causes the result to be

Permission denied (publickey).

Probably more importantly, it returns an error code (255 in this case), which you can handle in-script:

 if ! ssh -o PreferredAuthentications=publickey \
           $USERNAME@$HOST $COMMAND ; then
    echo "Sorry, key authentication failed for $HOST"
    exit 1
 fi

I'm not sure why it never occurred to me before, but it's a great way to account for those oddities which seem to crop up from time to time.

By the way, if you're unfamiliar with key-based authentication in SSH (or with SSH tricks in general), there are several posts from a few years back where some other bloggers and I covered various SSH tricks that you might be interested in.

Drop a comment and let me know how you handle these sorts of things. Your way is probably better than mine, and I'm interested in hearing about it!

  • http://www.ginsys.eu Serge van Ginderachter

    Strange that you have this problem. If you run them interactively, that is to be expected. If you run them 'headless', as eg in cron jobs, ssh will immediatley notice it doesn't have a proper tty for the password prompt, and return some error.

    I'm not sure in which cases you'd need your proposed solution?

  • http://www.standalone-sysadmin.com Matt Simmons

    @Serge

    In the cron script, I agree, you can test for errors and respond accordingly. It's when you're running a script from the command line, and it's looping through a series of hosts that it becomes irritating to type passwords again and again (and again).

    The case in point is the script I'm writing to deal with error ORA-01118: cannot add any more database files

    Basically, the script is going to need to connect to each of the standby database machines, verify the script suite is installed correctly, and if so, shut down the oracle instances, copy files to the destination, start the database back up, run SQL scripts, and then bring recovery to current.

    In the best-case scenario, each host has to be connected to twice. That's a lot of passwords to type, I'd much rather handle that prompt gracefully in my script. I'm using this technique so that my successor doesn't have to deal with it.

  • http://www.ginsys.eu Serge van Ginderachter

    O, that makes sense, I didn't get it you're running ths interactively.

  • http://itforme.wordpress.com/ Pete

    I’m the IT administrator for a software development company. We run a multi-platform continuous build system that requires very sophisticated communication between the nodes. Passwordless ssh across every machine under a variety of different users to orchestrate the different processes. I use Likewise Enterprise to join the *nix machines to the domain, then enable delegation on each one of those machines (to allow for nested ssh sessions). The windows users use a GSSAPI enabled version of putty. The result is all users and systems can ssh to eachother (multiple hops) without ever being prompted for a password. This has greatly reduces the overhead of manually having to generate and manage shared keys.

  • http://web.me.com/rick_gillette/home Rick Gillette

    I had the task to fix a script that automatically would login to any number of servers of Solaris, HPUX, and Linux of various flavors with different logins and passwords for any one user to do an automated OS/Monitoring/Console INVENTORY - some servers were part of a cluster, some part of a virtualized environment, some standalone.

    Not to go too long, this required a minimum of (2) scripts - One Bash which wrapped an Expect login routine based on who was running it - never as ROOT. The login problems with what you described were easily fixed by using this SSH option on the login portion of the Expect script:

    ssh -o StrictHostKeyChecking=no

    The reason we had to do this especially was that there were (3) separate possible logins based on what was handling the actual login authentication - Linux was different from AD, etc.

    Enjoy..

  • Alternative

    I've had this problem as well with some hosts. The solution was to use the -i parameter

    ssh -i ~/.ssh/ username@host

  • Alternative

    Something went wrong of course :) Here's the correct example:

    ssh -i ~/.ssh/[filename of key] user@host

  • http://holyhandgrenade.org Jeff Goldschrafe

    Also consider -o BatchMode=yes.

  • http://www.hollenback.net Phil Hollenback

    The problem is actually worse than this. In rare cases ssh connections can hang forever with no response, no matter what ssh connection settings you use.

    To guard against that, you need to wrap the ssh invocation in an alarm wrapper. In bash you can do something like this:


    TIMEOUT=30
    bash -c "ssh host 'date;uname -a' 2>&1" >$SshResult & \
    { sleep ${TIMEOUT}; eval 'kill -9 $!' &> /dev/null && echo timeout >> $SshResult ; }

    Fortunately that failure doesn't happen very often. It's mostly an issue in large installations of thousands of servers.

  • Pingback: Easy ssh configuration with .ssh/config file -bash - Amit Agarwal