speed up not controlled user authentication

[kisst]

In case the user is not a member of the LOCAL_MARKER_GROUP then the script will not return anything anyway so fall fast will speed up the authentication, my own test show significant changes.

$ time ssh non-iam-user-for-backup@my-server /bin/true

real	0m3.152s
user	0m0.037s
sys	0m0.016s
$ time ssh non-iam-user-for-backup@my-server /bin/true

real	0m0.368s
user	0m0.048s
sys	0m0.007s

[michaelwittig]

One scenario that I can imagine where this change breaks stuff is when users are created with other tools than aws-ec2-ssh. And I know some people doing this to get stable uid/gids also discussed in #46 . Let's see if someone responds...

[KusabiSensei]

Are we certain this isn't a premature optimization? To me, it does seem a bit like one.

What is the expected use case?

If it is as you seem to be implying, having a backup account on the instance in case IAM authentication fails... Why wouldn't a better solution be to configure sshd_config to have the desired authentication methods using Match directives? It would require no code changes, and would give you the flexibility to avoid the call out to AWS for users of your choosing.

If it is determined that this is a big issue, and we need to optimize it, why wouldn't the fix then be to change how the installer modifies sshd_config to only call the authorized_keys_command.sh script for users that are matched by using the group or some other mechanism? That, to me, would seem to be a better solution.

[kisst]

@KusabiSensei can you give a sample of that matching SSH config, what you suggest sounds sensible but not sure how would you implement it.

But the use-case is that the setup has users, which can't be added to the IAM,one of which connect every minute where few seconds delay makes a difference.
Plus for backup options we also need other users, as this setup has way too many element what can break and can't afford to be locked out.

[KusabiSensei]

@kisst For the SSH configuration, instead of globally setting AuthorizedKeysCommand and AuthorizedKeysCommandUser (as in the standard config with the installer), you would do something like this (The following is from an Ubuntu box, but the same should apply to other distros).

This implements option 2 from my first comment (Only calling the command for synced IAM users, which is in my opinion a better idea).

(As an aside: I don't believe that it rises to the level of changing the base behaviour in the code, however. The goal isn't to make login fast, it's to use the SSH keys stored in IAM).

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 1024
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin no
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile	%h/.ssh/authorized_keys
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PasswordAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes
Match Group iam-synced-users
    AuthorizedKeysCommand /usr/bin/authorized_keys_command.sh
    AuthorizedKeysCommandUser nobody

But the use-case is that the setup has users, which can't be added to the IAM, one of which connect every minute where few seconds delay makes a difference.

I could go into a long reply about this being a big red flag that raises a lot of questions, but I'll simply say that you may want to have a rethink as to why this "every minute connection" requirement exists for you. Perhaps there's a better way to accomplish what you are wanting to accomplish in your application (SQS, Redis, RESTful API, etc) running inside EC2. That may also increase your app uptime, and allow you to take advantage of other AWS features to increase your service availability and resilience.

[kisst]

but I'll simply say that you may want to have a rethink

Believe me I suggested against it, but it's out of scope to refactor the application, and however I do appropriate your concern about the bad architecture, I do have to say that this is not relevant for this topic.

I'm happy to close the PR and open a new one with SSH changes instead if the case that is the prefered method.

[KusabiSensei]

I would say that it is relevant, if you are looking to patch the code. I personally don't think this is a common enough (or on scale, large enough) problem to warrant a base patch. I would look to make sure that the base code does the simplest thing that works (which the sed susbtitutions do quite well).

---

I ran a quick test in us-east-1 (t2.micro, running Canonical's stock Ubuntu 16.04 AMI), to profile the authorized_keys_command.sh script (because that's what you are attempting to speed up. We should profile the script, rather than the network performance of the SSH client).

Skipping the IAM calls through a logic relief valve as you propose saves at most one half second (0.58s max) and an average of 0.38s while running the script (n=100). This is calculated by running the script with a user that doesn't exist in IAM (which runs only one of the two IAM calls).

A valid user passed to the script by sshd during auth and fetching a key (assuming one public key per IAM account) takes approximately 0.80s on average with a max of 0.88s for the script to return from the initial call (n=100). This is about twice the first set of numbers, because two calls are made with a valid user.

---

As explained, there is a way for you to achieve your goal by customizing off the base config. That customization may not be correct for everyone, however.

Since it's chasing after a small amount of time, I'm on the side of not introducing extra complexity where it's not actually needed under all use cases. That is why I called it a premature optimization originally (and the script profiling does appear to bear that out).

If @michaelwittig feels differently, then that is ultimately his call.

[michaelwittig]

@kisst in which AWS region are you running?

[kisst]

Not sure how is it relevant but the one where I first seen the issue is eu-west-1 but exist on all others I tried, eu-west-2 eu-central-1 and eu-west-3

[michaelwittig]

I guess it's relevant because iam is always in us-east-1. So @KusabiSensei sees lower overhead than we see from Europe because of network latency. I assume it's even worse for regions in asia...