draft-ylonen-sshkeybcp-00.txt Network Working Group T. Ylonen Internet-Draft SSH Communications Security Expires: August 22, 2013 G. Kent SecureIT M. Klein Bank of America M. Souppaya NIST February 18, 2013 Automated Access Using SSH Keys - Current Recommended Practice Abstract This document presents current recommended practice for configuring, managing, auditing, and associated policies around automated access to information systems, with particular emphasis on SSH user keys as authentication and authorization tokens but also looking into other automated access mechanisms, such as Kerberos. Starting with a review of authentication methods that can be configured for automated access, the document describes the risks involved when the management of automated access and SSH keys is neglected. It scopes the extent of the problem in particular organizations, provides a detailed roadmap for bringing automated access and SSH keys under control, and presents recommendations on continuous monitoring and ongoing management of automated access in information systems. Various remedial actions are presented and mapped to the problems they address and residual risks in the event the recommendations are not implemented. Guidance is also provided on how to organize management of automated access with the objective of reducing the system administration burden and organization operational cost, and on tools for automating the process. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Ylonen, et al. Expires August 22, 2013 [Page 1] Internet-Draft Automated Access Using SSH Keys February 2013 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on August 22, 2013. Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Purpose and Scope . . . . . . . . . . . . . . . . . . . . 3 1.2. Audience . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3. Structure of This Document . . . . . . . . . . . . . . . 4 1.4. Words Signifying Level of Requirement . . . . . . . . . . 5 1.5. Impact Levels for Information Systems . . . . . . . . . . 5 2. The Basics of SSH Protocol and Implementations . . . . . . . 7 2.1. The SSH Protocol . . . . . . . . . . . . . . . . . . . . 7 2.2. User Authentication in SSH . . . . . . . . . . . . . . . 8 2.2.1. Password Authentication . . . . . . . . . . . . . . . 8 2.2.2. Public Key Authentication . . . . . . . . . . . . . . 9 2.2.3. Kerberos Authentication . . . . . . . . . . . . . . . 9 2.2.4. Host-Based Authentication . . . . . . . . . . . . . . 10 2.2.5. Comparison of the Authentication Methods . . . . . . 10 2.2.6. Dangers of Unverified and Shared Host Keys . . . . . 11 2.3. Common Use Cases . . . . . . . . . . . . . . . . . . . . 12 2.3.1. Interactive Use . . . . . . . . . . . . . . . . . . . 12 2.3.2. Automated Access . . . . . . . . . . . . . . . . . . 13 2.3.3. File Transfers . . . . . . . . . . . . . . . . . . . 13 2.3.4. Point-to-Point Tunneling . . . . . . . . . . . . . . 14 2.3.5. Telecommunications Network Management . . . . . . . . 14 2.3.6. Additional Use Cases . . . . . . . . . . . . . . . . 14 3. Threats Arising from Poorly Managed Automated Access and SSH Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Ylonen, et al. Expires August 22, 2013 [Page 2] Internet-Draft Automated Access Using SSH Keys February 2013 3.1. Virus Spread Threat . . . . . . . . . . . . . . . . . . . 15 3.2. Unaudited Backdoor Threat . . . . . . . . . . . . . . . . 17 3.3. Leaked Keys May Provide Access for Extended Periods . . . 18 3.4. Keys Are Never Changed . . . . . . . . . . . . . . . . . 19 3.5. Lack of Proper Termination of Access . . . . . . . . . . 20 3.6. Use for Unintended Purpose . . . . . . . . . . . . . . . 21 3.7. Accidental Data Transfers and Human Errors . . . . . . . 21 3.8. Problem Under Radar . . . . . . . . . . . . . . . . . . . 22 4. Scoping and Auditing the Threat and SSH Key Management Situation . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5. Key Remediation Solution Planning and Deployment . . . . . . 26 5.1. Discovering SSH Keys and Trust Relationships . . . . . . 28 5.2. Moving Authorized Keys to Protected Locations . . . . . . 32 5.3. Monitoring Use of Trust Relationships and Keys . . . . . 32 5.4. Removing Trust Relationships That Are No Longer Used . . 34 5.5. Associating Trust Relationships with Application and/or Purpose . . . . . . . . . . . . . . . . . . . . . . . . . 35 5.6. Implementing Approval Process for Setting Up New Trust Relationships . . . . . . . . . . . . . . . . . . . . . . 36 5.7. Rotating Existing SSH User Keys . . . . . . . . . . . . . 38 5.8. Configuring Command Restrictions on Authorized Keys . . . 39 5.9. Configuring IP Address Restrictions on Authorized Keys . 40 5.10. Considerations for Software Tools . . . . . . . . . . . . 40 6. Continuous Monitoring and Management of SSH Keys and Automated Access . . . . . . . . . . . . . . . . . . . . . . 45 6.1. Continuous Monitoring of Automated Access . . . . . . . . 45 6.2. Creation of New Trust Relationships . . . . . . . . . . . 48 6.3. Removal of Trust Relationships . . . . . . . . . . . . . 48 6.4. Periodic Rotation of Trust Relationships . . . . . . . . 49 6.5. Facilitating Audits . . . . . . . . . . . . . . . . . . . 49 7. Reducing Cost and Improving Security by Automating Trust Relationship Setups . . . . . . . . . . . . . . . . . . . . . 49 8. Security Considerations . . . . . . . . . . . . . . . . . . . 49 9. Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . 51 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 58 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 59 1. Introduction 1.1. Purpose and Scope This document focuses on risks related to poorly managed automated access in information systems and particularly SSH user keys, and how to reduce the risks. It documents current best practice of managing SSH keys for cost-effectively minimizing the risks, and provides security policy recommendations and auditing guidelines relating to SSH keys and other automated access. It introduces the need for auditing the use of these automated mechanisms. Ylonen, et al. Expires August 22, 2013 [Page 3] Internet-Draft Automated Access Using SSH Keys February 2013 1.2. Audience This document is intended for information security policy makers, auditors, security managers, IT operations managers, system administrators, and others who are responsible for specifying, acquiring, testing, implementing, maintaining, and auditing identity and access management and SSH solutions. Portions of the document may be of interest to technically advanced end users and systems programmers involved with SSH and other automated access technologies. 1.3. Structure of This Document Section 1.4 specifies what certain words indicating level of requirement for compliance with this standard mean. Section 1.5 defines impact levels for information systems, including some important definitions relating to information systems having low impact themselves but having automated access to high-impact information systems. Section 2 summarizes the basics of the SSH protocol and implementations, with particular emphasis on authentication methods for automated access and typical use cases for automated access. Section 3 describes threats arising from poorly managed SSH user keys. Most of the threats are also relevant for other kinds of automated access. However, because of the ubiquity of SSH keys and the acuteness of addressing them the discussion focuses on SSH keys. Section 4 introduces simple metrics and questions that are useful in scoping the risks related to SSH user keys and gaining basic understanding of the size of the problem in an organization. The approach is suitable for both IT auditors responsible for verifying compliance with security policies as well as government and other policy makers wanting to measure the overall state of compliance and security across agencies. Section 5 introduces the process for detailed analysis of existing automated trust relationships and risks (with an emphasis on SSH user keys), as well as recommended steps for remediating the risks by relatively simple reconfiguration of existing keys, without necessarily having to install any new software (though use of software tools to assist the process will reduce the amount of manual work significantly, especially in larger environments). This section also discusses the specific threats addressed by each remediation step and risks involved in not implementing a particular step. Ylonen, et al. Expires August 22, 2013 [Page 4] Internet-Draft Automated Access Using SSH Keys February 2013 Section 6 provides recommendations for continuous monitoring and management of SSH keys and other automated trust relationships, as well as for auditing steps to be taken for ensuring that an organization keeps automated access under control after an initial remediation phase. Section 7 presents ways of reducing cost of managing SSH keys and improving security by reducing the number of system administrators who need root access for SSH key setups. Section 8 summarizes security considerations. Most of this document is about security and managing elements of security and access, and this section serves as a conclusion and summary of this document. Section 9 provides a glossary of technical terms used in this document. 1.4. Words Signifying Level of Requirement The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 1.5. Impact Levels for Information Systems The appropriate level of security and effort expended on security often depends on the level of impact from a failure or compromise of an information system. FIPS Publication 199 provides designations for impact levels on organizational information systems and a process for categorizing information systems. This document makes reference to the impact levels described therein (please see FIPS Publication 199 for exact definitions, this is just a simplifying summary): Low impact: Unauthorized disclosure, modification, destruction, or disruption of access have limited adverse effect on organizational operations, organizational assets, or individuals. Moderate impact: Unauthorized disclosure, modification, destruction, or disruption of access could be expected to have a serious adverse effect on organizational operations, organizational assets, or individuals. High impact: Unathorized disclosure, modification, destruction, or disruption of access could be expected to have a severe or catastrophic adverse effect on organizational operations, organizational assets, or individuals. Ylonen, et al. Expires August 22, 2013 [Page 5] Internet-Draft Automated Access Using SSH Keys February 2013 FIPS Publication 199 analyzes impact levels separately for confidentiality, integrity, and availablility. For the purposes of the current specification, the impact level of access to an account on an information system is taken to be the highest level of these three principles for the information system, since this specification primarily relates to operating system level access to information systems, and operating system level access can often be used to breach all three objectives simultaneously. Furthermore, experience has shown that once an attacker has operating-system level access to one user account on a computer, in a lot of cases various attack vectors (including bugs in system software and misconfigurations) can be utilized to escalate the access to high-level administrative access. That definitely compromises all three principles of information security. Configured trust relationships for automated access (e.g., using SSH user keys) may permit access from low-impact information systems to high-impact information systems without providing a password or other authentication credential from a user. This is particularly relevant if the authentication credential or authorized key permits access on the high-impact information system without restrictions on the commands that can be executed on the high-impact information system. In this case, access to the low-impact information system implies access to the high-impact information system. The information system owner inherently accepted this risk by allowing a low-impact system access to a high-impact system with providing compensation controls. Therefore, whenever a low-impact information system has a configured trust relationship permitting it to access a high-impact information system without a restriction on the commands that can be executed on the high-impact information system, the low-impact information system MUST be treated as having the impact level of the highest-impact information system that it can access using automated trust relationships. This implies that to avoid treating low-impact information systems as high-impact systems, there must be a well-defined boundary in the IT environment that trust relationships can only cross in the the direction allowing access from higher-impact systems into lower- impact systems, but not vice versa. If such boundary is relied on, it MUST be audited and continuously monitored to enforce its existence. Such a boundary could exist, for example, between development and production systems. It should be noted that several current SSH implementations (including OpenSSH) only permit configuring command restrictions for access based on SSH user keys. It is currently not possible to configure command restrictions for Kerberos-based authentication, Ylonen, et al. Expires August 22, 2013 [Page 6] Internet-Draft Automated Access Using SSH Keys February 2013 host-based authentication, hard-coded passwords, or passwords coming from password vaults, which has implications for the above requirement. Command restrictions are a compensation control that can be leveraged to minimize the exposure to the additional risks exposed to a high- impact system from allowing limited access to the hosted resources from a low-impact system. Command restrictions used for this purpose should be designed to be effective in limiting what actually can be done with the access, and should prevent interactive access and port forwarding. Furthermore, if a trust relationship has a command restriction that limits its use to file transfers but the directories from which files can be read or modified using it have not been restricted, it exposes the server to a more limited risk. The trust relationship may be used to read any file or directory on the server accessible to the user account used for file transfers, also outside the intended directories. It may also be used to write any file writable to the user account; it is fairly common to have configuration files on servers that are inappropriately world-writable. If a trust relation is restricted to file transfers but does not limit the directories that can be accessed, the originating information system SHOULD be considered as having at least the impact level of the highest-impact information system to which it has such access. 2. The Basics of SSH Protocol and Implementations SSH (Secure Shell) is a protocol and software tool for logging into a remote machine, executing commands remotely, and transferring files with a remote machine over a computer network. SSH can also be used for implementing a protected tunnel for delivering other services. SSH is very widely used for administering Linux and Unix systems, virtual machines, routers, firewalls, and other network devices. It is also embedded in many leading file transfer solutions, systems management solutions, identity management solutions, and privileged access management solutions. It is widely used for integrating IT systems and automating their operation. 2.1. The SSH Protocol The SSH protocol is an IETF standards track protocol and is described in RFC 4251 [RFC4251], RFC 4252 [RFC4252], RFC 4253 [RFC4253], and RFC 4254 [RFC4254]. Ylonen, et al. Expires August 22, 2013 [Page 7] Internet-Draft Automated Access Using SSH Keys February 2013 Many independent commercial and open source implementations are available, including Tectia SSH, OpenSSH, and many others. SSH is available from nearly all platforms, including Linux/Unix, Windows, mainframes, routers, telephone exchanges, mobile devices such as smartphones and tablets, various embedded devices, and many industrial automation systems. In the SSH protocol, an SSH client application initiates a TCP/IP connection over a network to a destination server, authenticates the remote server, and then sends a destination user name and authentication credentials to the server. Server authentication is done using host keys, and its primary purpose is to prevent man-in- the-middle attacks; however server authentication is beyond the scope of this document. 2.2. User Authentication in SSH The SSH protocol supports several mechanisms for authenticating users, including passwords, public key authentication, Kerberos, and host-based authentication. 2.2.1. Password Authentication There are two kinds of password authentication mechanisms in SSH: basic password authentication and keyboard-interactive authentication. The basic password authentication can only be used for traditional passwords, whereas keyboard-interactive authentication supports an interactive dialogue with a server-side authentication module, and can support various types of challenge- response systems and generally a broad range of authentication mechanisms. Keyboard-interactive authentication can implement any authentication method supported by PAM (Pluggable Authentication Modules) on Unix/ Linux/Mac servers, such as one-time password mechanisms and challenge-response systems. It could, for example, be used to implement SMS-based authentication as a second factor. Password authentication is commonly used for interactive users, but less commonly for automated access (through it is sometimes seen with hard-coded passwords in scripts and management systems, especially for managing routers and file transfers). Ylonen, et al. Expires August 22, 2013 [Page 8] Internet-Draft Automated Access Using SSH Keys February 2013 2.2.2. Public Key Authentication Public key authentication in SSH uses user keys to authenticate/ authorize the connection. The SSH client has an identity key, typically an RSA or DSA private key, and the server must have the corresponding public key configured as an authorized key for the destination user. The private key may be protected by a passphrase, in which case it is encrypted by a key derived from the passphrase (passphrases are common for interactive users but rarely used for automated access; experience has shown that the bulk of SSH user keys in most organizations are for automated access). Different SSH implementations use different file formats for configuring identity keys and authorized keys. Many widely used SSH implementations support configuring restrictions for SSH keys, including a forced command and/or a source restriction. These may be used, e.g., for limiting what can be done on the server using the key (command restrictions), and for limiting the source hosts from which the key can be used (source restrictions). Public key authentication is by far the most frequently used method of configuring automated access using SSH at the time of this writing, and represents best current practice. Identity keys and authorized keys are typically configured in a user's home directory under the ".ssh" or ".ssh2" subdirectory. 2.2.3. Kerberos Authentication Many organizations use Kerberos or Active Directory authentication with SSH. (Active Directory uses Kerberos internally.) Use of Kerberos (usually together with LDAP) or Active Directory usually serves several objectives: storing (interactive) user accounts in a centralized directory; sharing (interactive) user accounts on multiple operating systems; specifying access to resources based on Active Directory groups; and eliminating use of host keys. In practice, Kerberos is rarely used for non-interactive accounts. While there are some ways of configuring it, including keytab files, caching tickets, and using service processes to obtain tickets for Ylonen, et al. Expires August 22, 2013 [Page 9] Internet-Draft Automated Access Using SSH Keys February 2013 functional accounts, these approaches rely on having long-term credentials stored on the host or at least accessible to the process on the host that is obtaining tickets. These credentials can be exploited by an attacker largely in the same way as SSH keys, e.g., by using them to obtain a ticket granting ticket (TGT) for the functional account and using the ticket to gain access to other hosts or accounts that the functional account can access. One problem with Kerberos for automated access is that it is frequently used for implementing single sign-on automatically, and getting a Ticket Granting Ticket for one host implies the ability to log into many other hosts (often all hosts) in the domain without having to provide a password again. This means there are implicit trust relationships between the same account on all such hosts. 2.2.4. Host-Based Authentication Host-based authentication uses the source host's host key to authenticate the source host and to vouch for the identity of the user on the client side. At the same time, the host key securely identifies the source host. The private host key is only accessible to "root" (special privileged user) on the server, and ability to generate a digital signature by the private host key demonstrates the client process having access to the private host key of the host. Host-based authentication is rarely used and does not permit configuring command restrictions. 2.2.5. Comparison of the Authentication Methods All these authentication methods, when used for automated access, fundamentally rely on there being some information that is not accessible to anyone but a legitimate source user or root on the source host. For public key authentication this is demonstrated by reading the private key file; for Kerberos authentication by reading a keytab or ticket; and for host-based authentication by generating a signature including the client-side user name and host name using the client's host key. That said, public key authentication does allow encrypting the identity keys using a passphrase, which provides an extra layer of security. In practice, passphrases are rarely used for non- interactive access due to the complexity of configuring passphrases in scripts and the need to still store the passphrase somewhere. Also, the mechanism in public key authentication is simpler, which reduces the likelihood of programming errors. Ylonen, et al. Expires August 22, 2013 [Page 10] Internet-Draft Automated Access Using SSH Keys February 2013 Security of Kerberos authentication relies on the security of credentials between the host and the KDC as well as on the security of the KDC (Key Distribution Center). In non-interactive use, Kerberos authentication merely demonstrates access to a keytab file or a cached ticket (similar to public key authentication, even though the technical details are very different). KDCs are generally replicated and multiple KDCs usually synchronize information between themselves. It is thus fairly complex and involves a lot of code, which increases risk of programming and configuration errors and mismanagement. In interactive use, key loggers may be used to steal passwords used for obtaining Kerberos tickets (key loggers are common in malware that attacks laptops and desktops). A major advantage of public key authentication over the other methods is that it allows configuring a command restriction. The command restriction can be used for preventing virus spread and other attacks, as described in Section 3. For this reason, public key authentication SHOULD be used as the authentication mechanism for automated access with SSH. In practice, since public key authentication has been by far the easiest to configure and understand, and supports command restrictions, it has become the most widely used and current best practice method for configuring automated access. Use of password authentication (with hard-coded password) for automated access SHOULD NOT be used, because such hard-coded passwords may be obtained by attackers and used for furthering an attack to other systems. Generally it is very hard in practice to enforce proper rotation for hard-coded passwords. Also, even if passwords are stored in a password vault in a way that makes them accesible to a script, an attacker might modify the script to extract the password(s) from the vault for furthering an attack. 2.2.6. Dangers of Unverified and Shared Host Keys Many file transfer and systems management applications do not check host keys for hosts that they connect to and authenticate to using passwords. This permits a man-in-the-middle attack to be performed in the network (many tools are available for this and any device connected to a network through which the connection goes can be used for the attack - including, e.g., reprogrammed smart switches). A man-in-the-middle attack can be used by an attacker for obtaining the authentication passwords in such cases. Ylonen, et al. Expires August 22, 2013 [Page 11] Internet-Draft Automated Access Using SSH Keys February 2013 With Kerberos authentication, the Kerberos ticket can also be obtained by a man-in-the-middle attack, and together with IP address spoofing as the destination host be used to obtain a ticket granting ticket for the source user. Man-in-the-middle attacks are a risk regardless of authentication method if hosts keys are not properly verified. The attack permits injection of arbitrary commands into the session, and reading and modifying any transferred files (including injection of bogus file transfers). A successful man-in-the-middle attack from the network is as good as being able to use a trust relation leading to the destination host. Such man-in-the-middle attacks are practical, and are exploited in freely available attack tools, malware, as well as security software from multiple vendors for co-operative auditing purposes. Besides some applications not checking host keys, there are also some large enterprise that share the same host key on thousands of machines (for example, one Fortune 500 company is known to use the same host key on 14000 computers at the time of this writing). If any of the computers is compromised, they all become vulnerable to man-in-the-middle attacks. Therefore, while this document is not really about host keys, the destination host MUST be properly authenticated by the client for all automated access. An exception may be made for the very first connection to the server to simplify system administration. 2.3. Common Use Cases 2.3.1. Interactive Use SSH has become the standard used by system administrators for configuring and managing Unix and Linux computers, routers, and various other equipment remotely. It is also widely used by software developers, and in some organizations by ordinary end users for running applications remotely (particularly text-based legacy applications). Public key authentication is also sometimes used by system administrators and advanced end users (e.g., developers) for legitimate purposes in interactive use. Sometimes it is used as a single-sign-on mechanism. Sometimes it is used to control access rights among a large pool of system administrators to access, e.g., systems at retail locations through a number of jump accounts. Ylonen, et al. Expires August 22, 2013 [Page 12] Internet-Draft Automated Access Using SSH Keys February 2013 2.3.2. Automated Access SSH is very frequently used for automated access between systems. Such automated access is necessary for cost-efficiently managing large IT environments, for integrating applications, and for cost- effectively provisioning virtual machines in cloud services. Automated access refers to accessing a computer from another computer in an automated fashion. Automated access may be unrestricted, allowing any commands to be executed, or may be limited to specific commands or operations, such as file transfers (perhaps limited to a specific directory). Automated access is most commonly used with functional accounts, system accounts, service accounts, and other non-interactive user accounts (sometimes also called non-user accounts; they all mean more or less the same with some local variation from organization to organization). Such accounts are used by operating systems, middleware, databases, and applications for running processes and allowing automated system components to communicate with each other. System or service accounts are likely to have sensitive levels of access to system resources and are subject to compliance requirements. SSH is very frequently used for automated access between such accounts. Automated access using SSH is common also in Windows and mainframe environments, especially for file transfer applications. There are also various native mechanisms on Windows that can be used for automated access, but such mechanisms are beyond the scope of this document. Automated access has been largely ignored in Identity and Access Management. It is illustrative that in a large international bank with about 100,000 employees more than 1,5 million automated trust relationships were found. There were about 15 times more entry points to production systems based on automated access than there were traditional user accounts. It has been found that many organizations have many more entry points for automated access than they have interactive users. It is clear that such entry points cannot be ignored. Most trust relationships for automated access today are configured using SSH user keys. 2.3.3. File Transfers SSH is frequently used as a file transfer tool in itself, using the "scp" and "sftp" tools. The SFTP [SFTP] protocol is gaining Ylonen, et al. Expires August 22, 2013 [Page 13] Internet-Draft Automated Access Using SSH Keys February 2013 popularity in commercial and open source file transfer products, and a substantial fraction of the world's file transfers now use the SFTP protocol. Automated file transfers using SSH typically use public key authentication or hard-coded passwords. It is very common that access on the server is not in any way restricted to file transfers, and the authentication credentials could also be used for other purposes, including running arbitrary commands. Thus, the internal databases of file transfer applications (including scheduling applications) often contain access credentials for production servers. SFTP file transfers are increasingly used between organizations using automated trust relationships that cross organizational boundaries. It is extremely important to control what can be done on the server using such trust relationships! 2.3.4. Point-to-Point Tunneling SSH is frequently used for point-to-point tunneling for simplifying PCI DSS compliance. This usage case is usually fully automated, and typically uses public key authentication. 2.3.5. Telecommunications Network Management SSH is used massively for managing telecommunications networks and Internet infrastructure. Many if not most core routers in the Internet are configured using SSH. Many operators configure also edge routers and even customer premises routers and xDSL equipment using SSH. Lots of telecommunications equipment for LTE (Long Term Extension) and other telecommunications technologies also relies on SSH for configuration. Both public key authentication and passwords are used for authentication in telecommunications networks. Often, the authentication credentials (e.g., device passwords) are stored in one or more management system databases. Often, such management systems are also connected to office, customer service, and retail partner systems for provisioning access in real time and diagnosing problems. 2.3.6. Additional Use Cases SSH with automated access is also widely used for the following and many other purposes: systems management tools that automatically configure and monitor various systems; Ylonen, et al. Expires August 22, 2013 [Page 14] Internet-Draft Automated Access Using SSH Keys February 2013 logging into end hosts from privileged access management systems; configuration and vulnerability management systems; provisioning user accounts in identity management systems; various backup tools; disaster recovery systems and their control; emergency response systems; incident response systems; license audit tools; various IT audit tools; various tools for executing scripts on multiple servers; and IPMI and other BIOS-level management interfaces. 3. Threats Arising from Poorly Managed Automated Access and SSH Keys This section outlines some of the threats that have been identified with poorly managed SSH keys. The guidelines and recommendations of this document are intended to address these while minimizing the administrative burden from managing the keys. Several of the problems are present with many technologies for automated access. The issues must be addressed regardless of technology. 3.1. Virus Spread Threat A cyber weapon or virus can be engineered to use SSH keys to spread to nearly all servers within an organization once it has penetrated one server. This ruins layered defense, and experience has shown that viruses frequently manage to penetrate individual servers in an organization. A cyber weapon would likely use multiple attack vectors to penetrate an organization, and could use SSH keys (or other trust relationships such as Kerberos) to spread throughout the organization's server infrastructure in minutes after penetrating the first server. With this attack, a virus scans the system for all accessible SSH private keys (identity keys), and tries to log into other servers using these keys. Once in an account, the virus may try to escalate Ylonen, et al. Expires August 22, 2013 [Page 15] Internet-Draft Automated Access Using SSH Keys February 2013 privileges using "sudo" or vulnerabilities it identifies with the system. Once root or "Administrator", the virus can read private keys from all user accounts on the system. The virus can scan the local network for SSH servers and try each key with each server in turn. It may also first scan all scripts on the host (e.g., cron jobs and daily scripts), and extract a set of hosts to try first. The Morris worm in 1988 utilized automated trust relationships for automated access to spread in a similar manner (at that time based on ".rhosts" authentication). This attack vector can be very powerful, and its importance is increasing as systems management becomes more automated. More broadly, a recent IBM X-Force study found most attacks against Unix/Linux servers utilize SSH. Many computer forensics experts are aware of cases were SSH keys have been used to spread an attack from one server to another, and several high profile incidents in the last year have used SSH keys as an attack vector. Experience has shown that most large organizations have on the average 3 to 100+ SSH keys configured granting access to each Unix/ Linux server. Some keys grant high-level administrative access or access to sensitive accounts, such as those storing database files or critical software. The mesh of automated access relationships is so dense in many cases that it is likely that an attack can spread to nearly all servers in an organization after penetrating the first few servers, especially if other attack vectors are used to escalate privileges. Implementing SSH keys as an attack vector in a virus is quite easy, requiring only a few hundred lines of code. Once inside a server, a virus may open a backdoor, steal, alter, or corrupt data, subvert encryption systems or databases, or outright destroy the server. In the worst case, a virus or cyber weapon using multiple attack vectors could spread globally, Internet-wide, enterprise-wide in a matter of minutes. Combined with destruction technologies it could wipe out on-line data, hot/warm stand-by disaster recovery systems, and backups accessible via tape robots and render most servers in enterprises inoperable by wiping their operating systems, storage (including network drives), and reprogramming flash memories, device firmware, and configuration memories. The worst case could mean life with no functioning banks, retail chains unable to process payments or supply inventory, logistics companies unable to operate effectively, most government agencies down, gas stations without gas with refineries having shut down and logistics being very inefficient, no or limited telecommunications, Ylonen, et al. Expires August 22, 2013 [Page 16] Internet-Draft Automated Access Using SSH Keys February 2013 and significant parts of power generation and distribution networks down, for weeks, months, or even years. Of course, SSH keys would likely only play a small part in such an attack, but poorly managed automated access removes much of the defense in depth that is critical in limiting attacks, and thus can vastly expand the impact of such an attack. The virus spread threat can be reduced by combining several approaches: Mandating forced command restrictions for as many trust relationships as possible. Removing trust relationships that are no longer needed. Minimizing the number of trust relationships leading to root accounts (directly or indirectly). Minimizing implicit trust relationships arising from privilege escalation (e.g., "sudo"), single-sign-on (e.g., Kerberos), and host equivalence. 3.2. Unaudited Backdoor Threat Many large organizations mandate that all provileged access to their servers take place through a privileged access management system that frequently also records privileged access. However, widely used privileged access management solutions are based on a gateway host that an administrator logs into, and then logs into the end server. Key-based access (and other automated trust relationships) can be used for creating backdoors that bypasses such privileged access management systems. System administrators regularly gain access to various accounts in the course of legitimate administration work. An administrator may add a new authorized key to an account with a single command (e.g., "echo ...keydata... >>~/.ssh/authorized_keys"). As of this writing, most organizations never audit authorized keys for their user accounts, and thus the added key may remain unnoticed for years. Such a key can then be used to log into the account using any SSH client, bypassing the privileged access management. It thus provides a relatively permanent unaudited backdoor. Key-based backdoors can also circumvent password vaults and systems that change root (or other privileged account) passwords regularly. Ylonen, et al. Expires August 22, 2013 [Page 17] Internet-Draft Automated Access Using SSH Keys February 2013 Experience has shown that many organizations have no control or tracking of trust relationships for automated access. Any system administrator can create and install a user key pair as needed without any reporting, logging, or authorization. Needless to say, such practice undermines traditional controls for privileges access. The unaudited backdoor threat can be reduced by the following: Prevent non-root/non-Administrator users from granting automated access to accounts without proper approval. For example, move all authorized keys files to root-owned directories that are not writable by normal users. Continously monitor the environment to detect unauthorized trust relationships configured by someone having root access. Require proper explanation and valid purpose for the existence of every trust relationship and remove any unused trust relationships. Use privileged access management systems that capture SSH and other protocols using automated access on the network level or at the server. Enforce privileged access management also for connections using automated trust relationships. 3.3. Leaked Keys May Provide Access for Extended Periods At the time of this writing, most large enterprises, government agencies, and healthcare providers do not track which SSH keys their users, administrators, backup operators, and cleaners may have had access to and copied over the years, and never change their SSH keys. This means that anyone who may have obtained a copy of a key (e.g., by copying it from a host, accessing a backup, or having acquired some decommissioned equipment that was not securely wiped) may gain access to production servers in the organization. Identity keys can easily be taken out from an organization on a single piece of paper, or by taking a photograph of a screen using a cellphone. The problems created by leaked keys can be reduced by: Rotating all keys regularly. Configuring source restrictions for authorized keys, making it more difficult to exploit copied identity keys. Ylonen, et al. Expires August 22, 2013 [Page 18] Internet-Draft Automated Access Using SSH Keys February 2013 Using certificate-based authentication, which can provide revocation and expiration, but is cumbersome to manage (and still does not protect from some of the other threats). Using Kerberos autentication, which allows terminating access to the account; however, Kerberos authentication does not in itself prevent leaking keys that have not been changed to be used. No audit or discovery process can ever guarantee to find all copies of identity keys, as they are small files that could be hidden anywhere, and there could be copies on laptops, tablets, USB sticks, offline, and even on paper (they are small enough to be typed in). Even if source restrictions are configured for authorized keys, an attacker could later copy the identity key to one of the allowed hosts (possibly to a user account that is not permitted). Still, source restrictions can reduce the potential for abuse of leaked identity keys. Rotating all keys regularly is the primary guarantee of eventual termination of access using copied keys. Using Kerberos for automated access is sometimes proposed as a solution. However, it comes with several problems; with Kerberos, some kind of long-term authentication token is still stored on the host (e.g., a credential or cached long-term ticket in a keytab file). A ticket or credential in a keytab file is no more secure than a private key in a file; either must be readable to the user account using them and either is readable for root. 3.4. Keys Are Never Changed Most security policies and regulations mandate that all passwords must be changed regularly, e.g., every three months. Some security standards mandate that encryption keys must also be changed regularly. However, very few security policies at this time make it explicit that authentication/authorization keys should also be regularly changed. In a sense, authentication keys are even more critical than encryption keys, because once access to a user account has been gained, it is generally possible to access and modify any data for that user account - including reading and modifying memory of processes running under that user account and/or modifying any executable programs owned by that user account, thus subverting encryption systems and other critical applications. Most environments at the time of this writing do not use source restrictions on authorized keys. Therefore, a leaked key may be used Ylonen, et al. Expires August 22, 2013 [Page 19] Internet-Draft Automated Access Using SSH Keys February 2013 from any computer or network within the organization (unless limited by internal firewalls). The problem of keys never being changes can be addressed by: Rotating all keys (and other authentication credentials) regularly, especially those used for automated access. Configuring source restrictions or authorized keys (this somewhat reduces the risk without really addressing the issue). Besides rotating keys at regular intervals to avoid their leakage and to limit the duration of the exploitation window should they leak, this also applies to credentials for used for obtaining actual authentication credentials; for example, it is not enough to periodically obtain a new Kerberos ticket - one must also regularly change the authentication credentials used for obtaining an initial ticket. It is also not enough to issue a new certificate for the same private key - the private key must also be replaced by a new generated private key. 3.5. Lack of Proper Termination of Access At the time of this writing, most organizations do not track what each key is used for. Most organizations do not systematically remove keys when they are no longer needed, because they don't know what application each key is related to, and they don't know what might break if they remove a key. Removing a key that is needed could break critical production systems, and so system administrators generally don't do it and auditor's haven't been auditing them. Many organizations have used manual spreadsheets for tracking key usage. However, experience has shown that such spreadsheets are frequently out of date and inaccurate, have not been maintained throughout the organization. Many organizations have no monitoring of automated access whatsoever. Practically no organizations track which automated credentials each system administrator may have had access to. An administrator may have copied any such access credentials, and may continue to have access through such credentials even after termination of his/her normal user account. If keys are never changed, such access can continue for years after termination of employment. Proper termination of access can be ensured by: Preventing creation of backdoors. Ylonen, et al. Expires August 22, 2013 [Page 20] Internet-Draft Automated Access Using SSH Keys February 2013 Regularly rotating keys (ensures termination of access by copied keys latest at next key change). Optionally triggering immediate key rotation for private keys on accounts accessible by the person whose access is being terminated. 3.6. Use for Unintended Purpose Holes are commonly made in firewalls for file transfer purposes. When the file transfer is using SSH (or SFTP), it is important that a forced command be used on the server to ensure that the hole in the firewall cannot be used for other purposes, such as executing shell commands on the server. Another related use case is employees creating their own backdoors into the enterprise to circumvent corporate policies against uncontrolled remote access by opening an SSH connection from the office to their home machine with a port forwarding from the home machine back to the office machine. Such backdoors may provide hackers an entry point into the company intranet, especially if the home machine is compromised and the user's password obtained using, e.g., key logger malware. Various commercial products are also available for auditing SSH connections at a firewall to enforce that such holes are not used for unintended purposes regardless of server configuration. Various SSH implementations also permit port forwarding even when forced commands are used. Therefore, a trust relationship that is configured for file transfer only may actually be used to obtain a connection to any port at any host on the internal network (or external network, for hiding the source of an attack). The threat of uninteded use can be minimized by: Allowing SSH/SFTP through a firewall only when required and restricting sources and destinations to fewest systems required. Configuring forced commands for authorized keys used by external parties. Implementing tools to audit SSH connections at the firewall and monitoring use to ensure that access was not misused. 3.7. Accidental Data Transfers and Human Errors Ylonen, et al. Expires August 22, 2013 [Page 21] Internet-Draft Automated Access Using SSH Keys February 2013 Not all risks associated with unmanaged automated access arise from malicious behavior. If there is automated access from non-production systems to production systems, data may accidentially be copied from non-production systems into production systems, where the incorrect data may cause substantial loss of money. This actually happened at a major bank and was one of the triggers behind their key remediation project. People are also known to make human errors when manually setting up new trust relationships. For example, it is fairly easy for a security administrator to accidentally copy an authorized key to the root account on a host instead of some other account that was intended. Such errors can go undetected for years. It is also known that some key setups involve thousands of hosts. It is easy to miss one or more hosts when copying an authorized key to so many hosts manually. Debugging such errors can be very time consuming. Also, when manually fixing such problems, security administrators are likely to just copy the missing keys to the proper accounts, without, for example, checking whether they have accidentally been copied to the root account. 3.8. Problem Under Radar The SSH key management problem has been recognized in various circles for some time. The scope of the problem, and its relation to automated access overall, however, has not been widely understood. The problems have remained under the radar because they are deeply technical and obscure, in the domain of system administrators. Each system administrator typically only sees a small corner of the IT environment, and does not have a full picture. Administrators and their managers are often so busy that while they may recognize that there is a problem, they simply have not had time to analyze the scope or possible implications of the problem. There have also been no guidelines or training materials on how to address it. Most IT auditors do not have SSH key management or automated access more generally on their checklist, yet it is central to identity and access governance given the prevalence of automated access entry points to systems. SSH keys, or control of credentials for automated access more generally, has not been sufficiently highlighted in implementation and auditing guidelines for FFIEC, SOX, PCI, FISMA, HIPAA, NERC, or COBIT. Even many CISOs are only vaguely aware of the problem, and many CIOs and IT risk management professionals have never heard of it. Ylonen, et al. Expires August 22, 2013 [Page 22] Internet-Draft Automated Access Using SSH Keys February 2013 Training, books, and systems in the identity and access management space have largely only dealt with actual human users and control of interactive access by people. Automated access by machines has been largely ignored, despite many organizations now having many times more credentials for automated access to their systems than they have interactive accounts. Yet, at the same time, poorly managed SSH keys represent an existential risk for many major banks and enterprises and a systemic risk for the banking system and developed countries as a whole. To get addressed, the problem needs to be brought into the attention of IT security auditors, IT operations managers, security architects, CISOs, and IT risk management professionals. It must be addressed in security regulations, guidelines, standards, and internal security policies. A lot of education and training will be needed. 4. Scoping and Auditing the Threat and SSH Key Management Situation Addressing threats related to automated access and SSH keys begins with understanding the extent to which automated access and SSH keys are used in an organization, understanding how they are managed, and identifying areas likely to require further attention. SSH key management is generally relevant for organizations where at least one of the following is true (the list is not exhaustive, and other automated access technologies affect other organizations): significant number of Unix or Linux systems; significant use of SSH or SFTP on Windows or Mainframe; virtualization or cloud services that are internally managed using SSH (possibly inside automated scripts/tools/frameworks); web server farms that are managed over SSH; network devices (e.g., routers, switches, xDSL models, firewalls) configured and managed using SSH and/or automated management systems; significant number of automated file transfers using SFTP; password management or privileged access management tools using SSH to connect to end servers; or systems management tools using SSH to communicate with managed systems. Ylonen, et al. Expires August 22, 2013 [Page 23] Internet-Draft Automated Access Using SSH Keys February 2013 Many organizations have uncontrolled trust relationships for automated access between their development and test environments (often thousands of hosts) and their production servers. Often test and development systems are threated as low-impact systems, yet they may be able to access production systems without interactive authentication, or may be used to product code and software distributions that will be copied to production systems for execution. Results of the scoping phase help estimate risk exposure and compliance with mandatory regulations, and help evaluate whether a more detailed discovery and remediation project is warranted. Certain types of relatively easily obtainable information are useful in understanding the scope of the problem in an organization. The information may easily be obtained as part of an audit or regular review. The following metrics generally give insight into whether an organization is affected by the issue: total number of authorized keys in the environment; total number of authorized keys in the environment that grant access to a root account (any account with user id 0); total number of authorized keys in the environment that grant access to a system account or service account; total number of authorized keys without a forced command ("command" option in OpenSSH) (relates to virus spread risk); and total number of non-root service accounts or system accounts where a system administrator logging into the account can add new authorized keys for the account. There are also a number of scoping questions that where the answers can give insight into the severity of the problem: Does installing a new authorized key require approval from a second person 1) for keys granting root access 2) for keys granting access to non-root service accounts or system accounts 3) for keys granting access to interactive user accounts? How such approvals enforced? (Rationale: Is there any control in place? Might ask separately for high-impact, moderate-impact, and low- impact systems) Ylonen, et al. Expires August 22, 2013 [Page 24] Internet-Draft Automated Access Using SSH Keys February 2013 Are non-root users technically prevented from installing new authorized keys, e.g., by moving the authorized keys files to root-owned locations? Has this been verified to be the case 1) on all high-risk systems 2) on all moderate-impact systems? (Rationale: this is basically asking "Can users leave key-based backdoors?" - and a policy without enforcement is not effective) Is a continuous monitoring process in place for detecting authorized keys that are added for a user account without proper authorization 1) for root accounts 2) for service and system accounts 3) to interactive user accounts? (Rationale: root users cannot be technically prevented from adding new keys to any account, and thus continuous monitoring is needed to detect such violations) Is a policy in place for rotating SSH user keys? Has it been verified that all user keys have actually been rotated in accordance with the policy (and the private keys actually changed)? (Rationale: without rotation, copied keys remain valid forever, and some people have been known to recertify or reuse the same private key because it may pass audit and saves trouble, without actually addressing the problem) Has the reason of existence for every authorized key, including the application or business process it relates to, been documented 1) on high-impact systems 2) on moderate-impact systems 3) on low- impact systems? (Rationale: access to sensitive data should be controlled. Authorized keys provide access to systems and data. Thus their existence should be justified and controlled. Furthermore, without proper documentation it is difficult to know when to remove a key, such as when an application is decommissioned or a business process changed.) Are SSH keys systematically removed when they are no longer needed? (Rationale: very few organizations currently ever remove SSH keys, and copied keys may continue to provide access for many years to anyone who has had access to the systems. Furthermore, the more keys there are, the higher the virus spread risk.) Are SFTP file transfers used between the organization and other organization? Do all authorized keys used for external file transfers have a forced command restriction? (Rationale: such keys involve high risk of attack spread/virus spread between organizations) Are routers, telecom equipment, or other network devices in the organization managed using SSH (perhaps using a management system tool)? How are access credentials for routers managed? Ylonen, et al. Expires August 22, 2013 [Page 25] Internet-Draft Automated Access Using SSH Keys February 2013 The scoping information can be obtained relatively easily and scoping questions can be answered without having to install new software on servers. However, the answers are only approximate, and experience has shown that many organizations do not have any single person with a clear answer to the questions and that sometimes management perceptions do not match reality. Therefore the answers are best obtained and analyzed as part of a regular audit that actually verifies the answers, at least by representative sampling. In Unix/Linux environments, it is often possible to get a rough understanding of the scope of the problem by counting authorized keys in authorized keys files for every user and answering the above questions based on readily available information. A very quick way to estimate the total number of authorized keys on every Unix/Linux server is to use something like the following (as root) on all hosts and summing up the results: "find / -name authorized_keys -print | xargs cat | wc -l". If the outputs of the above are stored in a file, one per line, they can be easily summed by: "awk '{sum += $1;} END {printf("Total authorized keys: %d\n", sum);}' security risk). Some software tools may support a combination or mix of these approaches. Will it run as root on managed hosts, or can it use a non-root account and "sudo" (or equivalent) to perform limited operations as root? Ylonen, et al. Expires August 22, 2013 [Page 43] Internet-Draft Automated Access Using SSH Keys February 2013 Using non-root account and "sudo" allows the operations performed on hosts to be controlled and monitored, and reduces risk from management system compromise. Is the solution able to retry discovery, key setups, etc. on hosts that are down or unreachable at the time of the initial attempt? How does the solution cope with poor network connectivity? Does the solution support limiting updates on hosts or groups of hosts to maintenance windows for those hosts (e.g., based on host, host group, application, or business process)? If there are lock- down periods in the organization when changes are not allowed (e.g., before Christmas), is this supported by the solution? Does it support user accounts stored in LDAP or Active Directory? How does it prevent crashing LDAP or Active Directory servers by reading directory contents from all servers simultaneously? Can it discover keys from directories that are not readable by root (e.g., NFS directories using the "root_squash" option)? Does it work with SELinux, if such support is needed? Will key management be needed for devices other than general- purpose computers (sometimes called non-OS devices), such as routers, BIOS management ports - and are such devices supported by the solution? Will the solution copy private keys? Various security standards forbid making multiple copies of cryptographic keys. How can the solution save operational costs in SSH user key management in the organization? Have existing user key management costs been estimated on an annual level? Has the solution actually been used to manage SSH user keys in a sufficiently large production environment? What references are available for the SSH user key management functionality? What kind of support/assistance can the vendor provide for scoping, planning, implementing, and operating an SSH key remediation project and ongoing operation of automated access? What is the vendor's roadmap and how does it tie to the overall IT strategy of the enterprise (SSH environment, broader management of automated access, other identity and access management systems, and auditing of privileged access)? Ylonen, et al. Expires August 22, 2013 [Page 44] Internet-Draft Automated Access Using SSH Keys February 2013 6. Continuous Monitoring and Management of SSH Keys and Automated Access Ongoing operations for management of automated access include: continuously monitoring the environment to detect unauthorized changes to trust relationships used for automated access and other suspicious activity; creating new trust relationships (with proper approvals and restrictions); removing trust relationships that are no longer needed as the IT environment evolves; periodic rotation of credentials (e.g., SSH user keys) used for trust relationships; and providing information needed for audits. 6.1. Continuous Monitoring of Automated Access Proper management of automated access required continuous monitoring of the IT environment, because system administrators operating as root may add new trust relationships for any user account. Continuous monitoring is also prudent for detecting keys that are no longer used, identifying external keys, and identifying changes in the patterns of usage of automated access. Ideally, continuous monitoring is a real-time or near-realtime process. For some aspects of it, hourly or daily analysis would generally be perfectly sufficient. On the other hand, if implemented manually using audits, cost constraints may limit continuous monitoring to annual audits. Even when continuous monitoring is performed using software tools, auditors SHOULD do some random sampling and testing annually to verify that the continuous monitoring tools are working properly. In some respects, continuous monitoring resembles discovery. Configured SSH user keys and trust relationships need to be discoverd throughout the environment, and various checks need to be performed on them. Alerts, audit findings, or reports may be produced based on the results of the checks. Continuous monitoring MUST discover every trust relationship and authorized key throughout the managed IT environment. Ylonen, et al. Expires August 22, 2013 [Page 45] Internet-Draft Automated Access Using SSH Keys February 2013 For each found authorized key: if it is on a moderate-impact or high-impact host, it MUST be verified that properly authorized trust relationship exists justifying the existence of the authorized key, or a special authorization exists for just the authorized key indendent of trust relationships (such special authorizations SHOULD be listed in annual audit findings and their continued existence SHOULD be rejustified annually); have a command restriction (also when a pseudo-tty is requested) that matches the command specified approved for them (for trust relationships leading into low-impact systems it is sufficient to just check the existence of a command restriction), unless sufficient approval is found to waive the requirement for a particular authorized keys/trust relationships. For each identified trust relationship, including those involving authorized keys, the following must be done: if the trust relationship leads from a lower-impact host to a higher-impact host and has no command restriction (or the command restriction is deemed ineffective in what can be done with it), then the lower-impact host must be reclassified as having the impact level of the higher-impact host; for trust relationships leading to moderate-impact or high-impact hosts, it MUST be verified that a proper approval exists for the trust relationship and that a proper justification, application, or business process is specified for it; if the trust relationship crosses a specified internal boundary in a manner that requires special approval, existence of such special approval MUST be verified (such bounrary-crossing trust relationships SHOULD be listed in annual audit findings and their continued existence SHOULD be rejustified annually); and it MUST be checked whether the trust relationship has been used for a configured amount of time (at most 12 months), and if not, the trust relationships MUST be removed unless special justification for their continued existence is provided (any trust relationships that have not been used for 12 months SHOULD be listed as audit findings and their continued existence SHOULD be rejustified annually). The age of every authorized key MUST be verified, and key rotation MUST be triggered for all authorized keys that exceed a maximum age specified by policy that are not used for external connections (the Ylonen, et al. Expires August 22, 2013 [Page 46] Internet-Draft Automated Access Using SSH Keys February 2013 same SHOULD be done for authorized keys used for external connections), unless a special waiver is recorded for not rotating a particular key. Such special waivers SHOULD be listed in annual audit, and should be rejustified annually. Keys used for external connections that have not been rotated in the last 12 months SHOULD be listed in annual audit reports. As part of an annual audit, for moderate-impact and high-impact systems, if use of privileged access auditing systems are otherwise required for such systems, it SHOULD be verified that privileged access auditing cannot be bypassed by bypassed using SSH user key based access. The main rationale for the continuous monitoring of the environment and annual audits and requiring reporting and revalidation of certain aspects of automated access annually is to enforce proper policy (policies usually do not get implemented if their implementation is not controlled or if waivers are available). However, IT environments are complex and sometimes there is a need to have automated access relationships for special purposes that would not otherwise be advisable. Special approvals can be used for implementing such special cases, but they should be revisited annually. Failure to use forced commands risks the organization to virus spread and loss of defence in depth. Incidents involving viruses or cyberweapons could involve many thousands of hosts, and even if they are individually low-impact hosts, the overall impact could be very significant. Therefore command restrictions should be enforced even for low-impact systems in most cases. They should especially be enforced on high-impact systems as well as for trust relationships used with external connections. Failure to enforce approvals for incoming trust relationships allows creating unaudited backdoors, and if there is no continuous monitoring for unapproved trust relationships, such trust relationships are essentially permanent. It is recognized that in some organizations, developers and testers need to configure test systems dynamically and may not have support from dedicated system administrators to do so. However, such practice is not acceptable in production environments, particularly not on moderate-impact and high-impact systems. And even on low- impact systems, command restrictions should be used to reduce virus spread threat. While continuous monitoring can be implemented manually using periodic audits, a lot of effort will be saved by using software Ylonen, et al. Expires August 22, 2013 [Page 47] Internet-Draft Automated Access Using SSH Keys February 2013 tools for it. Automated tools can also do a much more thorough job and perform the monitoring much more frequently. 6.2. Creation of New Trust Relationships New trust relationships leading into moderate-impact or high-impact systems MUST NOT be created without proper approval as specified in the policy and MUST be associated with a proper justification for their existence, and SHOULD be associated with an application or business process. New trust relationships leading to low-impact hosts SHOULD have proper approval and justification for their existence. New trust relationships leading into moderate-impact and high-impact hosts MUST have a command restriction, unless a special approval has been obtained for not having a command restriction. New trust relationships leading into low-impact hosts SHOULD have a command restriction to limit virus spread possibilities. New trust relationships breaching a defined internal boundary MUST have special approval and justification. Special approvals SHOULD also be required for trust relationships leading to root accounts or other highly privileged accounts on moderate-impact or high-impact systems. A higher level of approval MAY be required for trust relationships leading to high-impact systems. Note that in some organizations tens of thousands or even hundreds of thousands of new trust relationships are set up annually, and setting them up may represent a substantial fraction of system administrator time and root access needs. Section 7 provides some ideas for reducing associated costs and root access needs. 6.3. Removal of Trust Relationships Trust relationships MUST be removed when they are no longer needed. Sometimes a trust relationship may be removed by express request, e.g., when a business process is changed so that it is no longer needed. Sometimes a trust relationship may be removed because the application or business process it relates to is decommissioned or replaced by another application. Ylonen, et al. Expires August 22, 2013 [Page 48] Internet-Draft Automated Access Using SSH Keys February 2013 Sometimes a trust relationship may be removed because continuous monitoring detects that it is no longer being used. This basically implies that some change made it unnecessary, but it was inadvertantly not removed at that time. (This scenario appears to be very common in practice). When trust relationships are removed, the associated authorized key (if it is key-based) MUST be removed from the authorized keys file of the destination server. When there are no trust relationships remaining using a particular identity key, the identity key SHOULD be removed. 6.4. Periodic Rotation of Trust Relationships XXX 6.5. Facilitating Audits XXX Making information easily available XXX Regular auditing of automated access management as part of normal IT audits. 7. Reducing Cost and Improving Security by Automating Trust Relationship Setups XXX discuss how key management costs can be reduced by automation and triggering key setups automatically when approved in a change control system XXX discuss how automation can reduce the number of system administrators who need root access and the frequency of root access XXX should this go under the planning section? 8. Security Considerations This document is all about security, including how to evaluate the impact of disruption or compromise of information systems, how to reduce the risk to information system from automated access, how to remediate current unmanaged base of SSH user key based trust relationships for automated access, and how to manage and continuously monitor automated access as an ongoing process. Ylonen, et al. Expires August 22, 2013 [Page 49] Internet-Draft Automated Access Using SSH Keys February 2013 Section 1.5 defined information system impact levels based on FIPS 199, but expanding on it by considering information systems having automated access to higher-impact information systems as also having the impact level of the higher-impact information system. Section 2.2.6 briefly discussed unmanaged host keys and how they can be used to compromise authentication and integrity protection using active network-level man-in-the-middle attacks. Section 3 discussed various threats arising from poorly managed automated access and SSH user keys, including virus spread threat, unaudited backdoor threat, leaked keys granting near-permanent access, and lack of proper termination of access when an employee leaves or changes roles. It also discussed how holes made in firewalls may be used for unintended purposes, including command execution, access to internal services, or for hiding source of attacks, if not properly controlled. Section 4 discussed scoping the threats and exposure of an organization to them as a quick precheck during audit, before engaging in a full discovery and remediation project. Section 5 provided recommendations on how to bring existing trust relationships for automated access, particularly SSH keys, under control. Section 6 provided recommendations for continuous monitoring and management of automated access and SSH user keys. As a summary, automated access between systems MUST NOT be overlooked. It has become so prevalent that many orgazations have many times more credentials for automated access to their information systems that they have user accounts for employees. Management of SSH keys is about managing access, with strong ties to identity and access management, security architecture, privileged access management, IT change control, and security audits. Cryptographic properties of the keys are in practice of little importance, as all keys generated with default settings by most commonly used SSH implementations are still cryptographically reasonably strong. Virus spread threat using automated trust relationships may remove defence in depth against attacks and malware. Automated access may provide pathways for bypassing existing privileged access management systems. Rogue administrators may use SSH user keys to create near- permanent unaudited backdoors, and leaked keys may be used for breaking into production servers. Even accidental access using Ylonen, et al. Expires August 22, 2013 [Page 50] Internet-Draft Automated Access Using SSH Keys February 2013 poorly configured trust relationships has in the past caused substantial financial losses. Risks of unmanaged, unaudited automated access are sufficiently high and the state of their management in some of the largest organizations in the world so appalling that all organizations should evaluate to what extent they use automated access within and between their information systems, how it is managed and audited, and whether they are exposed to the risks. IT security auditors, policy makers, and security architects are urged to take automated access and related issues on their agenda. 9. Glossary account: A user account on a computer. An account may belong to an actual person (interactive user) or may be used internally in a system (in which case it is sometimes called a functional account, process account, system account, or non-user account). Active Directory: A directory service created by Microsoft for Windows domain networks, providing a central repository for user information, user groups, and various other kinds of configuration information. Active Directory makes use of the LDAP and Kerberos protocols, among others, and can serve as an LDAP directory and Kerberos Key Distribution Center (KDC). Advanced Persistent Threat (APT): A group, such as a government, with the capability and intent to persistently target an entity using a variety of cyberwarfare techniques, such as espionage, social engineering, custom malware, and sophisticated hacking and evasion techniques. authorized key: A public key that has been configured as authorizing access to an account by anyone capable of using the corresponding private key (identity key) in the SSH protocol. An authorized key may be configured with certain restrictions, most notably a forced command and a source restriction. automated access: Access to a computer without an interactive user, generally machine-to-machine access. Automated access is often triggered from scripts or schedulers, e.g., by executing an SSH client or a file transfer application. Many programs may also use automated access using SSH internally, including many privileged access management systems and systems management tools. automated trust relationship: A trust relationship for automated access. Ylonen, et al. Expires August 22, 2013 [Page 51] Internet-Draft Automated Access Using SSH Keys February 2013 command restriction: See forced command. CIFS: Common Internet File System, a protocol used on Windows for file sharing. The protocol is unencrypted and may be read and subverted by a network-level attacker. The protocol is extremely widely used in Windows environments, less frequently with Unix/ Linux. CISO: Chief Information Security Officer. A person responsible for IT security in an organization. COBIT: Control Objectives for Information and Related Technology, a framework created by ISACA (Information Systems Audit and Control Association) for information technology (IT) management and IT governance. CryptoAuditor: A product from SSH Communications Security for controlling and auditing content of SSH sessions and other encrypted communications, including file transfers. Can also be used for auditing use of SSH/SFTP connections at a firewall and for privileged access auditing for key-based access. destination account: In an SSH connection or trust relationship, the user account for which authentication is provided and under which any commands or other operations performed by the connection are executed (acknowledging that some commands, such as "sudo", may further escalate privileges). destination host: In an SSH connection or trust relationship, the destination host of the connection. A destination host would typically run an SSH server. DSA: Digital Signature Algorithm. An algorithm for public-key cryptography, particularly digital signatures. It is a United States government standard, specified in FIPS 186-3. external key: An authorized key that is used from outside the organization (or outside the environment considered for SSH user key management purposes), or an identity key that is used for authenticating to outside the organization (or outside the environment considered for SSH user key management purposes). Key rotation can break external keys, and therefore it must be ensured that the other side of trust relationships involving external keys is also properly updated as part of rotation. Alternatively, rotation of external keys may be prevented, but that is not a sustainable solution long-term. Ylonen, et al. Expires August 22, 2013 [Page 52] Internet-Draft Automated Access Using SSH Keys February 2013 fingerprint: A hash value of a (public) key encoded into a string (e.g., into hexadecimal). Several fingerprint formats are in use by different SSH implementations. FISMA: Federal Information Security Management Act of 2002, a United States law that mandates how US government agencies must implement their it security. forced command: A restriction configured for an authorized key that prevents executing commands other than the specified command when logging in using the key. In Tectia SSH and OpenSSH, forced command can be configured by using a "command=" restriction in an authorized keys file. functional account: An account used for running applications or other processes, as opposed to an interactive account normally used by a person. Functional accounts are sometimes also called process accounts, system accounts, or non-user accounts (with slight nuances in meaning). host: A computer or other device on a network. A host may be a physical computer, a virtual machine, or any other logical or physical device that can communicate on a network, typically using one or more IP addresses. Some hosts may be multi-homed, meaning that they have more than one IP address. host certificate: A certificate for a host key for host authentication in the SSH protocol (typically an X.509v3 certificate). Host certificates can eliminate the need for distributing host keys to all communicating hosts, greatly simplifying management and rotation of host keys. Widely used with Tectia SSH to avoid copying host keys and to make rotating them easier. host credential: A Kerberos credential that is used for authenticating to a Kerberos KDC as a host principal. host key: A public key used for authenticating a host in the SSH protocol to hosts that want to communicate with it (each host also generally has its own private host key). Some hosts may have more than one host key (e.g., one for each algorithm). Host keys are used for autenticating hosts (machines) themselves, not users or accounts, whereas identity keys and authorized keys relate to authenticating users/accounts and authorizing access to accounts on hosts. Ylonen, et al. Expires August 22, 2013 [Page 53] Internet-Draft Automated Access Using SSH Keys February 2013 identity key: A private key that is used for authentication in the SSH protocol; grants access to the accounts for which the corresponding public key has been configured as an authorized key. indirect trust relationship: A sequence of trust relationships that indirectly leads to another account. For example, account A may be able to log into account B, which may be able to log into account C; then, account C indirectly trusts account A (and B directly trusts A and C directly trusts B). Indirect trust relationships may involve many kinds of trust relationships (e.g., SSH keys, Kerberos and privilege escalation). interactive user: A person (human) that uses a computer (and may type passwords or provide other authentication credentials as needed), as opposed to a computer that performs operations on another computer in an automated fashion. KDC: Key Distribution Center, a component of Kerberos. Kerberos: A centralized authentication and single-sign on system. Also used as part of Active Directory. See RFC 4120 [RFC4120]. key: A cryptographic key. In this document, keys generally refer to public key cryptography key pairs used for authentication of users and/or machines (using digital signatures). Examples include identity key and authorized keys. The SSH protocol also uses host keys that are used for authenticating SSH servers to SSH clients connecting them. Key Distribution Center: A component of Kerberos and Active Directory infrastructure that verifies credentials and issues tickets to principals (e.g., users and hosts). An Active Directory server includes a KDC. Frequently multiple KDCs synchronize information for redundancy. known host: A host whose host key is known (to a particular SSH client). LDAP: Lightweight Directory Access Protocol, a protocol for accessing and maintaining distributed directory information services. See RFC 4511 [RFC4511]. Ylonen, et al. Expires August 22, 2013 [Page 54] Internet-Draft Automated Access Using SSH Keys February 2013 locking down keys: This refers to moving authorized keys to root- owned (or otherwise protected) locations, so that non-root users cannot add new authorized keys to themselves. This helps prevent system administrators and users from creating key-based backdoors that may survive the termination of their account and bypass privileged access management systems. See Section 5.2 for more information. NERC: North American Electric Reliability Corporation, an organization that, among other things, maintains the Critical Infrastructure Protection (CIP) standards that set minimum security requirements for protecting power generation and distribution infrastructure. NFS: Network File System, a file sharing protocol widely used in Unix/Linux environments in enterprises and universities. The protocol is unencrypted and may be subverted by a network-level attacker, permitting modification of any file. (NFS4 adds some security but is rarely used at the time of this writing, or is used with the security features disabled.) OpenSSH: An open source implementation of SSH based on Tatu Ylonen's original free version of SSH from 1995 and further developed by the OpenBSD group. password logger: A software or hardware module for recording keystrokes, especially user names and passwords, typed by an interactive user. Password loggers are nowadays commonly included in various malware and used as part of Advanced Persistent Threat (APT) attacks. Hardware-based key loggers may used in conjunction with physical access to a desktop or laptop (perhaps using a social engineering attack, such as getting hired as a cleaner) to obtain passwords for entry into information systems. PCI DSS: A set of Data Security Standards defined by the Payment Card Industry Security Standards Council, an organization originally formed by major credit card companies. PKI: See Public Key Infrastructure. privilege escalation mechanism: A means for escalating a user's (or processes) privileges from those of one account to those of another account (particularly a root or Administrator account). Examples of privilege escalation mechanisms include intentional provilege escalation tools such as "sudo" and unintentional privilege escalation possibilities based on vulnerabilities and configuration errors (experience has shown that it is very often possible to find vulnerabilities or misconfigurations on that Ylonen, et al. Expires August 22, 2013 [Page 55] Internet-Draft Automated Access Using SSH Keys February 2013 enable privilege escalation once inside a host). An attacker having access to an account can generally change the configuration of the account to cause the user to unknowingly run the attacker's programs that may, e.g., steal the user's password and then use the password to spread the attack. Also, having high-level access on one host on a network may effectively imply access to every user account on every host whose home directory is in networked storage accessible through the same network as the compromised host. Against advanced attackers, even vulnerable embedded devices such as switches, printers and copiers can be used to perform network-level active attacks against other hosts. Some limit will have to be put on what theoretical possiblities are considered, however. Privilege escalation possibilities effectively imply additional trust relationships that may in turn imply a multitude of indirect trust relationships. Public Key Infrastructure: An arrangement that binds public keys with respective user identities using digital signatures issued by a certificate authority (CA). See RFC 3280 [RFC3280]. Putty: An Open Source SSH client for Windows. Reflection for Secure IT: A commercial version of SSH from Attachmate. root account: In Linux/Unix, a privileged account that is usually able to do anything in a computer (including reading any files and modifying any programs). In Windows, Local Administrator and Domain Administrator have similar or even broader power. (This document mostly talks about root access as SSH is mostly used on Linux/Unix and embedded devices, but the same issues often also apply to other technologies and the Windows environment.) rotating a key: Key rotation means changing the key, i.e., replacing it by a new key. The places that use the key or keys derived from it (e.g., authorized keys derived from an identity key, legitimate copies of the identity key, or certificates granted for a key) typically need to be correspondingly updated. With SSH user keys, it means replacing an identity key by a newly generated key and updating authorized keys correspondingly. See also external key. RSA: An algorithm for public-key cryptography based on the difficulty of factoring large integers, invented by Ron Rivest, Adi Shamir and Leonard Adleman. SELinux: Security-Enhanced Linux, a Linux feature that provides mechanisms for supporting access control security policies. SELinux is enabled by default on several Linux distributions (at Ylonen, et al. Expires August 22, 2013 [Page 56] Internet-Draft Automated Access Using SSH Keys February 2013 least in what is called "targeted" mode, where it protects selected services). SFTP: SSH File Transfer Protocol, a file transfer and file sharing protocol typically used with the SSH protocol and originally developed by Tatu Ylonen for ssh-2.0. The protocol is unofficially described in SFTP [SFTP]; there is no normative reference available at the time of this writing. source account: In an SSH connection or trust relationship, a source account is the user account on the host initiating the connection, typically the user account under which an SSH client runs. source host: In an SSH connection or trust relationship, a source host is the host initiating the connection (typically by running an SSH client). source restriction: A restriction configured for an authorized key that limits the IP addresses or host names from which login using the key may take place. In OpenSSH, source restrictions can be configured by using a "from=" restriction in an authorized keys file. SOX: Sarbanes-Oxley Act of 2002, also known as the Public Company Accounting Reform and Investor Protection Act, a United States law that, among other things, sets requirements for protecting certain sensitive information in listed companies. SSH: SSH (Secure Shell) is a protocol and tool for remote system administration, file transfers, and for tunneling TCP/IP communications securely, originally developed by Tatu Ylonen. SSH Communications Security: A company founded by Tatu Ylonen, the inventor of SSH, with products improving security and operational efficiency of large IT environments, particularly for large SSH environments. See http://www.ssh.com [2]. sudo: A privilege escalation mechanism/tool on Unix/Linux that can be used for executing commands as root from a non-root account. The operation of "sudo" depends on its configuration. In some configurations, certain accounts may perform any command as root using "sudo". In some other systems, certain users, such as members of a "wheel" group can perform commands as root by confirming the operation with the user's password. Several commercial tools also exist for the same purpose. Tectia Manager: A product for managing SSH host keys and configurations, from SSH Communications Security. Ylonen, et al. Expires August 22, 2013 [Page 57] Internet-Draft Automated Access Using SSH Keys February 2013 Tectia SSH: A commercial version of SSH servers and clients for Windows, z/OS (IBM mainframes), Unix, and Linux from SSH Communications Security. trust relationship: Something that permits a source account to log in to a destination account (possibly on a different computer). In a sense, the destination account trusts the source account, and if the source account is compromised, so is the destination account. An example is an authorized key (and corresponding identity key) configured for public key authentication in SSH. See also indirect trust relationship and privilege escalation. Universal SSH Key Manager: A product from SSH Communications Security for managing and monitoring SSH keys and other trust relationships for automated access. user key: A key that is used for granting access to a user account in the SSH protocol (as opposed to a host key, which does not grant access to anything but serves to authenticate a host). Both authorized keys and identity keys are user keys. 10. References [FIPS199] Evans, D. L., Bond, P. J., and A. L. Bement, "Standards for Security Categorization of Federal Information and Information Systems", FIPS Publication 199, February 2004. [FIPS200] Gutierrez, C. M. and W. Jeffrey, "Minimum Security Requirements for Federal Information and Information Systems", FIPS Publication 200, March 2006. [NIST800-53] Locke, G. and P. D. Gallagher, "Recommended Security Controls for Federal Information Systems and Organizations", NIST Special Publication 800-53 (revision 3 with updates as of 05-01-2010), August 2009. [KENT] Kent, G. and B. Shrestha, "Unsecured SSH - The Challenge of Managing SSH Keys and Associations", SecureIT White Paper, 2010. [RFC3280] Housley, R., Polk, W., Ford, W., and D. Solo, "Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile", RFC 4251, April 2002. Ylonen, et al. Expires August 22, 2013 [Page 58] Internet-Draft Automated Access Using SSH Keys February 2013 [RFC4120] Neuman, C., Yu, T., Hartman, S., and K. Raeburn, "The Kerberos Network Authentication Service (V5)", RFC 4251, July 2005. [RFC4251] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) Protocol Architecture", RFC 4251, January 2006. [RFC4252] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) Authentication Protocol", RFC 4252, January 2006. [RFC4253] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) Transport Layer Protocol", RFC 4253, January 2006. [RFC4254] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) Connection Protocol", RFC 4254, January 2006. [RFC4511] Sermersheim, J., "Lightweight Directory Access Protocol (LDAP): The Protocol", RFC 4511, June 2006. [SFTP] Galbraith, J. and O. Saarenmaa, "SSH File Transfer Protocol", draft-ietf-secsh-filexfer-13.txt (work in progress), June 2006. Authors' Addresses Tatu Ylonen SSH Communications Security Email: ylo@ssh.com URI: http://www.ssh.com Greg Kent SecureIT Email: gkent@secureit.com Mitchell Klein Bank of America Email: mitchell.p.klein@bankofamerica.com Murugiah Soyppaya NIST Email: soyppaya@nist.gov Ylonen, et al. Expires August 22, 2013 [Page 59] Internet-Draft Automated Access Using SSH Keys February 2013 Ylonen, et al. Expires August 22, 2013 [Page 60]