OpenSSL CVE-2014-0160 Heartbleed bug and Red Hat Enterprise Linux

OpenSSL CVE-2014-0160 Heartbleed bug and Red Hat Enterprise Linux

Issue

• Does CVE-2014-0160 affect Red Hat Enterprise Linux?
• Need fix for openssl heartbleed bug
• What versions of Red Hat Enterprise Linux are affected by openssl heartbleed vulnerability?
• Do we have a list of packages/services we ship with RHEL that need a restart after OpenSSL has been updated?

Environment

• Red Hat Enterprise Linux 7 Release Candidate (RC) not affected
• Red Hat Enterprise Linux 7 Beta affected
• Red Hat Enterprise Linux 6 affected
• Red Hat Enterprise Linux 5 not affected
• Red Hat Enterprise Linux 4 not affected
• For other affected products, refer to https://access.redhat.com/site/announcements/781953

Resolution

Step 1: Determine if RHEL system is vulnerable to flaw described in CVE-2014-0160

• Red Hat Enterprise Linux 7 Release Candidate (RC)
  
  • Red Hat Enterprise Linux 7 RC include OpenSSL version openssl-1.0.1e-34.el7 which includes a fix backported from openssl-1.0.1g
• Red Hat Enterprise Linux 7 Beta
  
  • OpenSSL versions openssl-1.0.1e-33.el7 and earlier include a flawed libssl.so library vulnerable to the issue
  • To determine openssl version, use the command: rpm -q openssl
  • Version openssl-1.0.1e-34.el7 included a fix backported from openssl-1.0.1g
  • See footnote for considerations specific to RHEL 7 Beta¹
• Red Hat Enterprise Linux 6
  
  • OpenSSL versions openssl-1.0.1e-15 through openssl-1.0.1e-16.el6_5.4 include a flawed libssl.so library vulnerable to the issue
  • The first affected version shipped with RHEL 6.5 (RHEL 6.4 and older shipped with the unaffected openssl-1.0.0 series)
    • Systems which report as RHEL 6.0 - 6.3 could still have been updated to a newer [vulnerable] openssl-1.0.1 series package
  • To determine openssl version, use the command: rpm -q openssl
  • Version openssl-1.0.1e-16.el6_5.7 included a fix backported from openssl-1.0.1g
• Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 4
  
  • Vulnerable OpenSSL 1.0.1 series versions never shipped in RHEL 5 or earlier

---

Optional Step 2: Look for and/or query processes which are using the vulnerable libssl library

• Run either of the following commands as root to display running processes which are using libssl
  
  • lsof | awk 'NR==1 || $0~/libssl.so.1.0.1e/'
    
  • grep libssl.so.1.0.1e /proc/*/maps | cut -d/ -f3 | sort -u | xargs -r -- ps uf
    
• Use Red Hat's Heartbleed Detector tool to query public-facing servers (or the version for internal systems)
  
• Problem: Both of the above commands find no processes, but one of the Heartbleed detector tools reports system vulnerable!
  

---

Step 3: Upgrade the openssl package

• Red Hat Enterprise Linux 7 Beta
  
  • Update to openssl-1.0.1e-34.el7 (which corrected the flaw) or later
• Red Hat Enterprise Linux 6
  
  • Update to openssl-1.0.1e-16.el6_5.7 (which corrected the flaw, as described in RHSA-2014:0376) or later
• As always, registered systems with internet access (or any RHEL 7 Beta system, or systems connected to Satellites, etc) can be updated via yum, e.g.:
  
  • yum update openssl
• Otherwise, use a connected system to download the package or download the package directly from the Customer Portal²
  
  • After that, transfer the package to the system in question and install it manually with yum, e.g.:
    
    • yum update <path-to-openssl*rpm>

---

Step 4: After updating openssl, restart all processes using the flawed libssl.so³

• The safest & simplest thing to do: perform a system reboot
  
• Alternatively: use the commands from Optional Step 2 to determine which processes need to be restarted and then act accordingly
  

---

Optional Step 5: Re-scan updated systems with one of the Heartbleed Detector tools

• Use the scanners mentioned above in Optional Step 2
• Problem: I updated openssl but the Heartbleed detector tools still report vulnerable!

---

Optional Step 6: Take additional remediation steps as desired

• Official statement from Red Hat Security Response Team:
  

  Red Hat is not aware of any public exploit being used in the wild for this issue prior to the date of disclosure. However, a number of public exploits were published shortly after the issue was disclosed.⁴ These exploits could lead to the disclosure of information handled by applications using OpenSSL, including private keys, session tokens, and data submitted by users, which could include authentication credentials. It is recommended that you assess the risk this could pose to your systems, and perform additional remediation as you deem appropriate.
  

---

Root Cause

• Official statement from Security Advisory RHSA-2014:0376:
  

  An information disclosure flaw was found in the way OpenSSL handled TLS and
  DTLS Heartbeat Extension packets. A malicious TLS or DTLS client or server
  could send a specially crafted TLS or DTLS Heartbeat packet to disclose a
  limited portion of memory per request from a connected client or server.
  Note that the disclosed portions of memory could potentially include
  sensitive information such as private keys. (CVE-2014-0160)
  

• For links to more detail, see the entry for CVE-2014-0160 in Red Hat's CVE Database

Linux Boot Process

Press the power button on your system, and after few moments you see the Linux login prompt.

Have you ever wondered what happens behind the scenes from the time you press the power button until the Linux login prompt appears?

The following are the 6 high level stages of a typical Linux boot process.

1. BIOS

• BIOS stands for Basic Input/Output System
• Performs some system integrity checks
• Searches, loads, and executes the boot loader program.
• It looks for boot loader in floppy, cd-rom, or hard drive. You can press a key (typically F12 of F2, but it depends on your system) during the BIOS startup to change the boot sequence.
• Once the boot loader program is detected and loaded into the memory, BIOS gives the control to it.
• So, in simple terms BIOS loads and executes the MBR boot loader.

2. MBR

• MBR stands for Master Boot Record.
• It is located in the 1st sector of the bootable disk. Typically /dev/hda, or /dev/sda
• MBR is less than 512 bytes in size. This has three components 1) primary boot loader info in 1st 446 bytes 2) partition table info in next 64 bytes 3) mbr validation check in last 2 bytes.
• It contains information about GRUB (or LILO in old systems).
• So, in simple terms MBR loads and executes the GRUB boot loader.

3. GRUB

• GRUB stands for Grand Unified Bootloader.
• If you have multiple kernel images installed on your system, you can choose which one to be executed.
• GRUB displays a splash screen, waits for few seconds, if you don’t enter anything, it loads the default kernel image as specified in the grub configuration file.
• GRUB has the knowledge of the filesystem (the older Linux loader LILO didn’t understand filesystem).
• Grub configuration file is /boot/grub/grub.conf (/etc/grub.conf is a link to this). The following is sample grub.conf of CentOS.

#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.el5PAE)
          root (hd0,0)
          kernel /boot/vmlinuz-2.6.18-194.el5PAE ro root=LABEL=/
          initrd /boot/initrd-2.6.18-194.el5PAE.img

• As you notice from the above info, it contains kernel and initrd image.
• So, in simple terms GRUB just loads and executes Kernel and initrd images.

4. Kernel

• Mounts the root file system as specified in the “root=” in grub.conf
• Kernel executes the /sbin/init program
• Since init was the 1st program to be executed by Linux Kernel, it has the process id (PID) of 1. Do a ‘ps -ef | grep init’ and check the pid.
• initrd stands for Initial RAM Disk.
• initrd is used by kernel as temporary root file system until kernel is booted and the real root file system is mounted. It also contains necessary drivers compiled inside, which helps it to access the hard drive partitions, and other hardware.

5. Init

• Looks at the /etc/inittab file to decide the Linux run level.
• Following are the available run levels
  • 0 – halt
  • 1 – Single user mode
  • 2 – Multiuser, without NFS
  • 3 – Full multiuser mode
  • 4 – unused
  • 5 – X11
  • 6 – reboot
• Init identifies the default initlevel from /etc/inittab and uses that to load all appropriate program.
• Execute ‘grep initdefault /etc/inittab’ on your system to identify the default run level
• If you want to get into trouble, you can set the default run level to 0 or 6. Since you know what 0 and 6 means, probably you might not do that.
• Typically you would set the default run level to either 3 or 5.

6. Runlevel programs

• When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level.
• Depending on your default init level setting, the system will execute the programs from one of the following directories.
  
  • Run level 0 – /etc/rc.d/rc0.d/
  • Run level 1 – /etc/rc.d/rc1.d/
  • Run level 2 – /etc/rc.d/rc2.d/
  • Run level 3 – /etc/rc.d/rc3.d/
  • Run level 4 – /etc/rc.d/rc4.d/
  • Run level 5 – /etc/rc.d/rc5.d/
  • Run level 6 – /etc/rc.d/rc6.d/
• Please note that there are also symbolic links available for these directory under /etc directly. So, /etc/rc0.d is linked to /etc/rc.d/rc0.d.
• Under the /etc/rc.d/rc*.d/ directories, you would see programs that start with S and K.
• Programs starts with S are used during startup. S for startup.
• Programs starts with K are used during shutdown. K for kill.
• There are numbers right next to S and K in the program names. Those are the sequence number in which the programs should be started or killed.
• For example, S12syslog is to start the syslog deamon, which has the sequence number of 12. S80sendmail is to start the sendmail daemon, which has the sequence number of 80. So, syslog program will be started before sendmail.

There you have it. That is what happens during the Linux boot process.

Unlock Linux accounts which are locked by PAM_tally2

pam_tally2 module is used to lock user accounts after certain number of failed ssh login attempts made to the system. This module keeps the count of attempted accesses and too many failed attempts.

pam_tally2 module comes in two parts, one is pam_tally2.so and another is pam_tally2. It is based on PAM module and can be used to examine and manipulate the counter file. It can display user login attempts counts, set counts on individual basis, unlock all user counts.


By default, pam_tally2 module is already installed on the most of the Linux distributions and it is controlled by PAM package itself. This article demonstrates on how to lock and unlock SSH accounts after reaching a certain failed number of login attempts.

How to Lock and Unlock User Accounts

Use ‘/etc/pam.d/password-auth‘ configuration file to configure login attempts accesses.
Open this file and add the following AUTH configuration line to it at beginning of the ‘auth‘ section.

auth        required      pam_tally2.so  file=/var/log/tallylog deny=3 even_deny_root unlock_time=1200

Next, add the following line to ‘account‘ section.

account     required      pam_tally2.so

Parameters

1. file=/var/log/tallylog – Default log file is used to keep login counts.
2. deny=3 – Deny access after 3 attempts and lock down user.
3. even_deny_root – Policy is also apply to root user.
4. unlock_time=1200 – Account will be locked till 20 Min. (remove this parameters if you want to lock down permanently till manually unlock.)

Once you’ve done with above configuration, now try to attempt 3 failed login attempts to server using any ‘username‘. After you made more than 3 attempts you will get the following message.

[root@tecmint ~]# ssh tecmint@172.16.25.126
tecmint@172.16.25.126's password:
Permission denied, please try again.
tecmint@172.16.25.126's password:
Permission denied, please try again.
tecmint@172.16.25.126's password:
Account locked due to 4 failed logins
Account locked due to 5 failed logins
Last login: Mon Apr 22 21:21:06 2013 from 172.16.16.52

Now, verify or check the counter that user attempts with the following command.

[root@tecmint ~]# pam_tally2 --user=tecmint
Login           Failures  Latest    failure     From
tecmint              5    04/22/13  21:22:37    172.16.16.52

How to reset or unlock the user account to enable access again.

[root@tecmint pam.d]# pam_tally2 --user=tecmint --reset
Login           Failures  Latest    failure     From
tecmint             5     04/22/13  17:10:42    172.16.16.52

Verify login attempt is reset or unlocked

[root@tecmint pam.d]# pam_tally2 --user=tecmint
Login           Failures   Latest   failure     From
tecmint            0

The PAM module is part of all Linux distribution and configuration provided about should work on all Linux distribution. Do ‘man pam_tally2‘ from the command line to know more about it.