I had to patch/update a bunch of servers running CentOS.
This are the steps and some notes that I would like to share with the internet.

Lets check if our OpenSSL is bugged or clean!
This is how!

[root@stream4 ~]# rpm -q -a | grep "openssl" | grep CVE-2014-0160 

If we don’t get anything we, probably, aren’t protected…

I’v found a machine with openssl-1.0.0-27.el6_4.2 and openssl-1.0.0-fips.

Both are save although they are kinda old versions…

Lets update our OpenSSL.

[root@stream4 ~]# yum update openssl

Updating:
 openssl x86_64 1.0.1e-16.el6_5.7 updates 1.5 M

When the system ask us if were are sure that we want to download it, we need to type y and enter!

Is this ok [y/N]: y

This is what we get at the end!

openssl-1.0.1e-16.el6_5.7.x86_64.rpm | 1.5 MB 00:00 
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
 Updating : openssl-1.0.1e-16.el6_5.7.x86_64 1/2 
 Cleanup : openssl-1.0.1e-16.el6_5.4.x86_64 2/2 
 Verifying : openssl-1.0.1e-16.el6_5.7.x86_64 1/2 
 Verifying : openssl-1.0.1e-16.el6_5.4.x86_64 2/2

 Updated:
 openssl.x86_64 0:1.0.1e-16.el6_5.7
 
Complete!

HOORAY!

Not lets re-check if our OpenSSL is patched agains heartbeat bug!

 [root@stream4 ~]# rpm -q --changelog openssl | grep CVE-2014-0160
- fix CVE-2014-0160 - information disclosure in TLS heartbeat extension

Nice! We got that feedback! We are now *safe*.
Lets check our OpenSSL version with openssl version -a.

[root@stream4 ~]# openssl version -a
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Apr 8 02:39:29 UTC 2014
platform: linux-x86_64
options: bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/etc/pki/tls"
engines: dynamic

Now we MUST restart all services that use libssl.
How do we know witch service is using it?

[root@stream4 ~]# lsof | grep -i libssl | grep DEL | awk '{print $1}' | sort | uniq
-bash: lsof: command not found

*Outch!* lets install it lsof.

[root@stream4 ~]# yum install lsof

Okay! Lets try it again…

[root@stream4 ~]# lsof | grep -i libssl | grep DEL | awk '{print $1}' | sort | uniq
httpd
master
mysqld
pickup
qmgr
vsftpd

Nice!
Lets see what our openssl change log says to us…

rpm -q --changelog openssl | grep -iE 'security|cve|vuln'

Nice?
Okay lets restart the services OR! the reboot the server!

[root@stream4 ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

[root@stream4 ~]# service mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]

[root@stream4 ~]# service vsftpd restart
Shutting down vsftpd: [ OK ]
Starting vsftpd for vsftpd: [ OK ]

I’v runned the lsof | grep -i libssl | grep DEL | awk ‘{print $1}’ | sort | uniq again and saw that vsftpd was still on the list… i had to stop and started the service… didn’t appear on it!

service vsftpd stop

and then

service vsftpd start

You might get other services!, like cpanel, nginx, pure-ftpd, RAID controllers etc…
The best and cleaner way its to REBOOT the server!

Change all your users passwords on the machine

After this we need to CHANGE ALL THE PASSWORDS on our server for a new ones and regenerate your certificates using a new private key.

How to get all the users on our machine?

awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd

Edit your SSH KNOWN HOSTS on your local computer

Now on your Mac OS X edit your .ssh/known_hosts with something like….

nano /Users/__YOUR_USER__/.ssh/known_hosts

Search for the host/ip entry line and remove it, other ways you wont be able to login since the RSA key has changed!

MacBook-Pro-de-Paulo:.ssh PJunior$ ssh [email protected]
The authenticity of host '193.1XX.XXX.XX (193.1XX.XXX.XX)' can't be established.
RSA key fingerprint is 6e:8e:40:44:e4:38:e4:06:1e:51:ee:ee:ee:26:ee:80.
Are you sure you want to continue connecting (yes/no)?

Tools and usefull links

Test your server with this online tools

https://filippo.io/Heartbleed/
https://lastpass.com/heartbleed/

Usefull links

http://www.digitalwhores.net/linux/protect-your-server-against-the-heartbleed-openssl-vulnerability/
https://www.digitalocean.com/community/articles/how-to-protect-your-server-against-the-heartbleed-openssl-vulnerability
http://www.liquidweb.com/kb/update-and-patch-openssl-for-heartbleed-vulnerability/
http://vexxhost.com/blog/2014/04/09/how-to-mitigate-and-fix-openssl-heartbeat-on-centos-or-ubuntu/