Challenge Of The Month [Scan 29]

September 2003,

http://project.honeynet.org

Submitted By:

Ramneek Puri & Varun Uppal

Challenge:

On August 10, 2003 a Linux Red Hat 7.2 system was compromised. Your mission is to analyze the compromised system. What makes this challenge unique is you are to analyze a live system. The image in question was ran within VMware. Once compromised, we suspended the image. The challenge to you is to download the suspended image, run it within VMware (you will get a console to the system with root access), and respond to the incident. When responding to the incident, you may do a live analysis of the system or you can first verify that the system has been compromised and then take it down for a dead analysis (or a combination of both). In either case, you will be expected to explain the impact you had on the evidence. Fortunately, this system was prepared for an incident and MD5 hashes were calculated for all files before the system was deployed. Note, this image was recovered from VMware Workstation 4.0, it will not work in older versions

Summary:

§ Special attention is paid while explaining the procedure undertaken & tools used.

§ Correlation of findings is done by using multiple utilities.

§ The main page is kept brief & to the point for better understanding, focusing on answering the questions & explaining the procedure followed.

§ Detailed analysis & explanation of procedure followed is provided via links

§ Though lot of additional information was also discovered & analyzed but the document below is limited to explaining the procedure pertaining to challenge questions.

Procedure for Incident Handling & Forensic Analysis of Linux Image

Download the image

§ Download the image from project.honeypoy.net to local forensic machine.

# wget http://project.honeynet.org/misc/files/linux-suspended.tar.bz2

# wget http://project.honeynet.org/scans/scan29/linux-suspended-md5s.gz

§ Verify the md5 check sum of downloaded vmware image.

# md5sum –c linux-suspended.tar.bz2 d95a8c351e048bd7d5596d6fc49b6d72

.

§ Follow step by step process to secure the evidence.

§ Cross verify that checksum of created image & actual disk images match.

Setup a copy of suspended image for dead analysis in sandbox environment

§ Download & install the vmware 4.0 evaluation copy.

§ Put a copy of Knoppix CD in cdrom drive

§ Configure the vmware image to boot from cdrom with suspended linux image environment

§ Let Knoppix boot with vmware configured with “bridged” network access.

§ The host OS network card will have a Private IP and will not be connected to any other system or network.

§ Install netcat on host OS & make it listen on port 1111 to receive outputs from vmware image.

§ Mounting the suspended image on Knoppix for analysis

Commands used: # mount -t ext3 -o ro,loop,nosuid,noexec,nodev,noatime /dev/sda1 /mnt/sda1

In order to examine the file systems, we mounted the images (on Knoppix Linux system) using a loop device. The images were mounted as read-only with no execute permission, no set UID programs, no device drivers, and no access time modification.

Setup another copy of suspended image for live analysis in sandbox environment

§ Download & install the vmware 4.0 evaluation copy.

§ Vmware configured with private IP network access “bridged”

The host OS will be having a Private IP and will not be connected to any other system or network

§ Install netcat on host OS & make it listen on port 1111 for outputs from vmware image.

Command C:> nc –l –p 1111 > * name of output_file*

§ Compile static binaries & trusted shell, put them in CD ¹

§ Start the suspended image.

§ Mount the CDROM with trusted shell & static binaries on suspended vmware image

§ Set the path

Command # PATH /mnt/cdrom

Building a Security Audit Toolkit http://netadmintools.com/part279.html

Knoppix http://www.knoppix.net/

Vmware www.vmware.com

Challenge Question 1. Describe the process you used to confirm that the live host was compromised while reducing the impact to the running system and minimizing your trust in the system.

Confirm the Incident.

Step 1(live analysis) : verify the md5 checksum of suspended linux image ( suspended-linux.md5)

Command: # md5sum –c host79-2003-08-06 | grep –v “OK”

Explanation: The command will verify the md5sum of all the files in file * host79-2003-08-06 & grep –v will invert the match i.e. all lines that don’t have “OK” will be displayed

Output:

/var/lib/slocate/slocate.db: FAILED

/var/lib/random-seed: FAILED

/var/lib/logrotate.status: FAILED

/var/log/messages: FAILED

/var/log/lastlog: FAILED open or read

/var/log/secure: FAILED

/var/log/maillog: FAILED

/var/log/wtmp: FAILED

/var/log/sa/sa14: FAILED open or read

/var/log/sa/sa15: FAILED open or read

/var/log/sa/sar14: FAILED open or read

/var/log/sa/sa16: FAILED open or read

/var/log/sa/sar15: FAILED open or read

/var/log/sa/sa06: FAILED open or read

/var/log/samba/log.smbd: FAILED open or read

/var/log/samba/smbd.log: FAILED open or read

/var/log/samba/log.nmbd: FAILED open or read

/var/log/samba/localhost.log: FAILED open or read

/var/log/xferlog: FAILED open or read

/var/log/httpd/error_log: FAILED open or read

/var/log/httpd/ssl_engine_log: FAILED open or read

/var/log/httpd/access_log: FAILED open or read

/var/log/httpd/ssl_request_log: FAILED open or read

/var/log/httpd/access_log.1: FAILED open or read

/var/log/httpd/error_log.1: FAILED open or read

/var/log/dmesg: FAILED open or read

/var/log/cron: FAILED

/var/log/boot.log: FAILED

/var/log/rpmpkgs: FAILED open or read

/var/cache/man/whatis: FAILED

/var/cache/samba/smbd.pid: FAILED

/var/cache/samba/connections.tdb: FAILED

/var/cache/samba/nmbd.pid: FAILED

/var/cache/samba/browse.dat: FAILED

/var/run/utmp: FAILED

/var/run/runlevel.dir: FAILED

/var/run/syslogd.pid: FAILED

/var/run/klogd.pid: FAILED

/var/run/apmd.pid: FAILED

/var/run/sshd.pid: FAILED

/var/run/sendmail.pid: FAILED

/var/run/gpm.pid: FAILED

/var/run/crond.pid: FAILED

/var/run/ftp.rips-all: FAILED open or read

/var/spool/anacron/cron.daily: FAILED

/var/spool/anacron/cron.weekly: FAILED

/tmp/root.md5: FAILED open or read

/etc/mtab: FAILED

/etc/rc.d/init.d/functions: FAILED

/etc/rc.d/rc.sysinit: FAILED

/etc/mail/statistics: FAILED

/etc/aliases.db: FAILED

/etc/adjtime: FAILED

/etc/samba/secrets.tdb: FAILED

/etc/httpd/conf/httpd.conf: FAILED

/usr/bin/top: FAILED

/bin/netstat: FAILED

/bin/ls: FAILED

/bin/ps: FAILED

/sbin/ifconfig: FAILED

Indication of Intrusion 1:

§ Lot of very commonly used system binaries have failed the checksum

§ Lot of system logs get “ FAILED open or read”

Step 2( Dead Analysis) : Check for Hidden files & directories

Command: # find /mnt/sda1 -name ".*" -type d -printf "%Tc %h/%f\n" > hidden_dir

Explanation: The command will find all hidden directories ( -type d) & pipe the output to file hidden_dir

Output:

Sun Aug 10 15:32:17 2003 /lib/.x

Wed Aug 6 11:51:48 2003 /root/.ssh

Wed Aug 6 11:13:19 2003 /root/.links

Command # find /mnt/sda1 -name ".*" -type f -printf "%Tc %h/%f\n" > hidden_dir

Explanation: The command will find all hidden files ( -type f) & pipe the output to file hidden_dir

Output:

Mon Jul 14 13:53:10 2003 /var/spool/at/.SEQ

Sun Aug 10 16:01:17 2003 /etc/opt/psybnc/tools/.chk

Mon Jul 9 05:56:20 2001 /etc/skel/.bash_logout

Mon Jul 9 05:56:20 2001 /etc/skel/.bash_profile

Mon Jul 9 05:56:20 2001 /etc/skel/.bashrc

Mon Jul 14 13:53:21 2003 /etc/.pwd.lock

Thu Aug 9 19:53:39 2001 /usr/lib/perl5/5.6.0/i386-linux/.packlist

Wed Apr 5 15:42:06 2000 /usr/share/doc/samba-2.2.1a/examples/VFS/.cvsignore

Mon Jul 9 05:56:19 2001 /usr/share/man/man1/..1.gz

Thu Mar 20 15:53:50 2003 /lib/.x/.boot

Sat Jun 10 14:00:15 2000 /root/.bash_logout

Wed Aug 23 12:02:38 1995 /root/.Xresources

Thu Jul 5 11:23:26 2001 /root/.bash_profile

Wed Aug 23 12:04:30 1995 /root/.bashrc

Sat Jun 10 14:09:02 2000 /root/.cshrc

Tue Jul 11 08:53:11 2000 /root/.tcshrc

Sat Aug 9 14:34:31 2003 /.autofsck

Sun Aug 10 15:54:04 2003 /.bash_history

Indication of Intrusion 2:

A hidden directory by the name of /lib/.x with time stamp of Aug 10 ( day of the incident as described in challenge )

Step 3 ( Dead Analysis) : Check for files with SUID & GUID bit set

Command # find /mnt/sda1 \( -perm -004000 -o -perm -002000 \) -type f –ls

Explanation: The command will find all the files in mounted file-system with SUID & GUID set

Output:

8988 24 -rwsr-sr-x 1 root root 24116 May 21 13:12 /dev/shm/k

59705 767 -rws--x--x 2 root root 785372 Aug 9 2001 /usr/bin/suidperl

59705 767 -rws--x--x 2 root root 785372 Aug 9 2001 /usr/bin/sperl5.6.0

59708 34 -rwsr-xr-x 1 root root 34476 Aug 27 2001 /usr/bin/chage

59710 36 -rwsr-xr-x 1 root root 36208 Aug 27 2001 /usr/bin/gpasswd

60053 37 -rwsr-xr-x 1 root root 37580 Aug 2 2001 /usr/bin/at

60085 13 -rwxr-sr-x 1 root mail 12500 Jun 30 2001 /usr/bin/lockfile

60137 25 -rwxr-sr-x 1 root slocate 25020 Jun 24 2001 /usr/bin/slocate

62367 14 -r-s--x--x 1 root root 13476 Aug 6 2001 /usr/bin/passwd

62406 7 -r-xr-sr-x 1 root tty 6444 Aug 28 2001 /usr/bin/wall

62414 13 -rws--x--x 1 root root 13136 Aug 26 2001 /usr/bin/chfn

62415 13 -rws--x--x 1 root root 12484 Aug 26 2001 /usr/bin/chsh

62433 6 -rws--x--x 1 root root 5456 Aug 26 2001 /usr/bin/newgrp

62444 9 -rwxr-sr-x 1 root tty 8744 Aug 26 2001 /usr/bin/write

62497 21 -rwsr-xr-x 1 root root 21280 Jun 24 2001 /usr/bin/crontab

62531 206 -rwsr-xr-x 1 root root 209948 Sep 6 2001 /usr/bin/ssh

62539 15 -rwsr-xr-x 1 root root 14588 Jul 24 2001 /usr/bin/rcp

62541 11 -rwsr-xr-x 1 root root 10940 Jul 24 2001 /usr/bin/rlogin

62542 8 -rwsr-xr-x 1 root root 7932 Jul 24 2001 /usr/bin/rsh

59390 19 -rwsr-xr-x 1 root root 18444 Aug 27 2001 /usr/sbin/ping6

59394 10 -rwsr-xr-x 1 root root 9804 Aug 27 2001 /usr/sbin/traceroute6

62351 7 -rwxr-sr-x 1 root utmp 6604 Jun 24 2001 /usr/sbin/utempter

62400 441 -r-sr-xr-x 1 root root 451076 Aug 31 2001 /usr/sbin/sendmail

62480 7 -rwsr-xr-x 1 root root 6340 Sep 9 2001 /usr/sbin/usernetctl

62545 20 -rwsr-xr-x 1 root root 20120 Jun 25 2001 /usr/sbin/traceroute

62595 11 -r-s--x--- 1 root apache 11244 Sep 5 2001 /usr/sbin/suexec

44757 23 -rwsr-xr-x