linux dependency hell

One of the old servers I discovered in a forgotten office was running Debian 4. We wanted to do a physical to virtual (P2V) migration so it was no longer running on the old hardware, which was about 8-10 years old. Unfortunately, this old box was not running SSH, and, as seems to happen with “things that have been forgotten”, nothing “just works”.

In order to run VMWare Converter you need to have ssh access. But, sshd was not running on the box, and it appeared the binaries were missing.

I tried to run aptitude install openssh-server and found there was a dependency problem where libc6-dev had been updated to 2.7-18lenny7, but libc6 was still using 2.7-18lenny4. All attempts to update libc6 were met with errors finding programs like locale, or ldconfig, or /etc/init.d/glibc.sh. The /etc/apt/sources.list was so old the mirror no longer existed, so I looked up Debian’s archives and changed it to http://archive.debian.org/debian-archive/debian and did an aptitude clean and aptitude update.

At this point I could actually download packages again, but upgrading still failed. After trying to clear aptitude’s cache and trying again, it still failed. So, I ran aptitude download libc6, and then ran dpkg-deb -x libc6*.deb libc6-unpacked

I then copied the ldconfig and glibc.sh programs from the extracted folder and put them back on the system where they were supposed to be. Then I ran dpkg -i libc6_2.7-18lenny8_amd64.deb, which successfully installed and allowed me to run aptitude upgrade to bring the whole box up to date and run aptitude install openssh-server.

Great, back to VMWare Converter. Enter the IP, name, and password… and error: Unable to query live Linux source. I tested out connecting to the box with an ssh client and was greeted with “Permission denied” as soon as I connected. Looking at the sshd_config revealed there it had no “PasswordAuthentication yes” line, so I added that and did service sshd restart. Now the VMWare Converter could connect and the migration started running.

The next problem was the import failed. Looking at the box start up it could not find the root partition on /dev/hda1. VMWare 5.0 uses LSI Logic SATA drives, so it was clear the old kernel was compiled without the correct drivers. Back to the old box, download the linux src, extract it, make menuconfig, I went with most of the defaults but added Executable Emulation for 32bit binaries on an amd64 core. Did a make, make-modules_install install. The old box was using lilo, but someone had tried to install grub, so I finished the config file and had it point to the old kernel with a boot option for the new kernel. Ran grub-install, rebooted, then ran the converter again.

The new kernel didn’t have the right NIC drivers, so I let it boot into the old kernel. It failed at the same point during the conversion, but this time I just booted it myself and selected my new kernel and both the LSI Logic and VMXNET3 network cards worked, and the services all started up.


VMware 2.0 to 5.0 Migration

The things you find in old closets. Sometimes they might be better left in the closet, hidden from view, but when it is an old server and I’m trying to secure your network, it has to be dragged into light and exorcised.

One of my favorite discoveries has been an old 2008 server (I was worried is was going to be Windows NT!) that was running VMWare Server 2.0. Now, I’ve been doing IT for 20 years, but I had never actually seen VMWare Server 2.0 before. So this was quite an exciting discover. I felt like an archaeologist unearthing an ancient Roman artifact.

After the initial laughter and sending screenshots to everyone I know I decided to migrate the one VM (a Debian 4 distro) that was running on the server to the production environment so it could be backed up and decommissioned properly. But, the big question was, would I be able to successfully migrate it from VMWare 2.0 to VMWare 5.0?

Since you can’t convert an VM that is running, and nobody had the password for the old VM, I just powered it off. Then I loaded up VMWare Converter, told it to convert an “other” image type, and pointed it at the old-vm-servere$ and browsed to the vmdk file. It took an hour to migrate it and convert it to an ESX 5.0 host with hardware level 8. I went ahead and added a VMXNET3 network card instead of the old VMWare 2.0 “Flexible” network card. Then I powered the guest on and rooted the password (edit startup command and add init=/bin/bash, then run mount -rw -o remount /, change the root password, and reboot). Once I logged in with my new root password I modified /etc/network/interfaces to use the new network card and restarted the server again just to make sure everything worked. And it did!

Needless to say, I am very impressed that VMWare has made it so easy to migrate from a 2.0 guest to their latest 5.0 environment. So often big companies will leave no migration paths. This just shows that VMWare is a good company with a great product!



OSX Firewalls – a dismal experience

I’m spoiled on unix firewalls extreme flexibility, and paradoxically, Windows firewall ease of configuration.

There should be a good middle ground in there. Mac does a great job of “being” unix, but with a much easier interface than Windows. Which is a feat. But, let me just put on my rant hat and rant pants. WHAT THE HELL IS WRONG WITH THE OSX FIREWALL!?!?

Why would you move from ipfilters to the more featureful PF firewall that the unix environment offers, and then only provide a brain dead interface that allows you to select Applications to allow through the firewall, and ZERO ability to limit the networks or IPs that are allowed to use those applications?

What kind of security is provided by either allowing a) the entire world to access Screen Sharing, or b) nobody…

Yes, you can make an argument that the corporate firewall, or even your home router, should be acting as hardware firewall to protect you. But when I go to Starbucks, who is protecting me there? When I’m in the airport, who is protecting me? Nobody is. Thanks Apple.

Microsoft gets it right in this department. And, as far as I am concerned, Apple doesn’t even actually offer a useable firewall. At least not out of the box.


Here is my solution: PFLists by Hany El Imam


This handy little app allows you to specify which networks or IP addresses are allowed to connect to which ports on your computer.

The only thing missing is Microsoft’s concept of “network location” so I can be more open at home and more secure at Starbucks.


Bulk Password Testing

Client has a ton of unix hosts, and they all have different passwords, and are not well-documented, and we need to secure them. Not wanting to root all of them or trying to type in a list of different possible passwords and accounts to try, you can use ncrack in an automated way to scan a network and test username and password combinations.

Install ncrack

apt-get install build-essential checkinstall libssl-dev  libssh-dev
wget http://nmap.org/ncrack/dist/ncrack-0.4ALPHA.tar.gz
tar xvfz ncrack-0.4ALPHA.tar.gz
cd ncrack-0.4ALPHA/

sudo checkinstall
sudo dpkg -i ncrack_0.4ALPHA-1_amd64.deb

Create a password list

For my purposes we had a list of passwords we could try. If you don’t have enough information to create a reasonable password list, you can grab a list of 500 passwords from skullsecurity.org.

wget http://downloads.skullsecurity.org/passwords/500-worst-passwords.txt

Run ncrack

Note that you can specify multiple user accounts to try as a comma separate list.

(Oh, and this is just sample output and not from one of our servers.)

ncrack -p 22 –user root -P 500-worst-passwords.txt

## sample output ##

Starting Ncrack 0.4ALPHA ( http://ncrack.org ) at 2011-05-05 16:50 EST
Stats: 0:00:18 elapsed; 0 services completed (1 total)
Rate: 0.09; Found: 0; About 6.80% done; ETC: 16:54 (0:04:07 remaining)
Stats: 0:01:46 elapsed; 0 services completed (1 total)
Rate: 3.77; Found: 0; About 78.40% done; ETC: 16:52 (0:00:29 remaining)

Discovered credentials for ssh on 22/tcp: 22/tcp ssh: ‘root’ ‘toor’

Ncrack done: 1 service scanned in 138.03 seconds.

Ncrack finished.


Ubuntu Desktop Still not Pro Level

Last year I wrote a few posts about trying out Ubuntu Desktop. After many frustrating weeks, I gave up on Ubuntu Desktop. I didn’t post why.

Ubuntu Desktop let’s you log in, and fairly easily download things from the app-store, and browse the internet. It manages to come close to feeling like you are “looking at a Mac”. But that’s it. Once you start using it, nothing is smooth, it doesn’t make a lot of sense. Configuration options you might want to change are just not available in the GUI so you have to drop to the console to run commands or edit files in a text editor. Apps you might want to use, like photo editing, or document writing, just don’t compare with the features in commercial products.

So, yes, you can install an email client, a Word-like program, something that works kind of like spreadsheet software. But I’ll be damned if any of them opened any existing documents or files without conversion errors. And anything I made could not be shared without errors. Calendaring was abysmal. You’d have to be hard pressed to choose GIMP over Photoshop.

Which means that, for me, Ubuntu Desktop might work for someone’s mom to check Yahoo! mail, or to browse Facebook. But it does not work the way a business professional would need it to. It won’t work in an enterprise environment that is Microsoft heavy.

Maybe some startups, or small groups of people could make it work. But, I suspect those folks are all using a Mac. Which *does* work, with just about everything I’ve ever needed it to do.


There are some Unix tools I like to use, which become very hard to run on Windows. But, they generally run on a Mac. And, for those times where you can’t use a Unix tool on Mac or Windows, I use VirtualBox to keep an Ubuntu Desktop install accessible. It actually works extremely well as a virtual instance full-screened on a second monitor and I no longer “hate using it” because it’s there as another tool I can use, not as an obstacle keeping me from doing every minor task I need to do.




Heartbleed Testing

With all the attention Heartbleed is getting right now, I wanted to test out my client’s servers and network devices. One of the easiest ways to check hosts and networks for vulnerabilities is with nmap. There is a new script for scanning for Heartbleed, but it requires LUA scripts, and a recent nmap version. 

Here is how to get everything working on an out-of-the box Unbutu 12.04 Desktop.

If you don’t have Ubuntu 12.04 Desktop, download it and install it using one of these methods:

  • Dual boot your computer
  • Replace your OS
  • Install to flash drive
  • Install on VirtualBox (my preferred solution, be sure to install the VirtualBox Extensions for both the host and guest)

If you don’t have a recent nmap, download requirements and install nmap from svn:

sudo apt-get update

sudo apt-get dist-upgrade

sudo reboot

sudo apt-get install build-essential autoconf checkinstall

sudo apt-get install subversion

svn co https://svn.nmap.org/nmap

cd nmap



sudo checkinstall


If you have a recent nmap, you can try to just download the latest requirements and heartbleed script

cd [install-path]/nmap/nselib/
sudo wget https://svn.nmap.org/nmap/nselib/tls.lua
cd [install-path]/nmap/scripts/
sudo wget https://svn.nmap.org/nmap/scripts/ssl-heartbleed.nse
sudo nmap –script-updatedb


Run nmap with the Heartbleed script:

nmap –datadir [install-path] -sV -p 443 –script ssl-heartbleed [server/network]


Example of a vulnerable system:

443/tcp open https
| ssl-heartbleed:
| The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. It allows for stealing information intended to be protected by SSL/TLS encryption.
| Risk factor: High
| Description:
| OpenSSL versions 1.0.1 and 1.0.2-beta releases (including 1.0.1f and 1.0.2-beta1) of OpenSSL are affected by the Heartbleed bug. The bug allows for reading memory of systems protected by the vulnerable OpenSSL versions and could allow for disclosure of otherwise encrypted confidential information as well as the encryption keys themselves.
| References:
| http://cvedetails.com/cve/2014-0160/
| http://www.openssl.org/news/secadv_20140407.txt
|_ https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-0160