# RootPrompt.org   Nothing but Unix.[Home] [Features] [Programming] [Mac OS X] [Search]


 Feature: Cracked! Part 1: Denial and Truth

In the first of a new series of articles Noel begins the story of when some Unix boxes that he helped admin were cracked.
"The phone rings or an email comes in and someone tells you that they have reason to believe that your box or boxes have been cracked. Do you believe it? Is it true? What do you do next? What should you not do? This is the situation any System Administrator can find themselves in. After all anyone can get these calls and emails. I got a call last month that someone was being attacked from a mail server. It turned out that it was not an attack it was an identd query that was being misunderstood. But some times the call is for real."

 (Submitted by Noel Wed May 3, 2000 )

  

Cracked!


Part 1

Denial and Truth


The phone rings or an email comes in and someone tells you that they have reason to believe that your box or boxes have been cracked. Do you believe it? Is it true? What do you do next? What should you not do? This is the situation any System Administrator can find themselves in. After all anyone can get these calls and emails. I got a call last month that someone was being attacked from a mail server. It turned out that it was not an attack it was an identd query that was being misunderstood. But some times the call is for real.

Let me tell you about when it was real.

In the fall of 1997 I was actively involved as a part time volunteer system administrator for a local community network of about twenty thousand registered users. It had been in operation for several years and had slowly grown over the years and had been added to as equipment was donated or purchased. The system had been set up and maintained by a couple of system administrators from a nearby college and they had been assigned other tasks as the community network moved from being run by the school to being run by a non profit organization.

We had quite a collection of different Unix boxes: several IBM RS6000s, a Sun sparc station, and a Dec Alpha box running OSF/1. They were all tied together with a maddening interdependence of services. For an example almost every machine exported some filesystem that everything else needed and mounted something from the other machines. It was so bad that if you needed to reboot anything or if something crashed then it would bring everything to a stop, and you would have to bring the boxes up in a specific order. For extra credit no two machines ran the same version of Unix. They all ran the version that they had on them when they were donated.

We were using shadow passwords but had been making all of our connections in the clear. We were not using anything like Kerberos or Secure Shell to log in with. In hind sight we were also not really keeping up with all of the publicly announced security problems. To be fair to us we were all volunteers working in our spare time and not working full time.

That was the situation we were in the day my phone rang. It was the community network's executive director, he had been called by the owner of an ISP who told him that his ISP had been cracked and that the cracker had been making some of their connections from our site. They said that this guy was really good and were convinced that if he was on a site he had cracked it.

I was very skeptical, after all how could he have cracked us, we read bugtraq, we upgraded Sendmail every few months we were tight, at least that was what I thought then. I was to become much more humble and experienced over the course of the next few months.

I called the ISP back and talked to the owner. He was not a technician and seemed more of a money man. He told me how they had been cracked several times by the same cracker and only recently had decided that the cracker was locked out. He told me how they would fix the holes and then he would get right back in using a new hole that they had not known about. They were running Linux and they told me how this guy had written his own loadable modules and other rootkit type programs. I was still skeptical about all of this. I thought that it was much more likely that the cracker was just using tools he had gotten from other places, and that he had not written any of them. That he was just a script kiddie.

I told him that I was confident that we were secure, but that I would be taking a long hard look at our boxes and thanked him for the warning and promised that I would get back to him when I had finished.

A little later on that day when I got some time I started poking around the system a little bit. I started a background job to look through all of the filesystems and generate a report of all set user ID programs. While that ran I looked at logs looking for connections from the ISP and generally looking for anything that looked out of the ordinary. At this time all the logs looked fine to me. We had thousands of logins each day after all. How could I find anything just glancing through it?

An hour or so later I checked on my file system job and found that it had finished. Reading through it I found the standard system stuff (quite a bit there) and then found in a users home directory something that should in no way be there three set user ID files owned by root.

I was shocked, my heart started racing. I realized that in my confidence and skepticism I had really just had my head stuck in the sand and blinders on. That we really had been cracked. From the dates on the files we had been cracked for five months. Five months, not yesterday, not last week but for months and I also realized that I could not count on the time stamps.

It took just a few seconds to verify that the three files where set user id shells for each one of our architectures and that all of our machines were cracked. Every last one of them. For at least five months if the time stamps were to be trusted. These shells when executed would instantly make the person executing them root.

From there it just gets worse. We had never run any tool like tripwire that would have allowed us to know for certain what had been changed on the system. So we had no way of knowing what changes the cracker had made to our boxes or what applications had been replaced with rootkit versions. It would be almost impossible without reinstalling the operating systems to be sure that we had gotten all of the back doors that he could have installed on the systems. In other words we were down the creek and had not had a paddle for a long time.

The first thing I did once I knew that we were cracked was call everyone that needed to be involved in recovering from this or dealing with this and let them all know what was going on. We all agreed not to put anything about the break in on any email, and to continue just like everything was normal. We were all concerned that if the cracker knew that we had spotted him he might "clean up" the system by trashing everything. The one change we did make over the next few days was to start using pgp for the few things we did need to transfer through mail.

I then made the very painful phone call to the ISP owner to tell them that yes we were cracked and thanks for the warning. I had been over confident and proud and I really did not want to make that call, but I had promised to call him back and I wanted to know everything he knew about the cracker so I called anyway. We both agreed to share information and that we would talk some more once we had collected some more information. He again told me how difficult it had been for them to clean their systems. That the second time the cracker got in that had not been aware of it for a while as he had used a loadable module to cover his tracks. He also gave me the phone number of a FBI agent from the computer crimes squad that was working to track the cracker down.

Next we started examining where the logins of the account that the set user id shells were in were coming from. We could see that some of the logins seemed to be the legitimate owner of the account and others were coming from other places. We were able to identify several hosts that the cracker seemed to be coming from. It seemed to be several people each coming from many different places.

It was almost impossible to find anything out about the cracker. We could not tell who he was, if it was more than one person, were he was from anything. All we had were logs that we could not really trust and a system that we did not completely control. All we could tell was that he had cracked all our boxes, he came from different machines from all over the world and that he had accounts on other systems. A lot of other systems.

We were all still convinced in our arrogance that there was no reason to assume that the person or persons that had cracked our boxes was highly skilled. We thought that it was much more likely that the person had just gotten in through some hole that we had not patched yet. After all the set user id programs were not hidden. Not a elite thing to do after all. As time was to go on we were to realize that this assumption was also just as wrong as thinking we were safe from being cracked.

By the end of the day we had identified some systems that we thought he was coming from and called their system administrators to enlist their help tracking the cracker back to his home. We had also left a message with an FBI agent that was working with the ISP to try and track down the cracker.

It had been a busy day, and yet we still knew nothing and thought that we knew more than we did. I never imagined the months ahead. The day had changed with a phone call and disbelief had changed to certainty, pride to humbleness, and yet not as humble as we were going to get.

This was day one.


Our content can be syndicated: Main page Mac Page

Copyright 1999-2005 Noel Davis. Noel also runs web sites about sailing and kayaking.
All trademarks are the property of their owners.
All articles are owned by their author