Cracked!
Part 1
Denial and Truth
The phone rings or an email comes in and someone tells you that
they have reason to believe that your box or boxes have been cracked.
Do you believe it? Is it true? What do you do next? What should you
not do? This is the situation any System Administrator can find themselves in. After
all anyone can get these calls and emails. I got a call last month
that someone was being attacked from a mail server. It turned out
that it was not an attack it was an identd
query that was being misunderstood. But some times the call is for real.
Let me tell you about when it was real.
In the fall of 1997 I was actively involved as a part time volunteer
system administrator for a local community network of about twenty
thousand registered users. It had been in operation for several years and had
slowly grown over the years and had been added to as equipment was
donated or purchased. The system had been set up and maintained by a
couple of system administrators from a nearby college and they had been
assigned other tasks as the community network moved from being run by
the school to being run by a non profit organization.
We had quite a collection of different Unix boxes: several IBM RS6000s, a
Sun sparc station, and a Dec Alpha box running OSF/1. They were all tied together
with a maddening interdependence of services. For an example almost
every machine exported some filesystem that everything else needed and
mounted something from the other machines. It was so bad that if you needed
to reboot anything or if something crashed then it would bring
everything to a stop, and you would have to bring the boxes up in a
specific order. For extra credit no two machines ran the same version
of Unix. They all ran the version that they had on them when they were donated.
We were using shadow passwords but had been making all of our
connections in the clear. We were not using anything like Kerberos or
Secure Shell to log in with. In hind sight we were also not really
keeping up with all of the publicly announced security problems. To
be fair to us we were all volunteers working in our spare time and not
working full time.
That was the situation we were in the day my phone rang. It was the
community network's executive director, he had been called by the
owner of an ISP who told him that his ISP had been cracked and that the
cracker had been making some of their connections from our site. They
said that this guy was really good and were convinced that if he was
on a site he had cracked it.
I was very skeptical, after all how could he have cracked us, we
read bugtraq, we upgraded Sendmail every few months we were tight, at
least that was what I thought then. I was to become much more
humble and experienced over the course of the next few months.
I called the ISP back and talked to the owner. He was not a
technician and seemed more of a money man. He told me how they had
been cracked
several times by the same cracker and only recently had decided that
the cracker was locked out. He told me how they would fix the holes
and then he would get right back in using a new hole that they had
not known about. They were running Linux and they told me how this
guy had written his own loadable modules and other rootkit type
programs. I was still skeptical about all of this. I thought that it
was much more likely that the cracker was just using tools he had
gotten from other places, and that he had not written any of
them. That he was just a script kiddie.
I told him that I was confident that we
were secure, but that I would be taking a long hard look at our boxes
and thanked him for the warning and promised that I would get back to
him when I had finished.
A little later on that day when I got some time I started poking
around the system a little bit. I started a background job to look
through all of the filesystems and
generate a report of all set user ID programs. While that ran I
looked at logs looking for connections from the ISP and generally
looking for anything that looked out of the ordinary. At this time
all the logs looked fine to me. We had thousands of logins each day
after all. How could I find anything just glancing through it?
An hour or so later I checked on my file system job and found that it
had finished. Reading through it I found the standard system stuff
(quite a bit there) and then found in a users home directory something
that should in no way be there three set user ID files owned by root.
I was shocked, my heart started racing. I realized that in my
confidence and skepticism I had really just had my head stuck in the
sand and blinders on. That we really had been cracked. From the
dates on the files we had been cracked for five months. Five months,
not yesterday, not last week but for months and I also realized that I
could not count on the time stamps.
It took just a few seconds to verify that the three files where set
user id shells for each one of our architectures and that all of our
machines were cracked. Every last one of them. For at least five
months if the time stamps were to be trusted. These shells when executed would instantly make the person
executing them root.
From there it just gets worse. We had never run any tool like
tripwire that would have allowed us to know for certain what had been
changed on the system. So we had no way of knowing what changes the
cracker had made to our boxes or what applications had been replaced with rootkit
versions. It would be almost impossible without reinstalling the
operating systems to be sure that we had gotten
all of the back doors that he could have installed on the systems. In
other words we were down the creek and had not had a paddle for a long
time.
The first thing I did once I knew that we were cracked was call
everyone that needed to be involved in recovering from this or dealing
with this and let them all know what was
going on. We all agreed not to put anything about the break in on any
email, and to continue just like everything was normal. We were all
concerned that if the cracker knew that we had spotted him he might
"clean up" the system by trashing everything. The one change we did
make over the next few days was to start using pgp for the few things
we did need to transfer through mail.
I then made the very painful phone call to the ISP owner to tell them that
yes we were cracked and thanks for the warning. I had been
over confident and proud and I really did not want to make that call,
but I had promised to call him back and I wanted to know everything he
knew about the cracker so I called anyway. We both agreed to share
information and that we would talk some more once we had collected
some more information. He again told me how difficult it had been
for them to clean their systems. That the second time the cracker got
in that had not been aware of it for a while as he had used a loadable
module to cover his tracks. He also gave me the phone number of a FBI
agent from the computer crimes squad that was working to track the
cracker down.
Next we started examining where the logins of the account that the set user id
shells were in were coming from. We could see that some of the logins seemed to be the
legitimate owner of the account and others were coming from other places. We were
able to identify several hosts that the cracker seemed to be coming
from. It seemed to be several people each
coming from many different places.
It was almost impossible to find anything out about the cracker. We
could not tell who he was, if it was more than one person, were he was
from anything. All we had were logs that we could not really trust and
a system that we did not completely control. All we could tell was that
he had cracked all our boxes, he came from different machines from all
over the world and that he had accounts on other systems. A lot of
other systems.
We were all still convinced in our arrogance that there was no reason
to assume that
the person or persons that had cracked our boxes was highly skilled.
We thought that it was much more likely that the person had just
gotten in through some hole that we had not patched yet. After all
the set user id programs were not hidden. Not a elite thing to do
after all. As time was to go on we were to realize that this
assumption was also just as wrong as thinking we were safe from being cracked.
By the end of the day we had identified some systems that we thought he
was
coming from and called their system administrators to enlist their
help tracking the cracker
back to his home. We had also left a message with an FBI agent that
was working with the ISP to try and track down the cracker.
It had been a busy day, and yet we still knew nothing and thought that
we knew more than we did. I never imagined the months ahead. The day
had changed with a phone call and disbelief had changed to certainty,
pride to humbleness, and yet not as humble as we were going to get.
This was day one.
| 