Let me be the first to say that I don’t normally do book reviews.  Frankly, I thought I left them behind years ago when they were required assignments for my undergraduate classes.  Nevertheless, I find myself writing one today, not because I must, but because there are so many cybersecurity professionals out there who are unfamiliar with this particular title, and I’d like to help change that a bit.

The book in question is The Cuckoo’s Egg: Tracking a Spy Through the Maze of Computer Espionage by Clifford Stoll.  It’s one of those books that I first read when I roamed the halls of high school many moons ago, yet somehow, I find myself inexorably drawn back to it every few years.  Even now, the lessons learned within its pages hold true for anyone who’s dared to dip their toe into the world of information security.  Written in 1989, The Cuckoo’s Egg is a first-person account of how Stoll, as a displaced astronomer whose funding has run out, became a manager of the computer system at the Lawrence Berkley National Laboratory (LBL) at University of California, Berkley.  He makes two things very clear from the start of the book:  that he is not an experienced system administrator or developer, and that is he is a proud Berkelyan (a fact that Stoll seems to mention at least twice every chapter, but that’s more a feature of his personality than any form of braggadocio or hubris; you’ll get used to it quickly).  Soon after taking the position, he receives an assignment that is designed to make him become more comfortable and familiarized with the university computer systems – tracking down a 75-cent discrepancy identified by the university’s accounting system used to charge users for whatever processing time they incurred using the LBL computers.  

For those who, up unto this point, did not realize that this story takes place during the earlier, halcyon days of computing, when every little ounce of computing power needed to be accounted for, and billed to a user, let me clarify something – at the time, LBL’s computer network was rented out to remote users – welcome to computer use in the 1980s.  An authorized user would log in, and any computer time they used would be billed to whatever account they had on file with the university.  That’s how things were done in that day and age.  And while most of us might brush off a discrepancy of 75-cents, that value represented a mere nine seconds of computer time.  Things were pricey for us IT folks back in the 80s.  Needless to say, it was an item that Stoll’s superiors had a bit of an interest in identifying and correcting.

This irregularity that Stoll was tasked with researching soon revealed itself to be something far more nefarious than a simple issue with the accounting system.  This “accounting error” that Stoll was originally tasked to track down was evidence inadvertently left by an unauthorized user.  A hacker had managed to infiltrate their systems.  One that had not only been using his university’s computer network, but also had given themselves superuser rights by exploiting a vulnerability in the movemail function of GNU Emacs (the method used by the hacker(s) is what Stoll refers to as the eponymous “Cuckoo’s Egg”).

In the pages that follow the author shares the next 10+ months of his life with the reader as he methodically peels back the layers of the onion Stoll stumbled onto.  Much to his credit, Stoll uses his talents as a physicist to apply the scientific method as he approaches the situation.  Each encounter begins with observation, and detailed documentation of the results.  As his examination of the intrusion continues, and through his often-frustrating attempts to identify the hacker(s), Stoll is led on a journey that begins through the networks of LBL, and quickly spans out to the communications network that was Tymnet, eventually reaching Europe.  In between, Stoll found that the trail of his elusive “guest” led a path to a veritable “who’s who” of defense contractors, and no small number of military installations that the hacker(s) had been able to successfully penetrate.  Without spoiling the rest, let’s just say that Stoll’s first adventure into cybersecurity, as unexpected as it may have been, was one for the ages.  It has a bit of something for everyone:  intrigue, romance, international espionage, comedy, and more than a little bit of cybersecurity. 

The How

I’m hoping at this point you’re wondering, “how did the hacker manage to do it?”  Well, that’s a great question.  It wasn’t some zero-day or advanced technique from the movies that did the trick.  No Hollywood movie magic here.  It was bad server configuration.  LBL’s systems used Berkley Unix (which is not too surprising considering that they’re at UC Berkley).  For the *nx users in the room, you’re probably already familiar with Emacs.  Well, one of the many utilities Emacs provided at the time was movemail, which allowed users to change spool file ownership and move it.  At LBL, it was installed with setiud root.

*nx gurus are already groaning at this last sentence.  For the rest of us, this means that movemail was set to run as root (admin for the Windows folks), and you didn’t need to be a user with elevated privileges to do this.  No sir!  Setuid is used to elevate the privileges of the current user with respect to a specific file or program.  So, a regular user can now run movemail with root/admin privileges.

Was this a bad idea?  Absolutely!  Did people know you could do this?  Kind of.  The information about the vulnerability was out there, but in the 1980s there was no real organized method to share this information.  Even Usenet, for those of us who remember it, was still new’ish at the time.  Sadly, most of this information was still shared by word of mouth.

The hacker took this a little bit further.  They used movemail to copy a script of their own in place of the system’s existing atrun utility (atrun runs jobs queued for later execution).  You shouldn’t be able to do this normally, but if you were doing it via movemail… that has setuid root, so now you can!  As part of this trick, they’d have a copy of the original atrun written to their directory for use when they were finished.

Here’s where the fun comes in.  atrun is periodically run with system privileges.  And the hacker’s version of atrun gave his account root access.  Now the hacker has superuser rights to that particular LBL server.

This is a bit simplified, mind you.  Here’s a more in-depth explanation on the Exploit Database for those who want to try at home.

As a bonus gift the hacker would often use this technique to supplant the login process with one of their own, giving them an opportunity to steal authentication credentials from legitimate users for later use.

When the hacker was done having their fun, they’d simply use the same technique to replace their version of atrun with the original version, and no one was the wiser.

A few things worth nothing: 

• The hacker was usually able to gain initial access to a system by using default accounts that were still using their default passwords, and older user accounts that had easily guessed passwords.
• It was not all that uncommon to configure movemail with setuid root at the time, even though it wasn’t designed for this type of implementation.  As a result, the hacker was able to use this technique repeatedly to great success.
• This vulnerability was known within the *nx community, and particularly by the alphabet agencies (CIA, DIA, NSA, etc.) at the time, but was not widely reported.  At the time, the concept of Common Vulnerability Databases (CVDs) did not exist.
• Thankfully, movemail was eventually updated and this vulnerability was patched.

Never stop testing your security

There are a few key takeaways from this book, but for me, the biggest is the number of times someone responds to Stoll’s findings and warnings with the phrase “that’s not possible.  Our system is secure…”, only to come back soon after and acknowledge that not only was there a serious security vulnerability found within their networks, but that it had been actively exploited, in some cases for years, before it was detected.  In many cases, it was only detected because someone from outside of their organization (i.e., Stoll) alerted them to the situation in the first place.  I know I briefly mentioned this in an earlier post, but it’s worth repeating:  no matter what you do, no matter how hard you try, there is no perfect security.  By no means do I mean to say that you shouldn’t strive for good security (otherwise what’s the point of this blog?), quite the contrary.  But one needs to accept that no matter how good you think your security is, there’s probably room for improvement.  

There’s a quote late Vince Lombardi, “perfection is not attainable, but If we chase perfection we can catch excellence”.  It is, without a doubt, one of my favorites.  And it’s especially true when it comes to cybersecurity.  Just think of all the cybersecurity incidents, breaches, data spills, etc. in the past few years alone that could have been prevented if this concept had been used.  It’s a dangerous thing to rest on one’s laurels when it comes to security, but people have a bad habit of doing it.  Why?  Because they’re people, and people are imperfect.  We also have a bad tendency to become complacent with things, including security  After all, if they don’t see anything going wrong then everything must be okay, right?

Well… not necessarily, but we don’t’ know unless we constantly keep an eye on what’s going on within our environment.  And that means reviewing how things are designed and configured.  It means reviewing your processes and procedures.  It means testing your systems for vulnerabilities.  It means putting everything under stressful conditions to see just how they’ll react.  It’s not always fun… unless you’re me, but then again no one has ever accused security engineers of being sane. 

Document Everything

Another big thing that made Stoll’s book stand out, was the fact that he documented as much as he possibly could.  As I mentioned earlier, he is a physicist, and documenting his research was natural for him.  But this was relatively uncommon for a lot of IT professionals at the time.  Sure, they may have some notes here and there, but Stoll kept a highly organized and highly detailed set of notes throughout this experience.  It was such an uncommon thing to encounter that they were often the first things requested during any discussions he had with any member of the law enforcement or intelligence communities.  You’d think that this would be common sense, and you’d be right.  That said, with all the things thrown at IT and cybersecurity professionals during the normal course of their day, it’s one of those things that often gets lost by the wayside.  My own philosophy is that at any point in time I could be hit by a bus, or one of my coworkers could win the lottery and call in retired (and yes, that has happened to me).  So, I treat my work the same way that I’d treat evidence.  I document it so that anyone with a technical background could replicate it without difficulty.

When it comes to information relating to any cybersecurity investigations I’ve ever been involved with, I document those as if they were… wait for it… evidence.  Because that’s precisely what they are.

Yes, you may get some pushback on the extra time it takes, but when the time comes to use that documentation for reference or to recreate a scenario for testing, you’ll be very glad that you have it.  If for no other reason, then the amount of time you save in not having to reinvent the wheel.

Observation is Key

This one has me smack my head a few times whenever I read The Cuckoo’s Egg – the hacker(s) had been gaining access to systems for years before Stoll stumbled onto them.  The same techniques were often used, and even when the hacker(s) had access to multiple accounts on a system, they’d simply use the same one almost every time.  How were they able to get away with this for so long?  Did they wipe the system logs, or replace them with logs showing normal activity on the system as a form of an audit replay attack?  No.  They really didn’t do anything to hide their actions.  The mindset seems to be that it would be so difficult to track them back to their point of origin that no one would make the effort, so why bother.  So how did they go unnoticed for so long?  Because no one was reviewing the logs.

.

.

.

Feel free to take a moment with that one.

Now one might be willing to give a slight benefit of the doubt to the players involved because this all happened in the 1980s, and the concept of cybersecurity, or any kind of security where it pertained to computers and information systems really wasn’t a concept that had truly taken root yet.  You’d think that it might have, considering the success of the 1983 film Wargames, but not really.  Change takes time.

You might even think that a good sysadmin would look at the logs on a semi-regular basis just to see what’s going on, right?  One thing Stoll makes very apparent without calling it out is that no one was looking at what was going on with their machines, and the bad guys most certainly took full advantage of that.  Like I said earlier, people tend to become complacent when they think their systems are working the way they’re supposed to.

One of the key steps in any penetration test is to observe your target.  The same goes for any attempts to protect against attacks.  Know your system.  Learn what it does, and how it does it.  Baseline that behavior.  And then observe it often to make sure that everything is operating normally.

Get Rid of Default, Aged, and Unused Accounts

I mentioned earlier how the hacker(s) were able to first crack the systems they’d target but let me explain it a bit more.  Put simply, when the hacker(s) would first attempt entry to a system, they’d try to use a known default account that was used with the operating system.  Since most of the systems they were targeting were either using either the Unix or VAX operating systems, this was rather simple (for example:  system, service, user, guest, etc. for VAX, and guest, adm, daemon, nobody, operator, etc. for Unix).  If no one had changed the default passwords, the bad guys had an easy time of things.  Things were usually easy.  Once they were in and used the movemail exploit to give themselves superuser rights, it was just a matter of identifying a user account that hadn’t been used recently.  They’d reset the password, then log in as that user and it was back to business as usual.

Today, finding a list of default accounts and passwords for any product is as simply as a Google query.  There are both legitimate, and not-so-legitimate sites dedicated to them – not to mention all the very helpful people on support forums who are all to happy to provide you with that information if you ever find yourself in need.

Lesson learned:  identify the default accounts for all systems and devices within your environment.  Those that aren’t needed should be deleted.  Those that are necessary for normal operations need to be configured with a password that is in-line with your organization’s policy on password complexities and expiration.  If you don’t happen to have one of those, now’s a good time to make one.

Second lesson learned:  If you don’t have an account expiration policy, you need one.  Orphaned accounts should not live within your environment.  They’re an unnecessary risk, and one that are very easily remedied.  Whenever an employee leaves the company, transfers out, is on extended leave, the account should either be removed or inactive (unless still required by the organization).  System, application, and service accounts should follow the organization’s policies regarding password complexity and expiration.  When the process or system they are designated for is no longer used, then they should be deactivated and retired.

I know that all this is common-sense, and comes off as more than a bit preachy.  I’m sorry about that.  But I’ve lost count of the number of times I’ve run across the issue during a penetration test, or even a simple Vulnerability Identification Scan (VIS).  One day I’ll share the story about the fastest system I even breached during a penetration test.  It took less than a minute, gave me admin rights to the system, and it was all because the client left an Internet-facing server with a default account enabled using the default credentials.

But that’s a story for another time…

As long as this review turned out to be, I left out the majority of the fun that you’ll find in The Cuckoo’s Egg so that anyone interested can grab a copy for themselves.  As I said earlier, I don’t normally do book reviews, but if you liked this one let me know and I’ll make sure to add some more to the list of things to write about in the future