Category: IT

The Palace of Harmonious Virtualization

Introduction

In my job, I get to think a lot about where things are going. I’m hearing day in and day out that security is a major stumbling block to fully virtualizing a datacenter and also for “cloud”.

In the case of the virtualized datacenter, what many call Private Cloud, this stumble usually happens when the security guy is brought in after the ball is already in motion and promptly puts a stop to things “until it’s secure”

A common reaction is to then stop, investigate and build a wall/secure the edge.

The issue I have with this is that a wall/edge solution is no longer good enough. An edge only solution is not going to help you deal with an insider and todays advanced persistent threats[Chuck Hollis’ blog]. It’s only a single layer of Defense in Depth [us-cert.gov]. Today’s threats go far beyond targeting the edge. In the virtual world, they take on a whole new level of importance.

In this article, I’ll endeavor to show you, based on the use of ancient and modern history, that building a wall around your virtual datacenter is only the first line of defense. Knowing what’s going on INSIDE the wall is critical to the protection of your data, your intellectual property and ultimately, your business.

Different cultures, common solutions

I’ve been traveling a lot lately. Just in the month of November I traveled to both China and Israel. In China I presented at the VMware vForum and RSA Conference events in Beijing. These three presentations were mostly on visibility into the virtual environment. While in Israel as the latest member of RSA’s Cross-Product Architecture Team we had meetings with our development teams there during which I got to have a number of great conversations around virtualization and security design.

Both countries were eye-opening experiences on a number of levels. China blew me away with its breath-taking rapid growth and the corresponding growth of my waistline from the excellent food (minus the scorpions on a stick)! Visiting China also helped set off a light bulb moment about defense in depth while walking through the Forbidden City. I mentioned this to my colleague Bob Griffin, one of the industry’s most accomplished security professionals. Out of that came his elegant blog posting [rsa.com] titled The Forbidden City and Defense in Depth. I’ll go into more detail about that in a moment.

Israel was amazing for its geo-political place in the world and shawarma! As I explained to a Jewish friend here in the US, to me, Jerusalem was a mix of bizarre, solemn and thought provoking all wrapped up in lafa bread with a dab of hummus. I loved every minute there and am very thankful to have experienced this wonderful place with my new Israeli friends.

(Have you figured out that I’m a foodie yet? I really need to get back to the gym after these trips….)

Both visits brought insight into history and how learning from history allows us to move forward into the future. You see, both countries have something to teach us about walls, perimeters and defense in depth and I’ll try to relate that to virtualization security and why you, as an IT professional need to involve your security team and why, as a Security Professional you need to jump in and get your hands dirty and not just build a wall.

 

China – The Great Wall and The Forbidden City

clip_image002[6]

China’s Great Wall was originally designed to keep out the invasions of nomadic peoples like the Mongols and according to Wikipedia; it’s over 5500 miles long. Like any good fortification or wall, it’s designed to keep out people or things you don’t want on the inside. It has controlled openings where only what/who you want to come through are allowed. Not unlike an IT Security firewall, eh?

clip_image004[6]

The Forbidden City was also designed to control access. Not access to a country, but to the Emperor and his entourage and ministers. As you can see by this photo, there are a series of walls and courtyards, all designed to limit access to only those allowed to specific places. Ultimately, ending up within an area just for the Emperor and his closest contacts. It was a fascinating walk through history and was what sparked this whole thinking about defense in depth and specifically how it relates to virtualization. I very much encourage you to read Bob Griffin’s blog post, referenced earlier. He goes into good depth on this topic.

Israel – Walls that separate

clip_image006[6]

Israel also has walls. These are walls surrounding the Old City of Jerusalem with controlled access points such as the Jaffa Gate, a narrow road that takes a 90-degree turn. It’s pretty amazing to see cars navigating this ancient road! The marks on the walls from bumpers scraping says it all!

In addition to ancient walls, there are the new walls separating regions, limiting and controlling access.

Defense in Depth and the Insider Threat

China

Each are examples of Defense In Depth. I was in China for two weeks and went to the Forbidden City over the weekend before Bob arrived. It struck me as I walked through the Forbidden City a few days before Bob and I did that I was clearly walking through a series of control points through thick walls. All designed to protect the Emperor and his family and not unlike the series of walls one must go through to get at data in the enterprise. For example, a VPN, an internal website and a download on that website. Each may have various levels of authentication needed to access information.

But what the walls didn’t account for was the insider threat. Security was deployed to control the checkpoints and was outward-facing to gauge risk to threats from the outside. But what if someone or something within the walls became a threat? Would security be able to react? Or after the fact would they be able to reconstruct what went wrong and adjust going forward?

Israel

In ancient Israel there were also walls and control points into the Old City. In modern day Israel, as I point out above, there are a number of newer walls. The difference is that there is also strong internal security. Intelligence is regularly gathered and acted upon. For example, as was related to me, internal intelligence sources acting on information, regularly change operations such as troop movement and transportation resources in order to mitigate potential threats. So, not only does the country look outward to its security but also inward. And when something does unfortunately happen, there is usually a pretty good amount of forensic information reconstructed to learn from so that it doesn’t happen again.

“Those who cannot remember the past are condemned to repeat it” – G. Santayana

So you are probably asking yourself now how this relates to virtualization and security? Well, every IT and security professional needs to begin to look inward to virtualization and private cloud. It’s not good enough to just build a wall around your virtualized datacenter.

One of the presentations I gave at the RSA Conference in Beijing was about visibility in the Private Cloud. My theory (a generalization, I’ll admit!), based on lots of discussions with customers, partners, sales people and community members, is that IT is pushing the use of virtualization and private cloud and the last people IT involves is Security. As was said by a gentleman at VMworld 2011 in Las Vegas during a security session: “I’m the security guy, I put the NO! in InNOvation”! (For the record, EVERY IT guy in the room agreed with him!)

Typically, when Security is presented with something to secure that they know little about, the first reaction is to build a perimeter or wall around it. But, as I laid out in the examples above and below, that’s just not good enough anymore. Visibility, or “intelligence” into the operations of virtualization is fundamentally important. You must now start to look inward to mitigate threats. You must be looking for the things that stand out and react accordingly. And finally, you need a way to reconstruct what happened when something DOES go bad (and it will!) and work to mitigate that failure in the future.

Examples of the insider threat

Many who have worked in IT have heard stories of a sysadmin gone bad. It’s someone in IT who gets ticked off and has the privileged role to do something destructive. In this case, because everything is virtualized, he has decided to copy the Domain Controller, Web App and Database virtual machines. No longer am I worried about just credit card data going out the door, I as a CIO/CISO have to be concerned with my intellectual property and the fully running application leaving my control!

Another example is one of not understanding the implications of virtualization and security. I encounter many people who haven’t quite connected the dots around protecting and monitoring privileged access to their virtualization environment. The number of ESX and vCenter servers that are unprotected is staggering! Just ask virtualization security expert Edward Haletky [virtualizationpractice.com]! He pointed me to some penetration testing that was specifically looking for vSphere components open on the Internet and the results were scary. Compromising these critical components is the analog equivalent of breaking the lock on your physical datacenter! You DO have a lock on your datacenter door and the racks inside, right? And you DO limit access to those resources?

So as we think about history, old and modern, and how they apply to different scenarios, we must reflect and ask some questions.

· Is it good enough for you to just build a wall?

· Is it good enough to be just looking outward, waiting for that nomadic hacker to cross over the hill?

· Who’s guarding inside the palace and can you trust them?

Plan, design, implement

The importance of good design should not be discounted. As an IT manager, when you are going through the process of the redesign your datacenter you bring in your network and storage teams and plan and design a solution that works for you. It should also be the time you bring in your security guy, get him up to speed and make him/her part of the team.

By working together to build strong controls of identity and access and leverage new capabilities of the virtual infrastructure you can provide the business potentially greater visibility and combine them with tools to manage the risk. Leveraging this kind of intelligence can protect your Palace of Harmonious Virtualization!

Bob said it best when he wrote, “Defense in depth is a valid idea, but the analogy needs to be with adaptive and responsive systems like those of human physiology.” After all, if history is any indication, the biggest threats will be when someone gets inside your infrastructure.

mike

—————————————————

I don’t call myself “a security guy”. It’s not my background. I’m an IT/Infrastructure guy who’s been involved with security at various times of his career. I’ve taken on as a mission during my time at RSA to bring IT and Security together because only together can we all fight these new and emerging threats.

A dinner with infamy


I’m writing this from 39119 feet above the corn fields of Wisconsin & Illinois, having just woken from a long nap brought on by a week in Las Vegas attending VMworld. Like every other attendee and exhibitor, I’m exhausted yet have a smile on my face. This was my third VMworld and like the other two I attended, it was a series of long days, lots of walking, meeting friends old and new and most of all, having fun and learning stuff.

I’ve got a couple of thoughts brewing on more technical blog posts that I’ll get to later. However, this post needs to stand alone and it’s a doozy!

The Wait

This post is about waiting 24+ years for an answer to a question that has weighed on myself and two former co-workers. It’s also about forgiveness for a brash youth who’s paid his debt to society and one of the most interesting, fun and enlightening nights of my life.

The Set Up

I was in the Pallazo, enjoying a drink with Brad Maltz (@bmaltz) and Keith Norbie (@keithnorbie) and Keith’s wife Kim. The iPhones were out and we were all tweeting while Keith’s wife watched us, being quite polite and not rolling her eyes.

I was trying to organize folks to meet up for Sushi and then head over to the Ellis Island Casino for vBeers and Karaoke when I saw a tweet from Christofer Hoff (@beaker) about having dinner with a celebrity. Here’s my response

image

Mind you, it’ wasn’t so much the celebrity bit that piqued my interest, it was steak and vino with a guy for whom food is something to be worshiped. I knew Hoff wasn’t going to the buffet.

He kindly DM’d me “You may attend if you’d like. 8pm” followed by an ask to dress decently. Well, of course, that was making it even more interesting so I said my goodbyes and went back to the hotel and pulled out my last clean, pressed dress shirt and pants out of the closet and proceeded to head off to Caesars Palace.

I arrived to find Hoff and Karl Renneker of VKernel at the bar but no mystery guest. We had a drink and I ask Hoff who’s the celebrity. Now, if you know him, you know that devilish smile he gets when he’s up to something (which is just about all the time, but I digress). He said “It’s a surprise…” and smirked.

Oooookay..  I think the anticipation of my reaction was killing him so a few minutes later, after discussing 3D televisions, back came the smile and he leaned over and said “Want to know the mystery guest?” “YES!” (the anticipation was killing me as well)

“It’s Mitnick”he responded.

<THUD>

I’m sure the look on my face was priceless. Hoff sure seemed to enjoy the reaction.

“Wow!” I said. “Really? Wow! I’ve got a question to ask him that’s been bugging me for 20 something years!”

The Back Story

You see, back in the mid/late 80’s, I was working at Digital Equipment Corp. Only nobody called it that. It was known internally and externally as DEC. You can read all about my time there here. I was working at DEC’s Littleton King Street facility, also known internally as LKG and home to DEC’s Networks and Communications group. I was the systems manager on the largest VAXcluster there, the DELNI:: cluster, supporting over 600 users.

DEC had been experiencing lots of hacking around that time and the buzz internally was that the hacker was getting into everything. We were finding crumbs on our systems so my co-workers and I decided to lay out a trap.

My two buddies in this endeavor are still my good friends to this day. Dean McGorrill was a master coder/hacker who could whip up almost anything in VAX Macro in a frighteningly short time.

Dave Cantor, who I still have dinner with every month, is also known as DaveC, DC, DuraCell and DCL Dave for his unbelievable mastery of the VMS DCL command language. Dave was also know for saying “Sigh!” a lot and because of that Dave, Dean and I became “Sigh Old, Sigh Mid and Sigh Young”.

Me, I was the young buck with a mastery for trying the latest things available, living on the razors edge of tech by putting beta code into production without anyone knowing. I pushed all the boundaries of VAXcluster technology and went on to work in the VMS Development Group where I built one of the most complex VAXclusters on the planet.

So, over one of our many post-work nights at Barnaby’s in Ayer, munching on buffalo wings and beer, we crafted a plan to catch the perp in the act and watch what he was doing. Deano wrote up some Macro to attach to the process of a username we thought was created by our hacker. We could watch everything he typed. (Remember, this was still the days of green screens and modems!) DaveC wrote some DCL to launch the watch.exe code when the hacker logged in and honestly, I can’t remember what part I contributed lo these many years later. hahaha!

So, we now thought we were smart and when the alarm went off, we scrambled into Deano’s cube and hunkered down to watch. We made one mistake though. We left the name of the Macro code that Dean had compiled as “WATCH.EXE”. We immediately realized our mistake but it was too late, we were committed at this point.

We watched as the user did a “SHOW PROCESS”. This is the DCL equivalent to “ps” on Unix. He listed all the processes and noticed Dean’s account logged in running WATCH.EXE. He did a “SHOW PROCESS/FUL MCGORRILL”. We about shat ourselves. We were made. The next thing that happened was epic.

DCL Overview

Before I go there, a quick review of the DCL Command Language. In DCL the system prompt is a “$” sign. A typical command would look like this.

$ SHOW PROCESS

or

$ MAIL

In DCL you can write scripts that can allow you to string together lots of commands. Unfortunately what DCL lacked at the time was a pipe. If you look thru DCL and you’re familiar with PowerShell, you’ll catch some similarities and that’s intentional as my friend Jeffery Snover, the architect of PowerShell and now lead architect of Windows Server at Microsoft, was an ex-DECcie and loved the Verb/Noun syntax.

So, as you write DCL scripts and want to add a comment, you use the “!”. Here’s an example:

!This is a comment

$ SHOW PROCESS

$Exit

Ok, back to our story. I know this is written in Inception style but I’ll wrap it all up to the dinner shortly.

Our hacker has noticed that Dean is running WATCH.EXE and proceeds to type on his terminal

$ !Hi there.

<Crap, we were so totally pwn’d, years before pwn’d became a term!>

$ !What’s the matter, cat got your tongue?

He then went on his merry way looking around the OS. Dean and Dave looked at me and I bolted for the datacenter. I ran at top speed from LKG2 to LKG1, opened the door to the raised floor lab that held some systems, the building networking gear and modem banks and slammed the Big Red Button that took down the lab. (I always wanted to do that!)

Back to dinner

Back at the bar, Hoff is still smirking and I quickly tell my tale, not nearly as detailed as above. We settle into dinner and shortly after Kevin walks in. He shows up wearing a classic geek t-shirt with a broken Rubik’s Cube and a screw driver and the word “Cheater” underneath the cube.

image

We made our introductions, ordered our dinner and enjoyed our wine. Kevin asked me where I work and Hoff snorted. I deflected that for now and told him I used to work at DEC a long time ago. “Really? Did you know….” and he mentioned a number of names, most I recognized and one of them a friend of mine, Derrell Piper. I told him I was a VMS system manager back in the day and Kevin lit up and said that was his favorite operating system.

I told Kevin that I met him once there in Vegas in 1991 at a DECUS convention. “That’s the one where they revoked my pass!” he said. I didn’t tell him that we were strictly forbidden to talk to him back then. :)

I proceeded to discuss some of my history at DEC, how I worked on systems at LKG and eventually moved up to VMS Engineering. I told him I worked for Andy Goldstein and he said that Andy visited him in jail and brought him comic books. Kevin was very grateful and I told him I’d say hi to Andy for him. I mention a few more names and one of those was Dave Cantor. Kevin said “I remember him!”

The Question

That’s when, after another glass of vino, I said “I’ve got this question to ask you….” and proceeded to describe the “Cat got your tongue” scenario. As I went on, Karl and Hoff were silent and Kevin was politely smiling.

I got to the “What’s the matter? Cat got your tongue?” part and Kevin said “That was ME!!” Hoff roared and we all started laughing our asses off.

I waited for that confirmation for almost 25 years and to get it directly from the source was priceless.

I quickly emailed DaveC and told him I had confirmation. He responded within moments with “Yup. I don’t think there was ever any doubt, but I’m glad you got that confirmation.”

Apologies and forgiveness

Throughout the conversation Kevin apologized for any pain he might have caused. I’m a pretty forgiving person. It was almost 25 years ago and he has since paid his dues. It’s all good and I hold no grudge. Life is too short. There are plenty of people who have done things that landed them in jail and they have not turned their lives around and tried to give back to society like he has. So, he’s makes a living but it’s now an honest living. Good on ya mate.

On our way out of Caesars Kevin gave me a copy of his book. I know I should have asked him to sign it, but I don’t know why I didn’t. In hindsight I’m kind of kicking myself I didn’t have him write “What’s the matter, cat got your tongue?” Maybe another time?

As a wrap up to all this, the next morning Hoff Tweets about the night before.

image

It really was one of the best nights a geek like myself could ask for. I can’t thank Hoff enough for allowing me to join them for dinner.

I hope you enjoyed this piece of Geek History. I find I’ve been in the middle of a lot of it and that’s one of the coolest things a geek like myself can hope for.

mike