Bear On Business

Ike Elliott, a friend and colleague, has a blog called Telecosm. It is worth reading.

The entry below was posted a month ago but it immediately sucked me into the fall of 1997. At the time, I worked for Kiewitt Diversified Group or KDG. So did Ike. You probably will not find any mention of KDG on our resumes as we changed the name a couple months later to Level 3 Communications. I always wondered whether the industry credited Ike for coining the term “Softswitch.” I was there when he came up with the term, so I know first-hand that he did. Ike shares a little-known fact in his post–Softswitch replaced the name “SIOS”, the SS7 to IP Operating System.

Thank you, Ike, for sharing this story.  It inspired me to write down what happened in the two months prior to you joining us.

The Birth of The Softswitch by Ike Elliott
Posted January 09, 2008 on Telecosm

“O God! That men should put any enemy in their mouths to steal away their brains!” - William Shakespeare

When I joined Level 3 in 1997 to build the voice engineering team, I was immediately informed of The First Commandment of Level 3: Thou shalt not purchase any circuit switches.

Like Moses of old, my prophetic superiors had several memorable warnings to reinforce the rule: “The first person to sign a purchase order for a circuit switch is signing their pink slip,” and “Buying a circuit switch at a telecom company is like putting an open bottle of whiskey in front of and alcoholic.”

At the time, and for the century leading up to that time, circuit switches had formed the backbone of every voice network in the world. It was going to be real hard to build a voice network without them, but the challenge of building the first switchless voice network was what enticed me to join the company in the first place.

At least they had ideas about how to build this switchless architecture. They had a funky box-drawing on the back of a napkin, called the SS7-to-IP-Operating-System, or SIOS. I’m not completely sure, but I think is was Dan Caruso that came up with the SIOS name and did that first drawing.

My job was to take this outline and make it a reality, so the first thing I did was recruit a team of the finest engineers and program managers I have ever worked with, and likely will ever work with: Andrew Dugan, Steve Higgins (who hired Jin-Jen Wang), Carolyn James, Rick Steele, Bruce Baker (who hired Steve Branch), Ray Waibel. The list goes on and on. Some of the great ones were already there, like Rob Hagens, Kevin Dundon, Andrea Gavalas, and Jack Waters, and we got a few more later on, by acquisition, like Jon Peterson and, briefly, Shawn Lewis.

The next thing I did was coin a better term for SIOS. We were trying to emulate a circuit switch in software, so I renamed the thing the softswitch, and in my first presentation to Jim Crowe (who was still in Omaha at the time), Jim latched onto the name, and like an aggressive virus, the term started to infect the industry.

The softwitch name spread quickly because as we constantly talked with vendors about what we wanted to buy, the vendors started to use the term, too. Of course, none of them had a softswitch to sell us, so we were getting kind of worried about actually building a softswitch-based business. Several of us were commuting from Colorado Springs to Denver, and on the long car pool rides we would brainstorm about building a softswitch out of the then-limited industry “parts inventory.”

One of the options came from an unexpected source: a few weeks after I joined Level 3, Ron Vidal pulled me into Dundon’s office and described a company out in Cambridge that might have a softswitch. Ron and I paid a visit to XCOM Technologies, and liked what we saw, and later Jack Waters and I paid another visit, and on April Fool’s Day 1998 we announced the acquisition of XCOM for about $165M in overvalued Level 3 stock. We didn’t know it at the time, but that acquisition has to rank at the top or near the top of all of the acquisitions Level 3 has made.

XCOM had a rudimentary softswitch prototype going, built with Visual Basic, PCs, and some rubber bands, but the real important piece was that XCOM had developed a protocol for controlling dial-up-modem banks from their softswitch, and had sold Ascend on building that protocol into their network access servers. It was the first real application of the softswitch concept, even though it was initially designed for the “half-call model” of dial-up internet access. Shawn Lewis was the CTO of XCOM, and deserves a ton of credit for building the protocol first.

Despite all of the due diligence, we found that the prototype was not scalable and not ready for prime time. Even after the engineering team, under the direction of Hagens after Lewis quickly departed, did 8 months more software development and we launched the managed modem product, the softswitch was highly unstable and crashed at just about every meal time. At one point we were doing pre-emptive softswitch reboots every night, just to keep the memory leaks from causing prime time crashes.

It took more patience than the execs thought they had, and more sleepless nights than the engineering and operations teams thought they could sustain, to nurse the platform to stability. Our customers also deserve our everlasting gratitude for sticking with us while we worked out the bugs. But our patience and hard work paid off, and within 15 months of launching the platform, we had the most reliable managed modem service in the industry, with an 80% cost advantage over our competitors. Our market share quickly grew, and at its peak the managed modem business was responsible for over $800M a year in high-margin revenues for Level 3, before broadband internet access started to erode the dial-up subscriber base.

Any of us who were involved in the effort look back on those times with a mixture of pride and pain. We have the certain knowledge that we created a groundbreaking platform and a huge money-making business. However, the effort required a great deal of sacrifice from a great many people, which makes it very hard to repeat. Many other key people joined the team and made great sacrifices: Brent Bourne, Kevin Paul, Ken Fischer, Dan Rock, Jason Bach…far too many to list, but all of them crucial to the ongoing success of the business unit.

Some say that Level 3 could not have averted bankruptcy during the bursting of the Internet bubble without the softswitch. I don’t know if that’s true, because Level 3 had some pretty awesome financial engineers that frequently worked miracles of their own. I do know, though, that the softswitch made the job easier for those financial engineers than it otherwise would have been.

Why did we do it? The promise of valuable stock options played a role, yes, but I think it was a greater factor for the team to know they were doing important work, and to know they were part of a great team. And one thing I know, for sure, is that without the First Commandment of Level 3, we would not have been the first to build and deploy an operational softswitch.

And, at the time I left in 2003, we still had no circuit switches.

Sure enough, Jim Crowe called me the following week. Though I still don’t know for certain, I suspect KOH encouraged him to take another look and give me a call. Either way, it startled me when I picked up the phone and heard “Dan, Jim Crowe here.” I doubt he ever called me prior to that. I idolized him but was intimidated at the same time.

“I took another look at that drawing. You might be onto something. Tell me how it works.”

I have no recollection of whether we talked for 2 minutes or 20 minutes. In any case, I must have been inspired to plow forward. I know I was excited.

Shortly after that, Kevin Dundon told me he has someone he wants me to meet. It was a guy that Jack knew from MCI–a guy that was working on how circuit switches need to work in the world of IP. “This guy knew all there was to know about SS7,” KD said. This is the guy who could architect the SS7 to IP vision.  He’s in my office now. You want to come talk to him?”

“What’s his name,” I inquired.

“Isaac,” responded KD.

“Isaac?” I repeated with a puzzled look.

“I think he likes to be called Ike,” shrugged KD.

I went in and, for the first time, met Ike Elliott.  KD had already taken him through the SIOS drawings and call flows. Ike was not familiar with LEC/co-carrier interconnection – his entire perspective was based on MCI, the innovative long distance carrier. But Ike was deeply familiar with the aspects of the call flows and interconnection concepts.

Ike nodded his head. “This could be fun,” he might have said.

Tomorrow, I will reprint an entry from Ike’s blog. It picks up the story from the day Ike saw the drawing and then joined us. It describes how Ike and a very talented team turned these crude ideas into the Softswitch.

(continued from yesterday’s post)

“How do I think it went?” I sarcastically responded. “It was the worse day in my career,” I relayed, with a tone that I was simply acknowledging what was obvious. Worldcom was now up about 20% since I left, and I was convinced more than ever that I made a mistake leaving Bernie Ebber’s fiefdom.

You see, Jim Crowe laughed at my drawings. He skimmed my call flow pages but didn’t even listen to me when I tried to explain what they meant. I recall Jim was in a jovial mood–Josh Howell had just resigned from Worldcom to join us, and Josh popped in just as we sat down. The conversation centered around the good old MFS Communications days and how the team was coming back together.

I wanted to talk about my stick-figure PowerPoint drawing. I knew it was crude and detailed–but was convinced it explained what Jim was talking about. But I found out I was wrong. Jim’s reaction was to flip though them, chuckle, and tell me to keep thinking about it. He seemed anxious to move onto something far more worthwhile of his time. I was devastated.

Kevin O’Hara and I were at the Omaha airport when he asked me the question, “How did you think that went?” I am certain I looked shell shocked. When I answered, “It couldn’t have gone worse,” Kevin simply replied, “I’m not so sure. Jim is a reader. My guess is he picks up what you gave him and reads it over. I’d be surprised if he doesn’t call you next week and ask some questions.”

I rolled my eyes.

(to be continued…)

It was either a Saturday or Sunday. I woke up early. I spent 3 or 4 hours and sketched the drawing from last week’s post. In one of the subsequent edits, we put forth the name “SIOS” or “SS7 to IP Operating System”.

With the drawing, I had three pages of call completion outlines, which described how SS7 would work to set up and complete the calls. It described how “gateways” would act as interfaces between traditional phones and with the PSTN. It showed how PSTN databases–such as Local Number Portability, or LNP–would be attached to the Internet cloud. All of this was crude but it was my attempt to provide substance to the notion of how “the marriage of SS7 and IP” would be the foundation of a whole new type of Internet telephone company.

I couldn’t wait for Monday morning to arrive. I knew I was onto something and couldn’t wait to share it.

Kevin Dundon–who knew about as much about this stuff as me–was the first I shared it with. He understood it in a second. We shared it with the others, including Jack Waters, John Scarano and Andrew Morley. They all seemed enthused, though their understanding (perhaps with the exception of Scarano) of some of the concepts was not as deep as KD and mine (due to our more extended work in LEC interconnection).

Next step was to sit down with Kevin O’Hara. “None of us were sure what Jim Crowe envisioned when he referred to the interconnection of SS7 and IP,” I explained. “After we batted around ideas among the team, I came up with this.” I showed him the drawing and took him through the call flows and other concepts.

Kevin hadn’t been involved in the nitty gritty details of co-carrier/LEC interconnection. He also left shortly after MFS Communications closed on UUNET so he didn’t have as deep of an understanding of the Internet. I don’t think he spent the time I did with Jack either. I knew going in that he didn’t understand what Jim was thinking. He listened to me but didn’t comment much.

“I’m heading to Omaha later in the week. Why don’t you join me and you can walk through this with Jim?”

I was excited. I knew Jim would be pleased. This would show Jim that he made a good decision to pry me from Worldcom. I couldn’t wait until we met.

(to be continued…)

(continued from Friday)

Telecom guys were intimidated by Internet guys. I knew about fiber optics, what SONET was, and how LEC interconnection worked. The latter was a big deal. I led MFS’s effort to interconnect its local switched networks with the RBOCs. These interconnection agreements–and the network interconnections that resulted–were brand-spanking-new. Very few people understood them. At that point in time, I likely had as good of an understanding (from a business perspective) as anyone in the industry. However, when it came to the Internet, I was very aware that I didn’t understand much–at least not compared to a guy like Jack Waters.

“Jack, what do you make of Jim’s notion of connecting up SS7 and IP?” I asked.

“Sounds like a great idea,” he responded casually.

Did he know what Jim was talking about? I couldn’t tell. He played off like he did. Did he think I knew? I played off like I did. I don’t know who let on first that they had no clue.

You see, Jack knew the guts of the Internet inside and out. He was part of the in-crowd who defined peering. He knew all there was to know about what TCP/IP was. He was integral in building the biggest Internet backbones of the era. However, I discovered, he knew about as much about SS7 and LEC interconnection as I knew about the Internet. Why would he know anything about these? His entire career was in the world of the Internet–which back then had nothing to do with the PSTN. In fact, the Internet guys despised the PSTN–it was archaic and, oh yeah, not scalable. “Scalable” was one of the favorite terms of the Internet crowd.

Jack was able to explain what was meant by “Internet Protocol”. He educated me on how peering and transit worked. He helped me begin to get an appreciation for what the Internet really was. I had done a fair amount of work on implementing massive Internet projects for UUNET. The education provided by Jack helped me piece together the work we did for UUNET. Through all of this, the power of IP as compared to the PSTN was becoming more clear.

By this point, it was the end of the week. I still didn’t know what Jim was talking about, but now I knew a bit more about IP. My mind was racing.

(to be continued)

(continued from yesterday)

I got into work early the next day. In those days, I drank Diet Pepsi, 20 ounces at a time. I was on my second by the time Kevin Dundon pranced in the office.

I gave him five minutes to settle in. I walked into his office and shut the door behind me. Small talk between KD and me consisted of me saying “hey” and him responding “huh”. So we quickly got past the small talk, and I dove right in.

“You know when Jim talks about SS7 and IP interconnection?”, I asked.

“Yeah, what about it?”, KD responded with his famous nervous chuckle.

“Do you have any clue what he is talking about?”, I grimaced.

KD looked somewhat startled. “I thought you knew,” was all he uttered. Damn. I felt better that I wasn’t the only ignorant one–but I also felt a bit fearful. Now we both would look like idiots.

The next step was to ask Scarano; “No idea” was his answer. Jon Yount was no help. Andrew Morley wasn’t in the weeds of LEC interconnection so he wasn’t able to contribute to discussion. Matt Geraghty was riding his bike.

Around this time, we hired our first Internet guys. Not a bad idea considering KDG was supposed to be an Internet company. Jack Waters, Joe Lawrence and Rob Hagens joined us from Internet MCI. I don’t recall if all three were there by November 1997, but at a minimum Jack was around.

“Maybe Jack knows what Jim is talking about,” I thought, feeling lots and lots of angst.

(to be continued on Sunday)

(continued from yesterday’s post)

It was October 1997. We had been at this KDG-thing for more than a month now. We were tasked with developing a business plan around the notion of SS7 and IP being interconnected. Huh? 

Remember, the year was 1997. I had only used the Internet for the first time in 1995. The expression “Internet Protocol” was rarely used by anyone–certainly no one I knew other than Jim. I was still trying to figure out how the Internet worked (okay–maybe that was a bit of an exaggeration in that I did a fair amount of implementing Internet networks as part of running major projects for UUNET.)

SS7 is an arcane protocol, created by Bell labs many years ago. It refers to how traditional telecom networks set up phone calls. When you dial someone’s phone number, the SS7 network kicks into high gear. It translates the phone number to a specific destination, checks to see if capacity is available in the phone network to complete the call and triggers the ringing of the phone on the other end. If someone answers the phone–and ONLY if someone answers–SS7 establishes a link between the two phones and then moves onto doing the same thing for the next phone call.  

What on earth does this have to do with IP? Oh, and by the way, what does “IP” stand for again? That’s right, Internet Protocol. 

Now I started to toss and turn at night. I was the leader of the team. Despite my strugglings, I knew more about this than anyone else on the team. At a minimum, I was convinced the rest of the team–including Jim and Kevin–expected me to know about this stuff. 

So I did the only thing someone in this position could reasonably be expected to do. I called Ron Beaumont and asked for my job back at Worldcom. Unfortunately, he said “no”, so I moved to Plan B.

Plan B was to confide in Kevin Dundon. He knew as much about this stuff as me–maybe more. KD would explain to me what Jim was talking about. Dundon might poke fun at me but at least I’d sleep better at night. If I was lucky, he would keep our secret between us.

So that is what I did…  (more to follow)

I left Worldcom in the summer of 1997 to join Kiewitt Diversified Group (KDG). At the time I agreed to join, only Jim Crowe, Kevin O’Hara and Doug Bradbury were part of the team. I knew Kevin well, but my relationship with Jim was fairly formal. In those days, Jim wasn’t the easiest guy to have a casual conversation with.

From the day I joined, Jim talked about a notion he had. “We want to build a business around interconnecting SS7 and IP networks,” he declared in September of 1997. I’d listen but didn’t comment. I suspect I nodded my head affirmatively, though I had little idea what he was talking about. I figured sooner or later he would shed a bit more light on the concept.

A few weeks passed. I heard him say it several more times. “Circuit switches will become obsolete.” “Voice will become free.” “All of this will result when the SS7 and IP worlds come together.” During these weeks, several more people joined us. Kevin Dundon. John Scarano. Matt Geraghty. Andrew Morley, Jon Yount. All guys who have been part of telecom for many years. All were guys I worked closely with at MFS and Ameritch.

After a while, I realized none of these guys asked questions either. In fact, no one talked about it at all–not even when Jim was back in Omaha. I noticed that they also nodded affirmatively every time Jim brought up the topic.

This went on for about two months. We were expected to write a business plan, with me playing the lead role in pulling it together. The notion behind the business plan was the interconnection of SS7 and IP. Maybe if I listened a bit more carefully, a light bulb would come on in my head. Maybe I shouldn’t have left Worldcom–the stock was up at least 10% since I resigned.

It became harder and harder to get a good night’s sleep… (to be continued)

This drawing is from November 10, 1997. The next version of this was given a name. A couple of months thereafter, a new name was given. This name became a household word–okay, only if your household included someone from the telecom industry. Do you know the initial name? Do you know the name that is now used? The answer will be provided in two days.