Last time around I vented, more than a bit, about Voice XML and how the end result was arguably worse than any problem the standard set out to solve.  You’re as locked into your platform provided are you were 20 years back and apps are harder to build and maintain than they were on the villainous “proprietary platforms” VXML replaced.

Voice over IP, VoIP going forward, gets no such criticism from me.  It was inevitable and, honestly, a pretty good idea.  I mean, who wants to pull two sets of cables through an enterprise, one for phones and another for data, when you only need one set?

The vision of VoIP… as suggested by Google Gemini

The problem was the same problem we always have with tech.  We thought we were all ready for VoIP about five to ten years, depending on the enterprise, before we really were.

I mean, we had been through Y2K, we had upgraded everything we could, gig EtherNet was most places, and your CEO had heard about Skype.  We were clearly there in the minds of many.  Our marketing team, ahead of their time and also reading the same airline magazine as the CEO, declared that 85% of our new sales would be on VoIP in 2003… and we didn’t even have a solution yet.  Work to be done.

Along the way we had to dispense with a bunch of silly ideas before we got down to the reality of how things had to work.

On the list of silly ideas were stand alone VoIP phones.  Perhaps silly is the wrong word… but practicality didn’t quite enter into it.  For a stretch in the mid-aughts I had a blue Pingtel Xpressa phone on my desk.  Really a nice looking phone.  It won a design award.  It was a solid piece of equipment.

The black version of the phone… it came in various colors

It had a web server in it so you could configure it from your desktop and was entirely run in Java, because we were still thinking that the JRE was a good idea back then. That means if you can find one today… and there are some on eBay… that they will still work.  I mean, they are a security nightmare and would be running a Russian or Chinese bot net within 5 minutes of being exposed to the open internet, but it would still make a SIP call if you could configure it correctly.

We had some other, more pedestrian physical SIP phones

In our lab we had an OnDo SIP server (later Brekeke PBX) setup in the lab and could make calls into other lab systems through it.  We also setup the server so that dialing specific error response number would return that response.  Dial “404” to get a 404 error sort of thing.  Handy and lots of fun and of no practical use at all outside of a testing environment.

The base theory was that we’d all buy a phone we liked and subscribe to some sort of VoIP phone service or some such.  But non-tech people have issues activating freaking cell phones.  In what world was a phone you needed to get on your wired network… Wi-Fi need not apply at that point in time as it was still in the slow ages… and then configure from your PC going to be a viable solution?

Pingtel and a few other SIP phone manufacturers faded away.  They were all kind of expensive, and the Pingtel itself was super pricey in a world where people love a quality product right up until they see the price, then they buy the cheapest, low effort unreliable garbage they can find.

A hot market… in all the wrong ways…

Pingtel itself was purchased by Northern Telecom, Nortel, which was desperate for anything that might save it from itself.  Pingtel was just another rock thrown to a drowning man.  The joke about Nortel was that if you had spent $1,000 on their stock around Y2K and $1,000 on your favorite domestic beer in the can, five years later the redemption value of the cans the beer came in were worth more than the stock and you at least had the pleasure of drinking the beer.

A few such vendors, Pingtel included, tried to pivot away to a soft phone idea, but for consumers that removed the reassurance of a physical phone, left the config issues, and put them in a market where Skype, which mostly just worked and was free to start while they were expecting you to BUY their soft phone AND subscribe so some service or another to be able to dial… who exactly?

Skype you could pay a bit more to and dial not just your pals on your friend’s list but real phone numbers.  I used to use it for that.  Hell, for a while I had a ten digit phone number so you could dial into my Skype account from the public phone network.

The SIP phone thing was going to be cheap public options, like Skype and expensive proprietary options, like Cisco’s phones, and specialized implementations for things like call centers. And it remains pretty much that way today.  Sure, Skype has gone away, but Microsoft Teams, Zoom, and similar software has replaced it.

So it wasn’t too much later that I ended up with one of these incomprehensible bad boys on my desk.

A Cisco 7961 Series VoIP phone

I mean, it is a UI nightmare and almost overtly hostile to the average user, but that is what the IT department loves about it.  It makes them feel good to tell end users to RTFM when they cannot negotiate the garbage menu system of an overly complicated device that they forced on their company.  I mean, it must them feel good, right?  They wouldn’t keep making these sorts of fucking decisions if they didn’t enjoy it, right?

Not that I don’t still see old PBX systems and Nortel phones still in use.  But if your company buys something new, they probably buy a full service solution from Cisco or RingCentral or Mitel, or a few other minor players in the market.  Or they just force Microsoft Teams on you and let you figure out how the Teams app client and the Teams web client are actually supposed to be the same product.

But the market had to make that choice, and that took a few years.  If you work in an office now you probably have some ancient relic system, some spiffy but over complicated integrated VoIP phone and service… or you have a cell phone that you use exclusively and you can’t even remember if you have a phone on your desk, much less what it is.

Meanwhile, as this was shaking out, the network infrastructure had to be made ready.

Yeah, we installed a lot of gig Ethernet switches for Y2K.  It was a big bragging point to get that gig port in your office for a time.  But the network is more than just a speed.  There were a pile of issues to get through, starting with things like class of service… how do I know what this data even is… and quality of service… how do I prioritize things correctly… and a bunch of subsidiary issues that early solutions caused, like the great buffer bloat crisis, which I always denigrate largely because it was an issue of our own making due to everybody having the same easy solution to whatever their immediate throughput issue was, all of which meant coming up with standards that would work across equipment from different vendors… or even equipment from the same vendor but configured by different individuals.  I mean, I was there in the 90s when we couldn’t get caller ID to reliable talk across phone companies.

And those are just a few of the real issues and doesn’t even get into some of the dumb bits.  We had a customer buy the IP Contact Center package I inherited at one point and, despite the very clear language about network requirements, they expected it to work over their WAN between offices which consisted something like a 5BaseT coax connection that had no QoS configured and was being actively used for real time database replication.

These sorts of issues always start with the customer calling up and saying the software doesn’t work and then, hours to days later, the discovery of the actual issue.

Then there was the question of the IVR servers and how they would handle VoIP traffic.  We came into the 2000s with a few vendors like Dialogic and NMS selling very expensive PCI telephony cards.  They were making money, were happy with their margins, and did not want to spoil that.

The card vendors were doing so well that IBM bought Dialogic because its mark up on their products were so good, and IBM wanted them to get better with VoIP.

NMS, our chosen vendor, offered up IP integrations that worked with their pricey CG-6000 cards, while IBM had a software solution that used the Dialogic software interface that so many vendors had already integrated with, but a software VoIP stack that charged, per port, the same price as the physical card solutions from Dialogic.

There was a lot of that sort of thing going around.  One of the RAD Group companies was likewise offering up a VoIP stack for license at a rate that boggled the mind.

This in a world where your typical CEO thought VoIP was going to be cheap because they could use Skype without charge to call their kid at college.

And, along those lines, there was the question of how many lines could a software VoIP stack handle.  Yeah, NMS wanted tens of thousands of dollars for that 4xT1/E1 card, but it had the processing power to handle it and leave your server to do the other work it needed to do, which often included network calls to back end systems like the database, speech reco server, and TTS server.

Let’s take a quick time out to remember a major mobile and long distance provider, Sprint, that was a customer of ours who had outsourced their IT to IBM.  Everybody who did that regretted it because IBM was very much in the “promise everything just to get the contract signed, then do as little as possible” mode.

The IBM people told Sprint they could solve server network load issue by running all of those external servers on the IVR server.  So each one ran its own copy of an Oracle database, a Nuance reco server, a ScanSoft TTS server, and a few other items.  That not only did NOT solve the network issue… all those Oracle databases needed a constant replication scheme to stay in sync… but loaded up the local CPU so that each server could only handle a small number of VoIP sessions which meant they needed more servers which exacerbated the replication issue and… well, it wasn’t my problem so I could laugh about it.  Oh, and it cost the customer a ton to license all of that software for so many servers.  Our implementation guide explicitly said not to do dumb things like this, but you cannot fight stupid some days, and IBM level stupid was some serious world class stupid.

Tales from the old days.  But Sprint has since sold itself to T-Mobile a deal, like every such large merger, promised to add more jobs to the company and improve consumer choice, and led to huge layoff and less competition in the mobile phone market… which should have been the obvious outcome to anybody alive since the 1980s, but which somehow is always surprise when the lies are revealed.

Anyway, it took about a decade or so for infrastructure to catch up with the idea and for the market to figure out how this was going to be handled.  NMS went out of business.  IBM spun off Dialogic, which is still around, but is a shadow of its former self.  Meanwhile the big network hardware companies, already pals with the IT teams, stepped in and took their place, setting up the IP versions of call queues and call routing.

Once laid out, VoIP just followed the path that the web did.  VoIP is more prone to issues with network disruptions, but after years of upgrades and learning, it settled down to being pretty much an analog of how the web works.  Voice browsers take the call, routed to the by a load balancing mechanism, then requests the VXML pages, reaching our to the reco or TTS servers at need.  Those sit in front of the app servers which dish up the VXML pages and handle all the backend data issues, like querying a database or using some other connection method to support the app.

All in all, it is probably easier in the end as well.  I mean, I find it easier to use something like WireShark to figure out what is going on than I did trying to piece together data from a line analyzer and the call detail record that whatever platform we were using might have been keeping.

A win in the end and one that, unlike VXML, lived up to most of the key promises.  I mean sure, there was an obsession with “caller presence” and a few other splashy but impractical features early on.  But in just replace what the phone system did… seems okay… unless you have to admin a Cisco implementation.  But IT people tell me they love that sort of thing.

And all that specialized telephony knowledge I had gained up until then?  Mostly just trivia now.

The series so far:

• Hanging on the Telephone
• Calling the Mall
• Finding Friends with the Popcorn Lady
• Jenny and the FCC
• What Your Area Code Used to Say About Your Location
• I Get a Modem then Start a BBS
• Tech Support at the Village
• Dominating the PowerBook Modem Market
• The Death of eWorld
• The Great Dial Tone Drought of 96
• When the Modem Business Ran Out at the Village
• The Village Attempts to Move Beyond Modems
• Anecdotes from the Village
• Usenet Newsgroups Part I – I find some Usenet Archives on CD-ROM
• Usenet Newsgroups Part II – What is on that CD and Why Your Hard Drive isn’t a Database
• Usenet Newsgroups Part III – Founding, Fame, Influence, and Foreshadowing
• From the Village to the Island
• YoYo Mechanic
• The Great Failure of Residential ISDN in the US Market
• Set Adrift from the Island
• Thinking on Enterprise Software
• Stock Options and the IPO
• Into the Telephony Subsystem
• Rambling about Speech Recognition
• Web Banking, The Acquisition, and the Start of the Great Decline
• The Special Business Unit
• The Very Long Post about “The Move”
• I Pick Sides in the War between the Phone Team and IT
• The Great Failure of Voice XML