Black Ops
For those of you who came
in late, WITDW is all about what’s wrong with IT at the Fortune 500 level–why
too much money is spent to produce systems that are late, don’t meet their
specifications, aren’t reliable, aren’t integrated, or just plain never work.
If your company doesn’t
have these problems, go read something else.
If it does, stick with me
and I’ll describe the problems, with real-world examples, and outline some
solutions that will start to solve these problems. What I propose may only make
your IT operations 5% more efficient (although I believe a 40-60% improvement is
attainable, over time). Let’s see, you run a $20B company that spends, say, 3%
on IT. That’s $600M (that you know about—more in a subsequent column). Let’s
just save 5% and we have $30M in savings.
And you know what? That
$30M (or even couple hundred million) probably isn’t even the important part of
this regimen. The important part is getting decent, integrated, reliable,
applications online on-time to help you get and retain customers, meet
regulatory requirements, hire and retain top employees, and build your products.
And, no, I’m not just
talking about adding little webby front-ends to your applications. Those are
easy. Getting a face-lift is a lot easier than curing a brain tumor, and your
systems almost certainly have brain tumors the size of peaches.
OK, for those still with
me after that unsavory analogy, let’s begin.
See No Evil
I’ll start with a study my
consulting partner Fred and I did for a cellular firm (a subsidiary of one of the RBOCs, not one of those startups with the clever names). The story is literally
true–not an allegory or urban legend. As usual, I will tell the story so as to
emphasize the one or two action items I will propose at the end of the column
(and I will call the firm Big Wireless Company or BWC). The goal here is not to
show how smart we were or how dumb the company was. The goal is to have you
apply these lessons to your enterprise.
BWC had begun selling
wireless voice several months prior when we were retained to audit their
customer billing system. BWC had hired a premier systems integrator / data
center operator (which we’ll call Giant Data Centers, GDC) to both build
their system and to host it at their data center.
At the point we were
called, the billing system was limping along, barely able to process the number
of calls logged each month, and badly needed new software features were months
late. The IT cost of each bill sent, i.e., the charge for data center usage,
billed back to the wireless company by GDC, was a large fraction of the average
customer bill. (Notice that the relationship here is similar to Cadillac
designing and maintaining your car and then selling you the gas, at a large
markup. Please supply your own estimate of how incented GDC was to spend time
redesigning and optimizing the software to attain better performance.)
We spent several days
onsite and at our office interviewing staff and running analysis programs on
data we had collected. A few highlights of our analysis:
-
The computer resources consumed per business
transaction were ten to a thousand times those observed in similar
applications we had studied.
-
The software requirements approval processes,
historical and ongoing, were totally inadequate. Essentially every idea for
system function proposed by an end-user became a system requirement. (Sorry,
users, that doesn’t work.) These requirements were then prioritized by one
under-experienced member of the technical staff without any discernable
rational analysis of development cost, hardware cost, or the dollar benefit
to BWC. His process was based essentially on intuition and who called his
office the most to complain, as near as he could explain it.
-
The system was then built piecemeal from these
requirements, essentially leaving out that pesky step of overall system
design and validation. (This largely explains the first observation above.)
-
At the inception, the project had put in place
reasonable processes and standards for software development. These were
abandoned as soon as possible.
And on and on—far too much
for this today’s column, but not too much for the 14 page report we sent to the
Project Director.
And now we come to the
interesting part (thought we’d never get there, I’m sure): What did BWC do with
this detailed list of critical problems and recommendations?
The Project Director acted
decisively. Within a day of receiving the report, he faxed us back an amended
copy with all the paragraphs critical of BWC decisions blacked out. The
report looked like one of those declassified CIA documents about UFOs. His
scrawled note instructed us to excise the marked sections and emphasize the
other sections, all of which referred to missteps by their contractor, GDC.
What did we do? After a
little shameful dilly-dallying, we submitted the document as written, and
subsequently briefed it to the CIO of BWC and his boss, the CIO of the RBOC
(along with his senior staff).
What did they do? The BWC
CIO, to use his own words, “fell on his sword,” accepting that errors had been
made on his watch and showing a keen interest in possible solutions. His boss,
the RBOC CIO nodded sagely. The Project Director sat in a corner and scowled at
us.
What should you do? Hire
consultants at $350 an hour to audit your systems and point out all your
mistakes? Well, actually, yes. But if that sounds too unpleasant, you have a
couple options. You can make sure to tell the consultants ahead of time what
answers you want, to avoid that messy work with the black marker. Many really
high-priced and famous firms won’t even need to be told. They’ll guess
what you want to hear and write it up beautifully. Now that’s service.
Oh, there is one more
option, if you’d like your systems to actually work as promised: Hire
management and staff to build your applications who have successfully built
applications of the same scale with similar technology. In this regard,
relevant experience trumps book learnin’. Our Project Director was an EE and
seemed smart enough, but he didn’t know anything about building a complex
software application. Most of the technical staff from BWC and GDC were pretty
capable, except they had never used the technology chosen for this project.
And finally, insure
skeptical oversight. Even if you build a perfect team, make sure the project is
reviewed by disinterested third party experts, early and fairly often. You’ll
cause some friction, but you’ll reduce the risk of catastrophic failure
greatly.
Oh, and how did this all
work out for BWC? Don’t know the details, but I do get a cellular bill every
month.
I check it very carefully.
|
