Disastrous Decision Making in IT

Disastrous Decision Making in IT from Robert Limbrey

The dichotomy of Capacity Management in a private cloud

The Fable

The two great heads of IT sat and stared at each other across a meeting room table.  It was late in the day, and thankfully their services had all been restored.  Now was the time for recriminations.  The CIO had been called into firefighting meetings with the board all day.  They knew he was going to be royally pissed off, but who was going to get the blame?

The beginning

The story began when service performance nose-dived.  It was always a busy period, the lead-up to Christmas, but this season had been marked by some hugely successful promotional campaigns, and their online services had been humming with traffic.  Nobody quite knew what caused it, but suddenly alarms started sounding.  Throughput slowed to a trickle - and response times rocketed through the roof.  Downtime.  At first the infrastructure team, plunged into a triage and diagnostics scenario, did what they always did.  Whilst some were busy pointing fingers, they formed a fire-fighting team, and quickly diagnosed the issue - they'd hit a capacity limit at a critical tier.  As quickly as they could, they provisioned some additional infrastructure and slowly brought the systems back online.

The background

But why had all this happened?  Some months ago, and at the advice of some highly-paid consultants, the CIO had restructured the business into a private cloud model.  The infrastructure team provided a service to the applications team, using state-of-the-art automation systems.  Each and every employee was soon enamoured with this new way of working, using ultra-modern interfaces to request and provision new capacity whenever they needed it.  Crucially, the capacity management function was disbanded - it just seemed irrelevant when you could provision capacity in just a few moments.

The inquisition

But as the heads of IT discussed the situation it seemed there were some crucial gaps they had overlooked. The VP of Applications confessed that there there was very little work being done in profiling service demand, and in collaborating with the application owners to forecast future demands.  He lacked the basic information to be able to determine service headroom - and crucially was unable to act proactively to provision the right amount of capacity.  In an honest exchange, the VP infrastructure also admitted to failings in managing commitment levels of the virtual estate, and in sizing the physical infrastructure needed to keep on top of demand.  In disbanding the Capacity Management function, they realized that they had fumbled - and in fact needed those skills in both of their teams.

The Conclusion

The ability to act pro-actively on infrastructure requirements distinguishes successful IT organisations from the crowd.  What these heads of IT had realised is that the private cloud model enhances the need for Capacity Management, instead of diminishing it.  The dichotomy of Capacity Management in the private cloud model is that these two functions belong to both sides of the table - to the provider, and to the consumer.  Working independently, they would be able to improve demand forecasts and diminish the risk of performance issues.  Working collaboratively, these twin dichotomies combine in a partnership that allows a most effective way of addressing and sizing capacity requirements to align and optimize cost and service headroom.

Take-aways

1. As a consumer, ensure you are continually well-informed on service demand and capacity profiles.  Use these profiles to work with your application owners in forecasting different 'what if' scenarios.  Use the results to identify which are the most important metrics, and prepare a plan of action when certain thresholds are reached.
2. As a provider, ensure you are continually tracking your infrastructure commitment levels and capacity levels.  Use the best sizing tools you can find to identify the right-size infrastructure to provision for future scalability.
3. Have your capacity management teams work collaboratively to form an effective partnership that ensures cost-efficient infrastructure delivery and most effective headroom management.

Will you wait for your own downtime before acting?

Dear CFO - 4 factors to evaluate before you invest in new IT

written in response to this great article on WSJ Blogs: http://t.co/lbyNeeMlxm

Dear CFO,

  before you decide on your strategy for investing in IT assets, whether on-premise or public cloud, there are a couple of important correlations I think it's important for you to consider.  

  Clearly, your decision ultimately will be based on:

• The fit with your longer-term business plans
• The measurable benefit to your business
• The investment needed

  Investments in IT capacity enable your business to transact a certain amount of business.  In similarity with other areas of your business, investments in capacity should be made to alleviate bottlenecks and increase the ability to transact business.  However, in IT there are certain complexities to factor in:

• comparing and contrasting capacity options has descended into a "dark art", with many stakeholders and an over-riding aversion to risk
• measuring capacity usage has become a specialized platform function, leading to difficulties in getting an end-to-end perspective of how much business can be transacted
• an increasingly agile enterprise is causing rapid fluctuations in capacity requirements, again with an aversion to risk

  Long ago, a management function was created to address these problems for the mainframe platform - Capacity Management.  That function can be leveraged again now, to allow you to plan ahead effectively for your long-term IT future.  Evaluating that function within your IT department, you should ensure it:

1. endeavours to provide a complete picture of capacity usage across all silos
2. provides visibility of service headroom, potential bottlenecks and abundances
3. couples together with your financial management controls, providing governance over capacity allocation
4. gives insight into future business scenarios, allowing investments to be rebalanced against the needs to transact business

In summary - if capacity management can be harnessed to better manage the costs of IT capacity, greater focus can be made on transformational activities that add value in other ways.

Accelerate Innovation - by increasing efficiency

I'm always highly impressed by the number of innovations coming from IT teams to tackle unique business challenges.  Some organisations leverage the power of Business Intelligence to improve marketing and operational analytics.  Other stories highlight the importance of connecting people and improving collaboration.  The more I read about IT innovators, the prouder I feel of our industry in general -> staffed full of amazing problem solvers, equipped with amazing technology to help them along the way.

But the life of an IT executive is not all about innovation.  That's the headline grabbing stuff.  If you'll pardon the duck analogy, there is an awful lot of paddling going on under the water to keep the lights on.  And when you look at the data - where up to 70% of the IT budget is being spent on operations, the struggle becomes immediately apparent.  

But what if there was a way of liberating some of that 70%?  After all, most datacenters are operating at less than 20% capacity at peak.  One UK customer told me that their windows estate was running at around 7% of capacity - during peak hours.  Even assuming a high-availability scenario, where capacity levels should be maintained at less than 50% --> there is still a significant amount of overspend on excess capacity: possibly as much as double.

Were we to apply some of our innovation capabilities to this problem, it should be clear that we as an industry are struggling with a conflict of interest around sizing.  Every risk-taking bone in our bodies is shouting out "more!" - the last thing we want to be associated with is a non-responsive service, crushed beneath the weight of demand.  Whenever we meet with a vendor, an outsourcer, a new market - the answer is always "more!".  

Entropy never decreases, and the voices for "more!" are louder and more consistent than ever.  And so we face a challenge.  "More!" is expensive.  Looking at Koomey's article on TCO, we can see that capital expense (software and hardware) accounts for about 1/3 of buying an asset.  In this model, a $10,000 server would have TCO of around $60,000 over a 3-year period.

A simple financial calculation should help deduce that running low utilization in datacenters may be wasting a huge amount of our IT budgets, on power, on facilities, on routine administration tasks.  For every 100 servers running at 10% utilization, that could equate to over $1M in potential savings every year.  

This has always been the great argument for virtualization.  Consolidate these windows servers, without introducing risks of compatibility, or costs for administration.  The great wave of virtualization has swept over us, leaving many organisations with highly virtualized landscapes, and little further room for optimization.

Except, that assumption is wrong.  Native sizing tools for virtualized landscapes depend on inefficient algorithms, using a flawed assumption - that every MHz is the same.  It turns out that we are still oversizing our virtualized landscapes - by as much as 2x.

The point of sizing efficiently is not an arbitrary one.  Every server that is not provisioned may be saving the enterprise $60,000 or more.  Re-investing these savings into creative projects will only help drive successful 

innovators, and generate more and more CIO headlines.

Capacity Planning the #Devops way

The notion of #Devops serves to accelerate time to market through greater cohesion in the release management life cycle. 

So called 'service virtualisation', such as offerings from IBM and CA LISA, enables modular testing practise by learning typical behaviour patterns of defined systems. The effect is a more tightly focused testing process that reduces the dependency on external (inert) services.

Release Automation, such as in the newly acquired Nolio solution, allows the testing process to be further streamlined by providing cohesion through the multistage process. The benefits are most highly felt where complex dependencies and configurations add magnitude to setup and teardown for QA.

Agile methods need agile release management processes, and this is the whole point of #Devops. However the risks in this agile thinking come in end- to-end performance.

The missing link here is provided by prerelease Capacity Planning (such as provided by CA Performance Optimizer) , a virtual integration lab that brings together the performance and capacity footprints of each component in the production service. And while some of those components may be changing and therefore measured through the release management process, others are not - and are measured in production. Creating and assimilating component performance models allows the impact of each sprint to show on IT operations.

Capacity Planning is a true #Devops process.  Only by adapting the capacity plan to take into account the step changes due to software release, can the risks of poor scalability and damaging impact be accurately guarded against.

Take a Capacity Healthcheck

Heart, lungs, liver, kidneys: we all recognize the importance of looking after your vital organs - and regularly take medical advice whenever we have concerns.  Yet it is not the fitness of individual organs that causes concern - it is the fitness of the weakest. 

But what about our IT enterprise?  It's not uncommon - even in 2013 - to come across siloed thinking that stifles the health of the organisation.  Purchasing decisions are made within the silos, leading to distorted allocations of capacity based on political, rather than engineering needs.  Further - due to the manifestation of these silos, entropy increases as financial accountability struggles to permeate the organisation.  Provisioning decisions are made on a risk-averse basis without insight into how business demands translate into capacity requirements.

And now it's time to change.

The financial crisis has caused a significant change in emphasis in most major corporations.  IDC estimate that over 50% of major corporations are actively planning investments in better capacity management functions.  Cloud-sourcing is on the increase to assist with the deferment of cost, and a refocus of investment on the core business.  Capacity has become a commodity, and in an open economy, is becoming subject to the same commercial forces that balance cost and quality in any marketplace.

But how can enterprises leverage their purchasing power, when they don't know how much commodity capacity they really need?  How can they right-size investments and defer costs without incurring risk to their top-line revenues?

Actually, this is a question that enterprises have addressed in many of their other lines of operation.  Full financial accountability has ensured the right-sizing of many other enterprise assets; whether that is employees, hot-desks, freight containers or manufacturing capacity.  Successful companies have figured out that costs must be aligned proportionately with revenues.  The only thing that makes IT different is the complexity, and the lack of insight.

So where to start in right-sizing IT?

The answer is perhaps startlingly clear - you should begin where you always begin, with your requirements in mind: measure capacity consumption across all silos.  The trick then is to bring in a method of normalization, a model library that provides weighting factors according to the make and configuration of your estate.  This same method can then be applied to plan a migration, transposing configurations easily to determine the optimum sizing on alternative real-estate.

So what's stopping you? Find a friendly capacity management service provider and ask them about their IT Capacity Healthcheck. If they know what they're doing: they'll get their scheduler out straight away.  And if they don't? Well - drop me a tweet or an email and I'll point you in the right direction.

Consumer/Provider : the twin forces in Capacity Management

Those schooled in traditional IT capacity management have long recognised the cause and effect of observed system behaviour.  Few have managed to bridge the gap in quantifying the correlation, however, and for good reason.  Straying too deep into this territory can leave you struggling with data overload, and no way of mapping volumetric and utilization data together. The age of the CMDB and automated discovery and mapping has changed the landscspe in thus regard.  Finally, using configuration mapping to correlate volumetric data against utilization data can be done reliably, consistently and accurately since all feeds are automated.

Correlating service throughput against observed utilization provides intelligence to optimise design, streamline performance, & predict and optimize application scalability.  But in a consumer/provider scenario there are two contexts to consider. Presenting the customer with data about your underlying infrastructure utilisation lays bare the margin or risk levels of your operating model. Equally, the customer's main concern is ensuring their service levels are not jeopardised, and they are not burdened with excessive costs for underutilized environments.

Despite the advantages commonly sought in quantifying the capacity of the physical environment, it is the capacity of the contractual environment that is crucial to the customer. In a cloud context, the provider must diligently ensure the reliability if their operating model. This is crucial to brand equity. But the customer's primary concern will be in managing the flexibility of their service based contract, and ensuring that risks are properly balanced against costs.