Proarch provides a key customer with agile teams to deliver a range of digital products. As the number of products and deployments has

increased, so has the Azure invoice. This has been exacerbated by SOX compliance and other strict governance requiring each of our customer’s clients to have separate deployments instead of building out multi-tenant applications.

With multiple suppliers, teams, subscriptions, applications, environments, developers, testers, customers, clients, users, and stakeholders, things had clearly got a little out of hand.

Sizing the problem

 

Data from our customer’s CSP indicated that monthly Azure spend had hit a heady £31,359. We set about identifying the real size of “the problem” and, whist the goal to “reduce the Azure bill” was clear, a more strategic approach was required, including specific objectives and measures around what we can save, and by when.

Setting expectations

 

 

 

 

After manual assessment and some debate on the “as-is” landscape, we identified numerous key strategies to reduce costs. Our projections indicated that we could potentially halve the spending. Recognizing that the customer operates in a highly regulated environment, and process and application changes would be required to achieve the objective, we gave ourselves 6 months to reach our target. We also agreed we would not reduce the end-user experience in any perceptible way.

Communicate regularly

Below you will find some of the key strategies to cost reduction. Being a data and analytics company, Proarch surfaced relevant analytics in PowerBI, published to the customer’s tenant and refreshed monthly to improve visibility.

Leader boards, regular refreshes, and a 30-minute monthly sync up were enough to keep momentum and continue to drive the right behavior.

Engage the business

Services deployed in Azure are provisioned by people or automated processes that people set up. By connecting deployed Azure resources with agile Product Owners, we generated a chain of accountability and visibility in the weekly Scrum of Scrum meetings. We found that the Product Owners were largely unaware of costs but were well-positioned to balance business and technical needs and influence the backlog to address inefficiencies.

We tracked this by building automation and Azure DevOps configuration to tag all Azure resources with the correct owner and an associated project.

 

 

• People and project names in the above have been pseudonymized.

Involve your techies

Presenting monthly costs by Azure subscription and service provides a solid foundation to root out savings across teams. With compliance forcing a separation between dev and ops in some environments, collaboration was crucial to get a handle on where improvements could be made, and which subject matter experts could contribute most.

 

Reap low hanging fruit early

During the initial assessment, we found Azure services that were not used at all! The best example being an HD Insight cluster, provisioned as a trial and never shut down. It had been running for months and only when we tried to find and assign an owner to the instance did it come to light. We removed it with haste, immediately saving £2,500 a month.

Like it, Lease it…

For non-production environments, we implemented a lease-based system. Every environment (project instance) has an owner. Every month the owner is asked if they want to renew the lease.

There are three potential outcomes:

1. Yes — the environment lease is refreshed for a month
2. No — the environment is torn down
3. No response — after 2 strikes the environment is torn down.

You said DevOps was a good thing

It is. DevOps is a very good thing. With the advent of DevOps, continuous integration and deployment make repeatable deployments easy. In some cases, too easy. Spinning up new environments is possible via a few clicks, one to test X, one to try Y, one to validate Z. Agility at its finest. However, with great power comes great responsibility, and the proliferation of new environments was exceeding the number of environments being cleaned up.

Providing accountability and visibility on a monthly basis built a culture of managing environments proactively.

Over two months we decommissioned SEVEN unnecessary environments.

Rightsizing

Bigger isn’t always better. Finding the right sizing for Azure services is an art. We monitored apps and instances and ensured they were scaled correctly. In traditional data centers, bigger was often better to maximize rack space, power, space, etc. However, with modern cloud PaaS, it really does pay to ensure you are not over-provisioned. Some services were found to be struggling to cope and went bigger, others smaller. In the end, we made a reasonable net gain and improved the process to scale up and down services more easily.

Up-front Architecture is dead, right?

In agile, architectures emerge. If your organization is siloed, Conway’s Law will likely ensure your technology implementations match. We identified such scenarios and applied architectural refactoring to consolidate and optimize the way we deploy and leverage Azure services whilst remaining within compliance boundaries.

 

Ride the Tech wave

Azure evolves fast. New services arrive, existing services change and old services are deprecated. CosmosDB recently released scaling per collection in addition to per database. This has allowed us to optimize the way we structure some of our stores and allocate RUs (capacity) where it is needed. Since I wrote the above three lines, Microsoft has now released a free tier for CosmosDB which will bring additional savings to some of our smaller deployments.

Rinse, Repeat, Kaizen

Accept that change won’t come easily or fast. You’ll be lucky to get one of the two, consider your customer a unicorn if you get both! The monthly sync up with project teams and owners keeps cost management in front of the mind. We discuss architectures for new products, changes in Azure, and new approached to improving the way we leverage Microsoft’s cloud and implement the best ideas.

Mission Accomplished

 

The short answer is YES!

 

The table below shows the continual improvement over the target to reach our objective of halving the spend over 6 months. The savings column is the Azure cost per month, compared back to the spend as of six months ago (from the most recent date).

Of course, when you knock > £15,000 off your Azure invoice, it makes a great business case to treat the team to a gift voucher or two and some additional time off…

 

 

PowerBI Dashboards

Below are the PowerBI visuals we use to manage our customer’s Azure spend. The names of projects and people have been anonymized with fake data, and the customer’s branding replaced with Proarch’s dark theme.

 

 

 

 

 

 

 

 

For any consulting requirements, please email us on cloud@proarch.com