Problem
One day a user of Turbo360 got an anomaly detection alert. There was a hugely expensive resource had just been deployed into Azure.
This resource was not in an area the user was familiar with and it was the day before everyone goes on xmas holidays.
Analysis
As response to Turbo360 alert the user was checking in the Turbo360 analysis section. You can see the huge cost spike below.
.png)
The resource this mapped back to was a SQL Azure database which had been provisioned in the last few days but had been dialed up to the largest possible SQL azure database size.
Upon checking with a few people no one knew what the database was for.
The application area owner reached out to a 3rd party who was engaged doing some work on an application in this area.
Initially the vendor was not sure either.
Upon investigation it turned out the developers in the 3rd party team were doing a migration of data from on-premise to Azure. To speed up the migration they had dialed up the Azure database to the maximum possible size.
They had however not informed anyone of their plan to do this and just assumed no one would notice and they would dial it back down before anyone noticed.
Solution
The problem here wasnt really that the customer had a scaled up database, the vendor may have had technical reasons for doing this. The problems really were two key things:
There was no communication or approval process in place to inform people of this change
With it being the last day before the xmas holiday, if they had forgotten to scale it down when done the customer would have spent $20k while everyone was on holiday
Fortunately in this case the Turbo360 anomaly detection caught this issue and the risk was mitigated.