IBM Storage for Data Resiliency Explained, including IDCD…

Where does IBM Storage Virtualize fit in the bigger picture of Data Resiliency?

Earlier this week I mentioned that 8.6.0 was now available, and one of the features it includes is Inline Data Corruption Detection. A few people have asked for some more details of what this does, but first off, have a quick look at this great summary of IBM Storage for Data Resiliency by Ian Shave :

Inline Data Corruption Detection

You may think that by the time data gets to a block storage device that it is just a random series of 1’s and 0’s – and while this is partially true there is a whole area of information mathematics that deals with just how random that data may be! To keep it simple, we are using Shannon Entropy here to calculate an entropy level of the data streams we are receiving – in Shannon’s terms, deriving an ‘information level’ in the data streams.

The information content, also called the surprisal or self-information, of an event E is a function which increases as the probability p ( E ) of an event decreases. When p ( E ) is close to 1, the surprisal of the event is low, but if p ( E ) is close to 0, the surprisal of the event is high. …

Put simply, we are sampling the destage streams from the cache to determine the overall Shannon Entropy nature of that stream – which may be a signal that a ransomware or other data corruption event is taking place.

Now there could be many reasons for the entropy change, and Machine Learning (ML) is needed to understand this and therefore real-time report when we spot something unusual. This is not something we want to dedicate valuable Storage system cores to performing. The information is therefore being streamed to IBM Storage Insights where the ML and end reporting/alerting can be performed. The idea is that over time we build a data lake of signals and use ML to understand these over time, train models that can act in near real time to then alert you the end user, or more likely your SIEM.

This is just step one in the process, and of course is just one of the signals you will be using as your Cyber Resilience maturity increases – as Ian says in the video above, this can then be another input to your SIEM that can provide further evidence to allow your SOAR to take action when needed.

We have lots of ideas in this area, and more will be coming in subsequent software, and hardware updates over the coming months/years.