ORIGINALLY POSTED 13th April 2017

Hi everyone,

Its been a long time since I’ve posted here and I apologize for that.

Over the last few months, we have been seeing more and more deduping Controllers such as the A9000 being deployed behind SVC.  There are definitely good ways and bad ways to perform over-allocation and dedup behind SVC.  What I want to do is help you learn from the lessons that other customers have learned.   Also – for readability I’m going to write about SVC in front of A9000 in this post (because A9000 is much shorter than saying “SVC, or any product using the Spectrum Virtualize software in front of any deduping controller like the A9000”) but know that everything I am talking about here will be just as valid for any other deduping controller behind any Spectrum Virtualize product.

I’m mostly focusing on the new breed of controllers that do Thin provisioning, compression and de-duplication (many of them do all three whether you want them to or not).   If you have a controller that only does thin provisioning then I’d normally recommend only using an over-allocation ratio of 1:1, but you can apply the logic below if you choose to.  I have an earlier blog post which is focused on doing thin provisioning on a controller behind SVC.

There is way too much information for me to put on a single blog post here – so I’m going to break it into 3 parts.

• Part 1: Planning for new deployments  
• Part 2: What to do if you’ve already deployed your A9000 before you found these useful blog posts
• Part 3: What to do if your deduping controller is running out of space or you have recently removed a lot of SVC volumes from your SVC storage pool.

---

Part 1: Planning for new deployments  

Here is my high level advice for A9000 behind SVC.  I’m deliberately putting the shock factor into these bullet points to make you read further – but I will soften my position later

• Whatever over-allocation ratio you think you can achieve – halve it
• Encryption or Compression in the SVC are dangerous for deduping controllers
• Have a plan for what you are going to do if you start running out of space on the deduping controller

For the purposes of this explanation we will assume that I have a A9000 with 100TB of usable physical capacity behind an SVC.  I know you can’t buy one of those, but it’s easier to do maths nice round numbers like 100.

Whatever over-allocation ratio you think you can achieve – halve it

So you’ve been talking to a salesman and trying to work out what over-allocation ratio you can achieve in your environment.  The salesman has probably told you that the industry average is around 5:1 overallocation ratio when using Compression and De-duplication (This doesn’t include thin provisioning savings).

Hopefully you then spent some time trying to work out what over-allocation ratio you can expect.  Of course we all know that whenever you see an average – some people get much better and some people get much worse than the average.  (There was a QI episode recently that pointed out that there is literally no such thing as an average Australian – which I found quite amusing: https://www.comedy.co.uk/tv/qi/episodes/14/7/ )

So lets assume you have done a lot of work and decided that you believe that 4:1 over-allocation should probably work in your environment.

The strongest advice I can give you is Halve it – go for 2:1 initially.

You are probably asking why that is.  Here’s my thinking:

In a standard controller you won’t deploy all of the capacity on day 1.  You’ll be adding new volumes every day or every week and gradually increasing the controller over-allocation ratio until you reach the 4:1, at which point you will stop.  During that time you will have a lot of time to observe the over-allocation ratio you are really achieving on your data.   If you notice one month in that your data is only achieving 3:1 rather than 4:1 – hopefully you’ll stop allocating new LUNs when you get to 3:1 .

But when you put a storage controller behind SVC, you will create all the LUNs at once on day one.   So in our example configuration you will allocate 400TB to the SVC.   That’s all fine.

One month later, you notice that your actual over-allocation ratio is only coming out at 3:1…. Oh dear.  So your analysis wasn’t right.  It’s now going to be quite a difficult and time consuming job to reduce the over-allocation ratio inside the A9000 – especially if you can’t simply add more capacity to that A9000.    Part 3 will go through some of the steps I would recommend in this scenario, but for now just know that it will be a lot of work and there is always the chance that you may run out of space and go offline before you can complete the migration tasks.

So if, instead of starting with 4:1, we start with 2:1 and create 200TB of LUNs on the A9000 to give to SVC.   During the first month you create volumes in SVC and give them to hosts, and you keep an eye on the A9000 over-allocation ratio.   The data shows that you really are achieving 4:1 over-allocation on your data.   That’s great news.  It’s now trivial to simply create another set of LUNs totaling 100TB of capacity and give them to SVC.  This is a really easy operation and won’t hurt at all and you’ve now increased to 3:1 over-allocation.   You can do the same thing in a months time to go to 4:1 if you need to and you think it’s safe.

I would always recommend staying a bit below the current ratio for safety – i.e. if the A9000 is reporting that you are achieving 3.3:1 over-allocation I would advise caution and recommend that you only over-allocate the A9000 by 3:1 – in my opinion it’s much better to be safe than sorry.

One other note – the A9000 recommends that you stay below 80% physical usage.  That’s there for obvious reasons (I hope).  So if you want to ensure that your A9000 never breaches the 80% used capacity then the over-allocation ratio should really be applied to 80% of the physical capacity rather than the total physical capacity.    So in this case with a 3:1 overallocation ratio, I would present 100 TB * 80 % * 3 = 240TB of LUNs to the SVC.

Encryption or Compression in the SVC are dangerous for deduping controllers

This one got your attention – didn’t it

Encrypted data is, by design un-compressible and not possible to dedup.  You might achieve 1.1:1 or perhaps as good as 1.5:1, but basically you shouldn’t expect to get any savings at all.

So what does this mean?  

Compression

You will have no control over how much of the data you are storing on your disks is already compressed (e.g. db2 compression, or Microsoft Office compressed files).  You will hopefully have thought about this when trying to predict your over-allocation ratio.  But if you create compressed volumes on the SVC and store them in controllers like the A9000, all of your data will be compressed.

The advice is either to configure the A9000 with 1:1 over-allocation when using compression in SVC, or simply don’t use compression in SVC storage pools which are storing data on an A9000

Encryption

You probably won’t be surprised to see that the advice is the same as for compression – don’t encrypt at the SVC layer, and remember to factor in application encryption into your expected over-allocation ratio.

However there’s a caveat here.   If you have purchased and installed the encryption license key on the SVC cluster, then by default all mdisks will be encrypted by SVC unless the SVC detects that they are already encrypting themselves.   This detection should work appropriately for all IBM controllers on the right code levels, but probably won’t work reliably for non-IBM controllers.

So the simplest solution is do not install the SVC encryption license.   This works well if you don’t want to encrypt data, or if all of your back-end controllers are self-encrypting.  

But what about if you have already enabled the encryption license on SVC?  Then the thing to do is to be really careful when adding the A9000 mdisk to the SVC. 

• If this is a new pool, then when you create the pool specify “-encrypt no”.  This tells SVC not to encrypt this pool at all.  You can ONLY do this when you create a pool.  After you’ve created it you can’t tell SVC not to encrypt. 
  • Note:  If all of the managed disks in a pool are self-encrypting then the pool will report encrypt=yes even if SVC is not configured to encrypt the mdisks.
• If you are adding the mdisks to an existing pool, and you have the encryption license installed and the encrypt property of the storage pool is yes.
  • If the unmanaged mdisk reports “encrypt = yes” then that means that SVC has detected that the back-end controller is self encrypting.  The SVC will not encrypt any data written to the managed disk.  This is safe.
    • Note that at this time SVC can only detect self encrypting managed disks on a subset of IBM controllers.
  • If the unmanaged mdisk reports “encrypt = no” then that means that SVC has detected that the back-end controller is not encrypting.
    • Running “chmdisk -encrypt yes <mdisk>” on the mdisk while it is still unmanaged tells SVC that the mdisk is self encrypting.  This is the only way to be certain that SVC will not encrypt data on that managed disk.  So I’d recommend in this case doing this before adding it to the pool. 
    • However this will mean that any future checking you or your colleagues do about what data is encrypted will be incorrect.
  • If you don’t have any unmanaged disks then create a small additional volume and map it to SVC.  This will determine whether the SVC is able to detect that this is a self-encrypting managed disk

Have a plan for what you are going to do if you start running out of space on the Deduping Controller

 This is the harder part.  What are you going to do if you ever reach 95% physical usage on your deduping controller.

You should realize that if the controller ever reaches 100% full, the SVC storage pool is almost certainly going to go offline and the only way to get the SVC storage pool back online is to get the back-end controller back below 100%

There are two types of approaches to this: 

1. Have a plan that allows you to add additional capacity to the controller
2. Reserve a set of space in the controller that makes it *seem* fuller than it really is, and that you can free up in an emergency situation

Approach 1 is fairly easy to understand.  Basically just have an emergency plan that means you can add additional physical capacity to the controller.  The specifics of this will be controller specific, but hopefully not too complex.  

Unfortunately not all controllers allow dynamic expansion, and of course all controllers will reach a point when they can no longer be expanded – so this may not be an option for your environment.

Approach 2 is harder to implement.

A/ Some controllers will have the ability to create a volume which isn’t compressed, deduped or thin provisioned.  If you can do this then simply create some of these volumes to reserve an amount of physical space, you probably want to name them something like emergency buffer space..  Then if you are reaching the limits for physical capacity you can simply delete one or more of these volumes to give yourself a temporary reprieve.

B/ If this not the case, then you need to create compressed and deduped volume and fill it with data that cannot be deduped or compressed to reserve the physical space.  

The simplest way of doing this is to create a volume on the back-end controller, present it to a host temporarily, and fill it with random data using something like dd or VDBench.   If you are using VDBench make sure you configure it to write data with a compression/dedup ratio of 1 .  

EasyTier Advice (added Sep 2017)

If the overallocating controller is a tier in a multi-tier or multi-controller storage pool then you should leave it on.  The benefits will far outweigh any other concerns.

However if you are creating a storage pool that only has a single overallocated controller in it, then EasyTier isn’t going to add much value.  EasyTier may also casue a small amount of wasted space in the overallocated controller due to moving the data around for balancing within the pool.  So for this configuration it makes sense to disable it for this storage pool.

---

Part 2: What to do if you’ve already deployed your A9000 before you found these useful blog posts  

Phase 1: How much do you need to be worried

So if you are reading this section, I’m going to assume that you deployed some A9000 or equivalent capacity to SVC and you’re now worrying about whether you allocated too much.

The advice is to simply look at your overallocation ratio from the backend controller ratio.  If you configured 5:1 from the controller to SVC and you are achieving approximately 5:1 on the data that you’ve written so far then there’s no need to panic.   Just keep an eye on it occasionally.

However if you configured 5:1 on the controller and you’re achieving about 4:1 or lower then it’s probably time to start making plans…

Phase 2:  My controller is nowhere near full yet, but I want to reduce my overallocation ratio

If you’re following best practices you should have allocated many mdisks from your controller to SVC.   General guidelines for an all flash controller is around 16 managed disks (or more) from one all flash system so that you can make use of parallel processing in the SVC layer.  

The best/simplest/fastest way to achieve this is to simply remove a few of the managed disks from the SVC storage pool.    The number of managed disks you remove will control  the new overallocation ratio.  

Example:

• you have a 64TB (physical) controller
• you are 5:1 overallocated
• you are presenting 16LUNs to SVC

If you want remove 1 managed disk – your new overallocation ratio will be approximately 4.7:1

After messing around in excel and going back to my secondary school maths for a while the formula for your new overallocation ratio after removing managed disks will be

NOR = OOR * ( NNL / ONL)

Where:

• NOR = New Overallocation Ratio
• OOR = Original Overallocation Ratio
• NNL = New Number of LUNs
• ONL = Original Number of LUNs

So for this example

5 * (15 / 16 )

= 4.6875  (or approximately 4.7)

I’ll admit that it took me a long time to reverse into that really simple and (retrospectively obvious) formula….

Assuming the storage pool isn’t full yet, you can follow this procedure:

1. Remove one of the managed disks from the storage pool using either the GUI or “rmmdisk -force”.  
   
   Note: This can time a long time to complete because it will copy all of the real_capacity from that managed disk onto free space in other managed disks within the same storage pool.
    
2. The overallocation ratio on the controller you are trying to manage should not increase significantly, because the data will either be migrated to other controllers or it will be migrated to another LUN on the same controller, but will dedup back to the source data.
    
3. Wait for the managed disk migration to complete, and the mdisk to show as “unmanaged”.
    
4. Now you can use the controller management interface to unmap the LUN, so that it’s no longer visible to SVC
   
   Note:  make sure you get the right LUN.   My advice is to only unmap it initially, just in case you make a mistake.   If you start by deleting the volume and you get the wrong one then you will have a very bad day.
    
5. Once the LUN has been unmapped and you have confirmed that the SVC is still happy, delete the LUN from the controller configuration.  

---

Part 3: What to do if your deduping controller is running out of space or you have recently removed a lot of SVC volumes from your SVC storage pool.

Part 3A: Emergency procedures – SVC storage pool with a single controller

So the worst has happened and you’ve just noticed that your overallocating controller is now reporting >90% full and you are really worried that you are about to run out of space.  What do you do?

Critcal:  This procedure only works if the overallocated controller is still accepting new writes.  As soon as the controller is completely full and not accepting new writes this will not work

If you are really really close to running out of space I strongly advise you to shut down all hosts, or at least all but the most essential hosts. Any hosts which are online and writing data to the storage pool risk causing you to run out of space on the controller, and if that happens it is very hard to recover.

Before Starting – note down your current overallocation ratio from the overallocated controller.  This action plan is going to write a large amount of zeros to the controller, and some controllers may count this in their overallocation savings.   So it makes sense to write down the value before you start, just in case writing the zeros makes the controller report an artificially high value.  

1. The first thing to do is to immediately create enough fully allocated volumes to fill your SVC storage pool to somewhere between 90 and 95% full, and let the volume formatting code fill this space with zeros.   Remember to label them clearly as volumes that should be deleted later.  I recommend using a number of small to medium sized volumes rather than one large one.
   • Important:  I don’t advise filling the pool up to 100%, because if you do that you may suffer from thin provisioned volumes going offline if the pool reaches 100%.  
   • Remember to monitor this to ensure that the pool never reaches 100%.  If it does simply delete some of the volumes created earlier to make more space available.
   • Do not create mirrored volumes
   • Set the volume sync rate to 100.  This is in the advanced panels somewhere, but if you can’t find it you can change it later with the chvidsk -syncrate command
      
2. This will overwrite any old copies of data that are “lying around” (after either an EasyTier migration or a volume deletion) with zeros.  And since these zeros will all dedup to each other and compress to almost nothing, the backend controller is almost certain to be able to reclaim space.
   
   Important: Most dedping controllers will not be able to immediately reclaim space as soon as you write the zeros to the disk.  The controllers will need to perform some form of backfground space reclamation process (often called garbage collection) to actually reclaim the space used by the data you just overwrote.   This means your controller free capacity might not go down until the day after you write the zeros (or perhaps even longer).    If you are really concerned then you should contact the controller support team and find out whether they have any method of knowing how much space is waiting to be reclaimed by the garbage collection process and let you know if you are making good progress or not.
    
3. Depending on the number of EasyTier migrations and other changes you’ve made to the pool in the past (such as deleting volumes), this may be all you need to do.   So you can either pause here or move straight into the next steps which are a bit more aggressive.
    
4. Free up additional space in your storage pool.   There are a number of ways you could do this, including:
   • Migrate some of your data to other storage pools within the SVC
   • Delete some volumes that are less critical to your business.  For example, you might have some FlashCopy backups of your production data that you can delete.
   • If you would rather add more space to this pool – see part 3B instead.
      
5. After every activity which frees up space in the pool, go back to step 1 and create more fully allocated volumes.
    
6. Once you are back to a fullness level that you are comfortable with, you should probably reduce your overallocation ratio from the controller.  Start by deleting the volumes you created in step 1. Once that is done, follow the instruction in “Part 2: What to do if you’ve already deployed your A9000 before you found these useful blog posts” above.

Part 3B: Emergency procedures – SVC storage pool with multiple tiers or multiple controllers

So this gets a bit more complicated….  The procedures here will need to be run on the command line instead of the GUI.  

Critcal:  This procedure only works if the overallocated controller is still accepting new writes.  As soon as the controller is completely full and not accepting new writes this will not work

Rather than copying and pasting all of the steps from above I will list the differences:

• In step 1, when you are creating the volume with the mkvdisk command,
  • use the “-mdisk” flag and provide a list of all of the mdisks from the controller which is running out of space
  • specify  “-eaytier off” to make sure that these fully allocated volumes filled with zeros don’t get moved to any other storage controllers.
  • Example:    mkvdisk -mdiskgrp mymdiskgroup -size 1 -unit tb -iogrp 0 -mdisk mdisk1:mdisk2:mdisk3 -syncrate 100 -easytier off
• In step 1 you want to create the fully allocated volumes until the managed disks are nearly full rather than trying to fill up the storage pool.  Since you will have other managed disks in the pool which have free space, you can actually go to 100% on an mdisk if you want to.  You can check this with the command
  • lsfreeextents <mdisk id/name>
  • The output shows the number of free extents in the managed disk.  If you multiply that value by the extent size of the storage pool you will get the free capacity of the managed disk.  Dividing this free capacity by the managed disk capacity will give you a percentage value.
• In step 4, you can choose the following additional options:
  • Add more capacity in the same tier as the overloaded controller, and let EasyTier balancing move a percentage of the data to the new managed disk
  • If you have enough space in the other controllers or tiers then you can delete one of the managed disks from the storage pool – just like in part 2 above.   However this can take a long time to complete and you wont reclaim any space until the procedure is entirely completed.
    • If you do this make sure that you don’t put any of the volumes created in step 1 onto this mdisk, otherwise you will spend a lot of time moving data you don’t care about to other controllers
  • For advanced users you can disable easytier on some volumes and use the migrateexts command to move data to other mdisks within the pool.

Part 3C:  Emergency procedures when the storage pool is offline due to the controller having reached 100% (or higher)

I’m reliably informed by a customer I met recently that some of the overallocating controllers are actually able to reach more than 100% full.   Therefore if everything is online at the SVC level when the controller is reporting 100% full then you can still try part 3A or 3B.  However once the storage pool goes offline those procedures will not work.

At this point your activities will depend heavily upon the specific controller at the back end, and I’m afraid I can’t give you specific advice.  You may be able to add more physical capacity to get you back up and running again, or there may be some controller specific tricks that they can use to get you back online enough to run

Part 3D:  You have just deleted  (or are just about to) a set of volumes from the storage pool and you’d like to clean up their data that they had stored

If you haven’t deleted them yet, then the simplest thing to do is to overwrite them all with zeros from a host.   This way you only overwrite their data without having to worry about filling up the entire storage pool.

Otherwise delete away and follow the steps in Part 3A  (without any of the more drastic action in step 4).