How to: Resolving Hive's Alerts

This article aims to thoroughly explain the alerts found in Hive, as well as provide next steps for Engineers to resolve them

1. StorageCraft Backups
2. Veeam Hot Storage Backups
3. Veeam Cold Storage Backups
4. Microsoft 365 Cloud-to-Cloud Backups
5. Dropbox Backups
6. Hive's internal alerts 
   
   

STORAGECRAFT BACKUP ALERTS

Alerts for StorageCraft are based around two main factors:

a) If files are detected in the Honeycomb Hot Storage tub, according to the alerting period"

• Volume Run - Designates that an image for a given ShadowProtect volume hasn't been detected in the cloud tub:
  
  This is generally caused when there is an issue the local side StorageCraft ShadowProtect console, or StorageCraft ImageManager's replication. We recommend:
  • Check the local storage for up-to-date ShadowProtect images, and verify ShadowProtect is running without issue
  • Access the ImageManager server to ensure replication is running through without issue for the managed folder containing the volume as per the above alert
  • Update to the latest version of ImageManager
  • Restart the StorageCraft ImageManager service, loading up ImageManager to check for a replicating image
  • Delete the replication job in ImageManager, and recreate it, ensuring you keep all of the same settings
  • Contact Probax Support for additional assistance, including in your email TeamViewer details to the ImageManager server, as well as windows authenticated to the local backup storage device. Please also mention the steps you taken and any significant results

b) If the backup files are healthy and chain back to the base image.  

• SP Chain Check Failure - This alert refers to cases where your ShadowProtect chain's images don't successfully chain back to the base image. This is normally due to errors during ImageManager's replication process, causing blocks of data to disallow chaining to the base image. We recommend:
  • Ensure Probax Scout is installed and running correctly - This will automatically resolve chain issues through uploading files to the cloud. 

• • In the event Probax Scout cannot be installed, take a directory listing of the local backup environment, and email Probax Support for assistance. 
• SP Time Since Chain Check Success - These alerts generally coincide with the above 'Volume Run' alerts, to which are advising you our internal chain checks haven't gone through successfully for a period of time.
  • Please refer to the 'Volume Run' alert for more information
• SP CRC Verification Check - This alert indicates that a single file within your backup set has a CRC verification failure, indicating that it is not useable in the backup chain. 
  • Please refer to the 'SP Chain Check Failure' alert for steps to resolve

VEEAM HOT STORAGE ALERTS

Hive is responsible for alerting you on the overall status of your backups. Hive's internals will take stock of any new images that find their way into Veeam Cloud Connect environment storage tub, and advise you based on:

a) If files are detected in the Honeycomb Hot Storage tub, according to the alerting period

• VCC-B System Backup Last Run - Designates that Hive hasn't detected a backup run (or a backup image) in the alerting period (107h), in our Veeam cloud environment.
  
  Without access to the local Veeam software, the amount of information the above alerts can give is limited. These Hive alerts are used to notify you that there may be something preventing backups from running through. To resolve these: 
  
  • Access the server processing backups for the alerting resource, and check the job running backups to the cloud. For our Veeam Backup and Replication users, the 'History' section of Veeam provides a good spot for reviewing the backup runs that have failed to this point in time. 
    

• • In some cases, the issue may be unrelated to an issue with the backup job itself, and instead be of basic non-software/cloud origin. These issues can arise if: 
    • The backup server is offline or unreachable - this results in Veeam not being able to process the backup.
    • The VM/machine being backed-up is offline or unreachable - this results in Veeam not being able to process the backup
    • The local backups being copied to the cloud are unreachable/haven't been created - the local storage/NAS device issue will prevent Veeam being able to process a backup
  • In the event the alert doesn't relate to the above, take note of the error message being received, as well as screenshots of any relevant findings on the local-side Veeam console. Please contact Probax Support with this information, and remote access to the site if possible. 

b) If the backup files are healthy and chain back to the base image.  

• VCC-B Last System Verification - This alert refers to cases where a backup image is detected in the cloud tub that has been flagged as potentially harmful or disruptive to your backup chain. These alerts normally occur in conjunction with the 'VCC-B System backup last run' alerts, and can help identify the presence of an incomplete file in the backup storage space. 

• • Please refer to the ''VCC-B System backup last run' alert for steps to resolve

VEEAM COLD STORAGE ALERTS

As with your Hot Storage alerts, Hive is responsible for alerting you on the overall status of your backups. When you have archival backups selected and have configured Hive Honeycomb Cold Storage, Hive's internals will alert you if they're not able to see any new GFS images in the desired alerting time.

• VCC-A Honeycomb Last Run - Designates that Hive hasn't detected an archival backup image (GFS image) in the alerting period (107h), in our Veeam Honeycomb Cold Storage environment. 
  
  Generally, these alerts are an indication that the configuration of archiving hasn't been set up correctly, as Probax handles Cold Storage retention though Hive. To resolve these: 
  
  • Please review our Knowledge Base article on configuring Honeycomb Cold storage, to ensure Hive has been correctly enabled to process archiving. If any changes were made, please allow up to 24 hours for the internal archival process to take full effect. 
  • In the event the alert doesn't relate to the above, please contact Probax Support with screenshots of your archival settings in Veeam and Hive, as per the KB above. 

VEEAM BACKUP FOR M365 ALERTS

Given the nature of o365 backups being cloud-to-cloud, the information gathered and presented through alerts is based solely on what Veeam can retrieve from Microsoft, in relation to the organisation being backed up. O365 alerts arise when an item selected for backup cannot be backed up by Veeam, regardless of the type of item.

• OneDrive/SharePoint/Mailbox Not Found - Designates that Veeam was unable to backup a OneDrive/SharePoint/Mailbox item in the last few backup runs. The time of the alert indicates how long this alert has been active (in this case, 202 hours).
  
  There are a few reasons why these alerts may arise, though in majority of circumstances, we see these alerts when items have been turned 'on' for backup by mistake.  
  • Firstly, check the alert type for the user/SharePoint and determine if this is an item that is licensed. You can review this in your organisation configuration by checking the license type:
    
  • Next, check if the alerting item is still active in your M365 organisation - For many organisations, items that become dormant or unused will alert as Veeam's back-end will detect there's nothing new to backup. In these cases, consider turning the item off for backup
  • Confirm the item has an M365 license, a items without M365 licenses can cause issues when attempting to be backed up by Veeam.
  • In the event the alert doesn't relate to the above, please contact Probax Support
• Authentication Status - This alert is designed to advise you that Hive is having some difficulty authenticating to your organisation. Backups will not recommence until this is resolved. 
  
  • For most customers, this should advise you to get in touch with support, and switch over to App-Only authentication
  • Customers adamant on staying with legacy authentication will need to re-enter their credentials into Hive, and confirm their app-password and secret password have not expired.
• O365 Roles and Permissions Status - These alerts typically appear when an M365 organization has been onboarded, but doesn't have the necessary permissions applied to it to allow Hive to correctly determine user/license types.
  
  
  • Simply ensure you have followed steps 2, 3 and 4 of KnowledgeBase article here:
    https://kb.probax.io/m365setup
  • Please contact support if you would like assistance with these steps, or they haven't alleviated the issue after 24 hours

DROPBOX Backup alerts

Probax's proprietary backup for Dropbox only has the one alert that will appear, and generally indicates that Dropbox admin user assigned to the backups. 

• Dropbox Link Status - Designates that Hive requires you to perform a re-link of your backup environment. Instructions can be found in this KB
  
• Dropbox Time Since Backup - Designates that Hive hasn't seen a successful backup in a given amount of time. These generally occur when there's an issue with Hive connecting through to the Dropbox API. 
  
  • Please check the Probax app within your Dropbox account to make sure the status is OK, and the backup admin user is still active. This user can be found by accessing your Dropbox resource in Hive 
    
    
    
    
  • The first place to check is your backup restore-point calendar, for images that have been pushed through in the last day or so. If you are able to see images from the last few days, the alert may be referring to a previous alert entry, and thereby erroneous. In the event the alert doesn't relate to the above, please contact Probax Support with screenshots of your archival settings in Veeam and Hive, as per the KB above.

HIVE SPECIFIC ALERTS

Hive has inbuilt alerting to assist you in keeping track of your backups and Hive accounts.

• Honeycomb Hot/Cold AU (End User) - This alert is letting you know that the usage for this customer has gone above your pre-set quota. For non-internal customer accounts. this won't change any backups running through, but just acts as a notification that your customer's backups have exceeded your allocated amount
  
  
  • Start by accessing the customer's account who's alert you are seeing this for
  • Find the customer's 'billing' tab, and the pencil icon next to the billable the quota is exceeded for:
    
  • Select a quota size greater than your customer's current usage
    
    
    

For technical support questions, please contact support@probax.io or 1300 776 229.

For sales and product information, contact your Partner Manager at sales@probax.io or 1300 776 229.