Storagezilla's Swap File

S3 storage classes. Move that data.

S3 Intelligent Tiering, Storage Class Analysis and Lifecycle Management. When do you use them?

The difference between Intelligent Tiering and Storage Class Analysis with Lifecycle Management is that with Intelligent Tiering you expect that infrequently accessed data may need to be promoted back to S3 Standard from S3 Infrequent Access. In contrast, with Storage Class Analysis and Lifecycle Management you expect any data you will be moving will not need to be promoted back from whence it came. Lets look at both. 

S3 Intelligent Tiering is an S3 storage class where the access pattern of objects is monitored. You pay a monitoring fee for this but ongoing analysis of that monitoring results in data objects being moved between a standard access tier of S3 and a lower cost infrequently accessed tier of S3. The cost savings of moving inactive data to the infrequent access storage class can be substantial, depending on how much inactive data you have.

There are some things to keep in mind. There is a minimum object size requirement, objects less than 128KB will not be moved, also the benefit comes from data which is not being accessed all the time. If a lot of your data is read frequently, like the objects contain deduplicated segments that could be required many times, S3 Intelligent Tiering might bring you no value. Keep that type of data in S3 Standard.

What if you have data in regular S3 Standard and want to find out if you could save some money by moving it yourself to S3 Infrequent Access? Turn on S3 Storage Class Analysis and run it for 30 days. It will watch the access patterns of the objects in S3 Standard and after 30 days it will display how much data has been accessed in relation to the data you have analysed. With those number in hand you can then turn to Lifecycle Management transition and expiry rules to automate the move of the infrequently accessed objects into S3 Infrequent Access; or to Glacier. Or, use Lifecycle Management to delete data you no longer need.

You may have to do some work to ensure you can target the correct objects when you create the transition or expiry rules. If you’re not working with creation dates, S3 object tags will matter. Lifecycle Management allows you to automate moves down the S3 storage classes and into the coldest tier of storage, Glacier Deep Archive. There are transition charges accrued when you use lifecycle rules so take note of the cost.

Regardless of if you want the storage system to automatically move your data between storage classes, or you want to do it yourself, you can do it with S3.

S3 public access. You might forget but it doesn’t.

While there are many reasons someone might open up an S3 bucket to the world there is no reason to forget that it is open and then read about it on the web later. AWS Trusted Advisor, AWS Config with custom rules and S3 pre-signed URLs are more than capable of reducing, eliminating, or preventing the creation of publicly accessible buckets.

 

AWS Trusted Advisor has a (free) check that identifies open access permissions or open bucket policies applied to S3 buckets. It costs you nothing but the time it takes to run the check. 

 

So, run the check. 

 

There is another option if you’re the type that likes to immediately act on the results of a check and are willing to do some bespoke work. AWS Config, CloudWatch and Lambda can be used together to create custom rules and then act on any exceptions to those rules. 

 

In brief;

you create the set of custom rules for AWS Config;

AWS Config will check all your buckets and notify CloudWatch if anything is out of policy;

CloudWatch can trigger Lambda when exceptions are found;

Lambda will then apply your security policy to the non-compliant objects and buckets.

 

AWS Config will automate the detection and enforcement so you don’t have to. Want it to mail the bucket creator asking why they have a bucket open? You could configure it to do that too. 

 

A note of caution in handing everything over to automation. If you are overzealous you may end up closing something you shouldn’t, so evaluate what your rules will change before you deliver the Admin’s justice to any renegade bucket. 

 

Finally, when removing the Block Public Access default settings on an S3 bucket ask yourself if using an S3 pre-signed URL would do the job instead? An authorised user can create a time limited URL that will grant object access to whomever you want.

 

No one should lose their job because someone opened a bucket up to the public two years ago and then left the company to go work somewhere else. The system remembers everything, just ask it to tell you.

 

Frequently.

Amazon FSx for Windows File Server is pretty neat.

Mucking around with Amazon FSx for Windows File Server this morning I deployed a multi-availability zone Windows file server, a managed Active Directory and a bunch of Windows clients in less than an hour. The FSx network attached storage was enabled for dedupe, automatic backup and had synchronous replication for high availability built in.

It made me wonder “What the hell do people need me for?” then I remembered that this stuff isn’t natural for non-infrastructure types.

Just like many people live better today than the average billionaire did 100 years ago, many IT infrastructure people work better today than their peers with near unlimited budgets did just ten years ago. No one can buy time, but you can develop services that will save it for everyone.

Developing new storage systems is a lot like developing new smart phones. The hard problems have been solved, in this case writing data and reading it back successfully, but you need new features to drive interest. Just like the inevitability of death and taxes you should expect new features for FSx for Windows File Server in good time.

Protecting SAP HANA with Veeam Backup & Replication

SAP HANA is the in memory source of truth for SAP applications, such a critical component requires regular backup to ensure business operations can be resumed after a failure or corruption. The Veeam plug-in for SAP HANA allows HANA database administrators to use Veeam managed infrastructure as a destination for their backups, while giving Veeam administrators visibility into HANA backup and recovery operations. 

Enterprise databases can come with their own built in backup and recovery tools but because they do so does not mean that those backups should live outside of the rest of the data protection infrastructure. A good data protection infrastructure is consistent, reusable and available. If the effort has been spent to build and maintain an infrastructure designed for high throughput data transfer, storage capacity optimisation and lifecycle data management it is in the interest of everyone to reuse this infrastructure whenever possible.

When using Veeam infrastructure to store backups the lifecycle of all HANA backups is managed by database administrators alone. Only they can initiate backups, start restores and clean up expired backup data.

There are three main types of HANA backup, full/delta/incremental backups, snapshot backups and log backups. A full backup contains all the data required to recover a HANA database, or, since HANA allows for multiple logically isolated database containers to share the same HANA deployment, a full backup might contain multiple database tenants.

A delta backup is all the data that has changed since the last full backup, incremental backups capture the changed data from any previous backup, not just the last full backup. A HANA database recovery may use a combination of a full backup, a delta backup and multiple incremental backups to restore the selected database(s) to the desired point in time.

With a snapshot based backups you can synchronise the creation of an internal HANA snapshot with an external storage snapshot, such as a vSphere snapshot or a snapshot on a storage array. This is a full backup that can be recovered from and while fine for DR purposes it should be noted that snapshot based recovery is not supported with multi-tenant HANA deployments.

Log backups utilise redo logs to recover a database to a point in time. As the logging window can be short log backups alone are not enough to recover a database and require a full backup along with any delta or incremental backups to support the recovery operation. By default log backups run every 15 minutes but will not begin until the first full backup is taken from a database server.

When installed on the database server, or members of a database cluster, the Veeam plug-in for SAP HANA contains Veeam transport services which allows HANA data to be piped out through the SAP Backint API an into Veeam managed infrastructure. When data is sent to the plug-in it is compressed, deduplicated and then transmitted to a Veeam backup repository for retention.

A backup repository in Veeam is a storage location where Veeam stores and manages backup data. From the point of view of a HANA administrator the Veeam backup repository is just another storage target.

Using tools such as SAP Cockpit or SQL scripts backups can be scheduled and restores initiated. During such operations a non-editable job appears in Veeam user interfaces providing awareness and logging, but not control, over the backup process. The Veeam administrator can see backup jobs occurring but the control point always remains with the HANA database administrator.

On the matter of control, Backup Copy Jobs carried out by the Veeam Server are undesirable as they would result in databases backups not managed by a database administrator. Copy jobs have therefore been disabled. Object level recovery is not available for HANA so there is no Veeam Explorer to facilitate such a restore operation. Any restore operation is driven through database administrator tools and requires that the database(s) be shut down, and while HANA has built in replication technology backups can only be taken from the active database, not the replicas.

Data is like gas, it expands to fill available storage and when presented with useable general purpose storage database administrators tend fill it with backups that no one else has visibility into. The Veeam plug-in for SAP HANA keeps the control in the hands of the HANA database administrators but provides assurance to the backup administrators that backups are happening and they are being written to centralised infrastructure designed to support the backup and recovery of mission critical systems.