Google Cloud Storage Autoclass feature

I found out about this new GCS feature the other day from a customer engineer at Google and it is an awesome money saver!

GCS Autoclass is a feature in Cloud Storage that, as the name suggests, automatically sets the storage class of an object down the tiers to the coldest tier you want to reduce the costs of storing objects.

How it works

When you enable autoclass and set the coldest storage tier on the bucket, GCS manages the classes as follows:

1. All objects start in the Standard storage class.

   1. The bucket itself will always have a default storage class set to Standard storage and any requests to change this will fail.

   2. Any attempt to manually change the storage class of an object during a rewrite or copy operation will succeed, however the storage class change will be ignored and the object will be set to Standard storage.

2. Objects in the bucket will transition to colder storage if not accessed.

   1. By default, Nearline is the terminal storage class unless you set it to a different class on the bucket.

   2. Objects smaller than 128KB will never transition and are permanently stored in Standard storage. They are also exempt from the Autoclass management fee. Object metadata is not considered when determining the object size.

3. When you read an object, it is automatically transitioned to Standard storage from whatever class it was residing in.

   1. Reading or editing metadata does not affect the storage class of the object.

Autoclass Behaviour

When you create a bucket with Autoclass enabled, objects of at least 128KB in size transition as follows:

• When an object is accessed, it is transitioned to Standard storage.

• If an object if not accessed in 30 days, it is transitioned to Nearline storage.

If the bucket is configured with Nearline storage as the terminal class, Autoclass will only change the state of an object stored in Nearline storage if that object is accessed.

If the bucket is configured to use Archive storage as the terminal class, objects will continue to be transitioned to colder storage classes as follows:

• Any object that hasn’t been accessed for 90 days, it is transitioned to Coldline storage. These objects will spend at least 30 days in Standard storage and 60 days in Nearline storage.

• Any object that hasn’t been accessed for 365 days will be transitioned to Archive storage. These objects will spend at least 30 days in Standard storage, 60 days in Nearline storages and 275 days in Coldline storage.

• Any object that is accessed in any storage class will be transitioned to Standard storage and the process will start again.

Objects that are eligible for transition between storage classes will be transitioned asynchronously, so there can be a lag between when an object is eligible for transition and when it actually appears in that storage class. During this lag period, objects are charged at the ‘pre-transition’ rate.

When to use Autoclass

Autoclass is designed to reduce the management overhead of having multiple buckets at different storage tiers to reduce storage costs in GCP.

You should use Autoclass when your access patterns fit into one or both of the below:

1. Your data has a variety of access frequencies.

2. Your data access patterns are unknown or unpredictable.

You should NOT use Autoclass when the majority of your buckets data fits into a specific storage class access pattern or if you are using Sensitive Data Protection to scan the content of your bucket.

Audit Logs in Cloud Monitoring

Google has made the below metrics available in Monitoring to track storage class transitions:

• autoclass/transition_operation_count - The number of storage class transitions initiated by Autoclass.

• autoclass/transitioned_bytes_count - The total number of bytes transitioned by Autoclass.

Both metrics can be grouped by source or destination storage classes involved in transitions.

Objects in Standard storage are reported as REGIONAL if they are stored in a region and as MULTI-REGIONAL otherwise.

Limitations

1. As of the time of this writing, Autoclass must be enabled at bucket creation time. You may only disable Autoclass on existing buckets.

2. Currently, you can only use Autoclass with Nearline storage as the terminal storage class. A rollout on 2nd November 2023 will allow you to change the terminal storage class to Archive.

3. You cannot enable either of the below Object Lifecycle Management configurations when using Autoclass:

   1. A rule using the SetStorageClass action.

   2. A rule using the matchesStorageClass action.

   Requests that would result in a bucket to have both Autoclass and either of the above Object Lifecycle Management configuration rules will fail.

4. Object composition requires the source and composed objects to all use the same storage class, so composing fails unless all source objects are storage as Standard storage at the time of the composition request.

Pricing

Pricing on Autoclass enabled buckets is the same for normal buckets with the following exceptions:

1. Retrieval fees are never charged.

2. Early deletion fees are never charged.

3. All operations are charged at the Standard storage rate.

4. There is operation charge when Autoclass transitions an object to a colder storage class.

5. There is no Class A operation change when Autoclass transitions an object from Nearline to Standard storage class.

6. There IS a Class A operation charge when Autoclass transitions an object from Coldline or Archive storage to Standard storage class.

7. An Autoclass Management Fee applies. (At the time of writing, this is $0.0025 per 1000 eligible objects stored per month. This fee is prorated to the ms.)

Setup

Configuring Autoclass is only possible on new buckets at this stage. v2 of Autoclass will enable the ability to configure existing buckets with the feature.

When creating a bucket in the console, you will be presented with an option to select the default storage class or use Autoclass as shown in the image.

That’s it! You now have Autoclass configured on your new bucket.