Cloud Storage and file storage

Block, file, or object

Cloud Storage FUSE provides some file system semantics on top of buckets, but it isn't fully POSIX: it lacks hard links, file locking, and write concurrency control like Filestore has.

Object storage with unlimited capacity and access, no minimum object size, low-latency access (no offline retrieval), and 99.999999999% (eleven nines) annual durability in every class.

• Objects are stored in buckets, whose names are globally unique.
• The storage class is metadata on each object. The bucket has a default class (Standard if you don't set one) that new objects inherit.
• Changing the bucket's default class doesn't affect existing objects. To change an object's class you rewrite it or use lifecycle management.

gcloud storage buckets create gs://my-bucket --location=us-central1 --default-storage-class=STANDARD

Storage classes

There are legacy classes that can't be selected in the console; Google recommends using Standard instead.

Autoclass

It automatically moves each object to the right class based on how it is accessed:

• All objects start in Standard, and the bucket's default class is fixed to Standard.
• Objects that aren't read move progressively to colder classes; by default they stop at Nearline, and you can configure Archive as the terminal class.
• Reading an object's data moves it back to Standard. Reading or editing only metadata doesn't.
• Objects smaller than 128 KiB always stay in Standard.
• It has a management fee and an enablement charge, but no retrieval or early deletion fees (except as part of enablement).

Bucket locations

• In regional, dual-region, and multi-region locations, failover and failback are automatic (RTO 0) without changing access paths.
• The location is set when you create the bucket. To change it, copy the data to another bucket or use bucket relocation, which requires Storage Intelligence.

Fully managed file servers for Compute Engine VMs, GKE clusters, Cloud Run, VMware Engine, or on-premises machines (over Cloud VPN or Interconnect). Several clients read and write the same file system at once, for example GKE PersistentVolumes mounted read-write by many nodes. Capacity scales without downtime.

Protocols: NFSv3 on every tier and NFSv4.1 (with authentication and in-transit encryption) on zonal, regional, and enterprise. With custom performance, IOPS are configured independently of capacity.

Google Cloud NetApp Volumes

Fully managed storage with NetApp data management features, to bring file and block applications to Google Cloud without re-architecting them.

• Multiprotocol: NFSv3, NFSv4.1, and NFSv4.2, SMB, iSCSI, NVMe/TCP, and S3 (ONTAP mode).
• Volumes from 1 GiB to 3 PiB provisioned in seconds.
• Automated snapshots, backups, auto-tiering to cold storage, asynchronous volume replication, rapid cloning, and FlexCache.
• Service levels such as Flex Unified (independent capacity and performance; Default and ONTAP modes) and Standard, among others.

Google Cloud Managed Lustre

A high-performance, fully managed parallel file system for AI and HPC, developed with DDN.

It mounts from Compute Engine and GKE, transfers data to and from Cloud Storage in a managed way, and encrypts data at rest and in transit.

# Upload a file
gcloud storage cp ./report.csv gs://my-bucket/
# Sync a directory
gcloud storage rsync --recursive ./local-dir gs://my-bucket/folder

• Storage Transfer Service is optimized for more than 1 TiB. It moves data between S3, Azure Blob, Cloud Storage, on-premises systems (with Docker agents that access local NFS), public URLs, and HDFS. It automates and schedules transfers, encrypts in transit, and validates integrity. It has no SLA.
• Transfer Appliance also has an online mode that sends copied data to Cloud Storage. After copying the data, Google wipes the device.
• For BigQuery, loading from Cloud Storage uses bq load (previous article).

Exam scenarios

Common exam traps