Cloud load balancing > Documentation > Guides | SSL Proxy Load Balancing overview | My 30+ minutes reading time

 Sept. 15, 2021

Here is the link. 

SSL Proxy Load Balancing overview

 

SSL Proxy Load Balancing is a reverse proxy load balancer that distributes SSL traffic coming from the internet to virtual machine (VM) instances in your Google Cloud VPC network.

When using SSL Proxy Load Balancing for your SSL traffic, user SSL (TLS) connections are terminated at the load balancing layer, and then proxied to the closest available backend instances by using either SSL (recommended) or TCP. For the types of backends that are supported, see Backends.

With the Premium Tier, SSL Proxy Load Balancing can be configured as a global load balancing service. With Standard Tier, the SSL proxy load balancer handles load balancing regionally. For details, see Load balancer behavior in Network Service Tiers.

In this example, traffic from users in Iowa and Boston is terminated at the load balancing layer, and a separate connection is established to the selected backend.

Cloud Load Balancing with SSL termination (click to enlarge)

SSL Proxy Load Balancing is intended for non-HTTP(S) traffic. For HTTP(S) traffic, we recommend that you use HTTP(S) Load Balancing.

For information about how the Google Cloud load balancers differ from each other, see the following documents:

• Load balancing overview
• Choosing a load balancer
• Load balancer features

Benefits

Following are some benefits of using SSL Proxy Load Balancing:

• IPv6 termination. SSL Proxy Load Balancing supports both IPv4 and IPv6 addresses for client traffic. Client IPv6 requests are terminated at the load balancing layer, and then proxied over IPv4 to your VMs.

• Intelligent routing. The load balancer can route requests to backend locations where there is capacity. In contrast, an L3/L4 load balancer must route to regional backends without considering capacity. The use of smarter routing allows provisioning at N+1 or N+2 instead of x*N.

• Better utilization of backends. SSL processing can be very CPU-intensive if the ciphers used are not CPU efficient. To maximize CPU performance, use ECDSA SSL certificates and TLS 1.2, and prefer the ECDHE-ECDSA-AES128-GCM-SHA256 cipher suite for SSL between the load balancer and your backend instances.

• Certificate management. Your customer-facing SSL certificates can be either certificates that you obtain and manage (self-managed certificates), or certificates that Google obtains and manages for you (Google-managed certificates). Google-managed SSL certificates each support up to 100 domains. Multiple domains are supported for Google-managed certificates. You only need to provision certificates on the load balancer. On your VMs, you can simplify management by using self-signed certificates.

• Security patching. If vulnerabilities arise in the SSL or TCP stack, we apply patches at the load balancer automatically to keep your VMs safe.

• Support for the following well-known ports: 25, 43, 110, 143, 195, 443, 465, 587, 700, 993, 995, 1883, 3389, 5222, 5432, 5671, 5672, 5900, 5901, 6379, 8085, 8099, 9092, 9200, and 9300. When you use Google-managed SSL certificates with SSL Proxy Load Balancing, the frontend port for traffic must be 443 to enable the Google-managed SSL certificates to be provisioned and renewed.

• SSL policies. SSL policies give you the ability to control the features of SSL that your SSL proxy load balancer negotiates with clients.

• Geographic control over where TLS is terminated. The SSL proxy load balancer terminates TLS in locations that are distributed globally, so as to minimize latency between clients and the load balancer. If you require geographic control over where TLS is terminated, you should use Network Load Balancing instead, and terminate TLS on backends that are located in regions appropriate to your needs.

Architecture

The following are components of SSL proxy load balancers.

Forwarding rules and IP addresses

Forwarding rules route traffic by IP address, port, and protocol to a load balancing configuration consisting of a target proxy and a backend service.

Each forwarding rule provides a single IP address that you can use in DNS records for your application. No DNS-based load balancing is required. You can either reserve a static IP address that you can use or let Cloud Load Balancing assign one for you. We recommend that you reserve a static IP address; otherwise, you must update your DNS record with the newly- assigned ephemeral IP address whenever you delete a forwarding rule and create a new one.

Each external forwarding rule that you use in an SSL proxy load balancer can reference exactly one of the ports listed in: Port specifications for forwarding rules.

Target proxies

SSL Proxy Load Balancing terminates SSL connections from the client and creates new connections to the backends. The target proxy routes incoming requests directly to the backend service.

By default, the original client IP address and port information is not preserved. You can preserve this information by using the PROXY protocol.

SSL certificates

You must install one or more SSL certificates on the target SSL proxy. These certificates are used by target SSL proxies to secure communications between a Google Front End (GFE) and the client. These can be self-managed or Google-managed SSL certificates. For information about SSL certificate limits and quotas, see SSL certificates on the load balancing quotas page.

You can create SSL policies to control the features of SSL that your load balancer negotiates. For details, see SSL policies overview.

Sending traffic over unencrypted TCP between the load balancing layer and backend instances enables you to offload SSL processing from your backends; however, it also reduces security. Therefore, we do not recommend it. For the best security, use end-to-end encryption for your SSL proxy load balancer deployment. For more information, see Encryption from the load balancer to the backends.

For general information about how Google encrypts user traffic, see the Encryption in Transit in Google Cloud white paper.

Backend services

Backend services direct incoming traffic to one or more attached backends. Each backend is composed of an instance group or network endpoint group, and information about the backend's serving capacity. Backend serving capacity can be based on CPU or requests per second (RPS).

Each backend service specifies the health checks to perform for the available backends.

To ensure minimal interruptions to your users, you can enable connection draining on backend services. Such interruptions might happen when a backend is terminated, removed manually, or removed by an autoscaler. To learn more about using connection draining to minimize service interruptions, see Enabling connection draining.

Firewall rules

The backend instances must allow connections from the following sources:

• The load balancer Google Front End (GFE) for all requests sent to your backends
• Health check probes

To allow this traffic, you must create ingress firewall rules. The ports for these firewall rules must allow traffic as follows:

• To the destination port for each backend service's health check.

• For instance group backends: Determined by the mapping between the backend service's named port and the port numbers associated with that named port on each instance group. The numbers can vary among instance groups assigned to the same backend service.

• For GCE_VM_IP_PORT NEG backends: To the port numbers of the endpoints.

Firewall rules are implemented at the VM instance level, not on GFE proxies. You cannot use Google Cloud firewall rules to prevent traffic from reaching the load balancer. You can use Google Cloud Armor to achieve this.

For more information about health check probes and why it's necessary to allow traffic from them, see Probe IP ranges and firewall rules.

For SSL proxy load balancers and TCP proxy load balancers, the required source ranges are as follows:

• 130.211.0.0/22
• 35.191.0.0/16

These ranges apply to health checks and requests from the GFE.

Important: Make sure to allow packets from the full health check ranges. If your firewall rule allows packets from only a subset of the ranges, you might see health check failures because the load balancer can't communicate with your backends. This causes connection timeouts.

Source IP addresses

The source IP address for packets, as seen by the backends, is not the Google Cloud external IP address of the load balancer. In other words, there are two TCP connections:

• Connection 1, from original client to the load balancer (GFE):

  • Source IP address: the original client (or external IP address if the client is behind NAT or a forward proxy).
  • Destination IP address: your load balancer's IP address.

• Connection 2, from the load balancer (GFE) to the backend VM or endpoint:

  • Source IP address: an IP address in one of the ranges specified in Firewall rules.

  • Destination IP address: the internal IP address of the backend VM or container in the Virtual Private Cloud (VPC) network.

Preserving client source IP addresses

To preserve the original source IP addresses of incoming connections to the load balancer, you can configure the load balancer to prepend a PROXY protocol version 1 header to retain the original connection information. For more information, see Update proxy protocol header for the proxy.

Open ports

The SSL proxy load balancers are reverse proxy load balancers. The load balancer terminates incoming connections, and then opens new connections from the load balancer to the backends. These load balancers are implemented using Google Front End (GFE) proxies worldwide.

GFEs have several open ports to support other Google services that run on the same architecture. To see a list of some of the ports likely to be open on GFEs, see Forwarding rule: Port specifications. There might be other open ports for other Google services running on GFEs.

Running a port scan on the IP address of a GFE-based load balancer is not useful from an auditing perspective for the following reasons:

• A port scan (for example, with nmap) generally expects no response packet or a TCP RST packet when performing TCP SYN probing. GFEs will send SYN-ACK packets in response to SYN probes for a variety of ports if your load balancer uses a Premium Tier IP address. However, GFEs only send packets to your backends in response to packets sent to your load balancer's IP address and the destination port configured on its forwarding rule. Packets sent to different load balancer IP addresses or your load balancer's IP address on a port not configured in your forwarding rule do not result in packets being sent to your load balancer's backends. Even without any special configuration, Google infrastructure and GFEs provide defense-in-depth for DDoS attacks and SYN floods.

• Packets sent to the IP address of your load balancer could be answered by any GFE in Google's fleet; however, scanning a load balancer IP address and destination port combination only interrogates a single GFE per TCP connection. The IP address of your load balancer is not assigned to a single device or system. Thus, scanning the IP address of a GFE-based load balancer does not scan all the GFEs in Google's fleet.

With that in mind, the following are some more effective ways to audit the security of your backend instances:

• A security auditor should inspect the forwarding rules configuration for the load balancer's configuration. The forwarding rules define the destination port for which your load balancer accepts packets and forwards them to the backends. For GFE-based load balancers, each external forwarding rule can only reference a single destination TCP port.

• A security auditor should inspect the firewall rule configuration applicable to backend VMs. The firewall rules that you set block traffic from the GFEs to the backend VMs, but do not block incoming traffic to the GFEs. For best practices, see the firewall rules section.

Traffic distribution

The way an SSL proxy load balancer distributes traffic to its backends depends on the balancing mode and the hashing method selected to choose a backend (session affinity).

How connections are distributed

TCP Proxy Load Balancing can be configured as a global load balancing service with Premium Tier, and as a regional service in the Standard Tier.

For Premium Tier:

• You can have only one backend service, and the backend service can have backends in multiple regions. For global load balancing, you deploy your backends in multiple regions, and the load balancer automatically directs traffic to the region closest to the user. If a region is at capacity, the load balancer automatically directs new connections to another region with available capacity. Existing user connections remain in the current region.
• Google advertises your load balancer's IP address from all points of presence, worldwide. Each load balancer IP address is global anycast.
• If you configure a backend service with backends in multiple regions, Google Front Ends (GFEs) attempt to direct requests to healthy backend instance groups or NEGs in the region closest to the user. Details for the process are documented on this page.

For Standard Tier:

• Google advertises your load balancer's IP address from points of presence associated with the forwarding rule's region. The load balancer uses a regional external IP address.

• You can configure backends in the same region as the forwarding rule. The process documented here still applies, but GFEs only direct requests to healthy backends in that one region.

Request distribution process:

The balancing mode and choice of target define backend fullness from the perspective of each zonal GCE_VM_IP_PORT NEG, zonal instance group, or zone of a regional instance group. Distribution within a zone is then done with consistent hashing.

The load balancer uses the following process:

1. The forwarding rule's external IP address is advertised by edge routers at the borders of Google's network. Each advertisement lists a next hop to a Layer 3/4 load balancing system (Maglev) as close to the user as possible.
2. Maglev systems inspect the source IP address of the incoming packet. They direct the incoming request to the Maglev systems that Google's geo-IP systems determine are as close to the user as possible.
3. The Maglev systems route traffic to a first-layer Google Front End (GFE). The first-layer GFE terminates TLS if required and then routes traffic to second-layer GFEs according to this process:
1. If a backend service uses instance group or GCE_VM_IP_PORT NEG backends, the first layer-GFEs prefer second-layer GFEs that are located in or near the region that contains the instance group or NEG.
2. For backend buckets and backend services with hybrid NEGs, serverless NEGs, and internet NEGs, the first-layer GFEs choose second-layer GFEs in a subset of regions such that the round trip time between the two GFEs is minimized.

   Second-layer GFE preference is not a guarantee, and it can dynamically change based on Google's network conditions and maintenance.

   Second-layer GFEs are aware of health check status and actual backend capacity usage.

4. The second-layer GFE directs requests to backends in zones within its region.
5. For Premium Tier, sometimes second-layer GFEs send requests to backends in zones of different regions. This behavior is called spillover.

Spillover is governed by two principles:

• Spillover is possible when all backends known to a second-layer GFE are at capacity or are unhealthy.
• The second-layer GFE has information for healthy, available backends in zones of a different region.

The second-layer GFEs are typically configured to serve a subset of backend locations.

Spillover behavior does not exhaust all possible Google Cloud zones. If you need to direct traffic away from backends in a particular zone or in an entire region, you must set the capacity scaler to zero. Configuring backends to fail health checks does not guarantee that the second-layer GFE spills over to backends in zones of a different region.

6. When distributing requests to backends, GFEs operate at a zonal level.

   With a low number of connections, second-layer GFEs sometimes prefer one zone in a region over the other zones. This preference is normal and expected. The distribution among zones in the region doesn't become even until the load balancer receives more connections.

Balancing mode

When you add a backend to the backend service, you set a load balancing mode.

For SSL Proxy Load Balancing, the balancing mode can be CONNECTION or UTILIZATION.

If the load balancing mode is CONNECTION, the load is spread based on how many concurrent connections the backend can handle. You must also specify exactly one of the following parameters: maxConnections (except for regional managed instance groups), maxConnectionsPerInstance, or maxConnectionsPerEndpoint.

If the load balancing mode is UTILIZATION, the load is spread based on the utilization of instances in an instance group.

For information about comparing the load balancer types and the supported balancing modes, see Load balancing methods.

Session affinity

Session affinity sends all requests from the same client to the same backend, if the backend is healthy and has capacity.

SSL Proxy Load Balancing offers client IP affinity, which forwards all requests from the same client IP address to the same backend.

Failover

If a backend becomes unhealthy, traffic is automatically redirected to healthy backends within the same region. If all backends within a region are unhealthy, traffic is distributed to healthy backends in other regions (Premium Tier only). If all backends are unhealthy, the load balancer drops traffic.

Load balancing for GKE applications

If you are building applications in Google Kubernetes Engine, you can use standalone NEGs to load balance traffic directly to containers. With standalone NEGs you are responsible for creating the Service object that creates the NEG, and then associating the NEG with the backend service so that the load balancer can connect to the Pods.

Related GKE documentation:

• Container-native load balancing through standalone NEGs

Limitations

• SSL proxy load balancers each have a single backend service resource. Changes to the backend service are not instantaneous. It can take several minutes for changes to propagate to Google Front Ends (GFEs).

• SSL proxy load balancers do not support client certificate-based authentication, also known as mutual TLS authentication.

• Although SSL Proxy Load Balancing can handle HTTPS traffic, we don't recommend this. You should instead use HTTP(S) Load Balancing for HTTPS traffic. HTTP(S) Load Balancing also does the following, which makes it a better choice in most cases:

  • Negotiates HTTP/2 and HTTP/3.
  • Rejects invalid HTTP requests or responses.
  • Forwards requests to different VMs based on URL host and path.
  • Integrates with Cloud CDN.
  • Spreads the request load more evenly among backend instances, providing better backend utilization. HTTPS load balances each request separately, whereas SSL Proxy Load Balancing sends all bytes from the same SSL or TCP connection to the same backend instance.

• For SSL proxy load balancers with Google-managed SSL certificates, the frontend ports must include 443 for the certificates to be provisioned and renewed successfully.

  SSL Proxy Load Balancing can be used for other protocols that use SSL, such as WebSockets and IMAP over SSL.

• SSL proxy load balancers support only lowercase characters in domains in a common name (CN) attribute or a subject alternative name (SAN) attribute of the certificate. Certificates with uppercase characters in domains are returned only when set as the primary certificate in the target proxy.

• SSL proxy load balancers do not support VPC Network Peering.

What's next

• To configure an SSL proxy load balancer, see Setting up SSL Proxy Load Balancing.
• To create, modify, or delete an SSL policy, see Using SSL policies.
• To set up monitoring for your SSL proxy load balancer, see Using monitoring.
• To view a list of Google points of presence (PoPs), see GFE locations.

419-10688 140 ST SURREY V3T 0M6 | $425,000 dollars

Apr 12, 2021

$415,000

 

Jan 26, 2019

$322,000

$425,000 

Year built	2018
Description	Strata Apartment - Frame
Bedrooms	1
Baths	1
Carports
Garages
Land size
First floor area
Second floor area
Basement finish area
Strata area	637
Building storeys	4

Gain Agility Through GCP (Cloud Next '19)

Sept. 14, 2021

Here is the link. 

Colgate had over 1,100 VMs on-premises running their existing Windows environment. Seeking a major upgrade in agility and scaling, they decided it was time to migrate to the cloud. Come learn about how Colgate, who had just a small IT team dedicated to the job, leveraged Velostrata and implementation partners to migrate an ECommerce and other VMs to the cloud.

Gain Agility Through GCP → http://bit.ly/2TXq7aW Subscribe to the GCP Channel → https://bit.ly/GCloudPlatform Speaker(s): Naomi Klamen, James Modica, Vincent Ricci Session ID: ARC214 product:Cloud For Enterprise,Cloud IAM,Compute Engine; fullname:Naomi Klamen; event: Google Cloud Next 2019; re_ty: Publish; product: Cloud - General; fullname: Naomi Klamen, James Modica, Vincent Ricci;

Cloud Bigtable > documentation > Guides: Overview of Bigtable | My 60+ minutes study

Sept. 14, 2021

Here is the link. 

Overview of Bigtable

 

Cloud Bigtable is a sparsely populated table that can scale to billions of rows and thousands of columns, enabling you to store terabytes or even petabytes of data. A single value in each row is indexed; this value is known as the row key. Bigtable is ideal for storing very large amounts of single-keyed data with very low latency. It supports high read and write throughput at low latency, and it is an ideal data source for MapReduce operations.

Bigtable is exposed to applications through multiple client libraries, including a supported extension to the Apache HBase library for Java. As a result, it integrates with the existing Apache ecosystem of open-source Big Data software.

Bigtable's powerful back-end servers offer several key advantages over a self-managed HBase installation:

• Incredible scalability. Bigtable scales in direct proportion to the number of machines in your cluster. A self-managed HBase installation has a design bottleneck that limits the performance after a certain threshold is reached. Bigtable does not have this bottleneck, so you can scale your cluster up to handle more reads and writes.
• Simple administration. Bigtable handles upgrades and restarts transparently, and it automatically maintains high data durability. To replicate your data, simply add a second cluster to your instance, and replication starts automatically. No more managing replicas or regions; just design your table schemas, and Bigtable will handle the rest for you.
• Cluster resizing without downtime. You can increase the size of a Bigtable cluster for a few hours to handle a large load, then reduce the cluster's size again—all without any downtime. After you change a cluster's size, it typically takes just a few minutes under load for Bigtable to balance performance across all of the nodes in your cluster.

What it's good for

Bigtable is ideal for applications that need very high throughput and scalability for key/value data, where each value is typically no larger than 10 MB. Bigtable also excels as a storage engine for batch MapReduce operations, stream processing/analytics, and machine-learning applications.

You can use Bigtable to store and query all of the following types of data:

• Time-series data, such as CPU and memory usage over time for multiple servers.
• Marketing data, such as purchase histories and customer preferences.
• Financial data, such as transaction histories, stock prices, and currency exchange rates.
• Internet of Things data, such as usage reports from energy meters and home appliances.
• Graph data, such as information about how users are connected to one another.

Bigtable storage model

Bigtable stores data in massively scalable tables, each of which is a sorted key/value map. The table is composed of rows, each of which typically describes a single entity, and columns, which contain individual values for each row. Each row is indexed by a single row key, and columns that are related to one another are typically grouped together into a column family. Each column is identified by a combination of the column family and a column qualifier, which is a unique name within the column family.

Each row/column intersection can contain multiple cells. Each cell contains a unique timestamped version of the data for that row and column. Storing multiple cells in a column provides a record of how the stored data for that row and column has changed over time. Bigtable tables are sparse; if a column is not used in a particular row, it does not take up any space.

A few things to notice in this illustration:

• Columns can be unused in a row.
• Each cell in a given row and column has a unique timestamp (t).

Bigtable architecture

The following diagram shows a simplified version of Bigtable's overall architecture:

As the diagram illustrates, all client requests go through a front-end server before they are sent to a Bigtable node. (In the original Bigtable paper, these nodes are called "tablet servers.") The nodes are organized into a Bigtable cluster, which belongs to a Bigtable instance, a container for the cluster.

Note: The diagram shows an instance with a single cluster. You can also add clusters to replicate your data, which improves data availability and durability.

Each node in the cluster handles a subset of the requests to the cluster. By adding nodes to a cluster, you can increase the number of simultaneous requests that the cluster can handle, as well as the maximum throughput for the entire cluster. If you enable replication by adding a second cluster, you can also send different types of traffic to different clusters, and you can fail over to one cluster if the other cluster becomes unavailable.

A Bigtable table is sharded into blocks of contiguous rows, called tablets, to help balance the workload of queries. (Tablets are similar to HBase regions.) Tablets are stored on Colossus, Google's file system, in SSTable format. An SSTable provides a persistent, ordered immutable map from keys to values, where both keys and values are arbitrary byte strings. Each tablet is associated with a specific Bigtable node. In addition to the SSTable files, all writes are stored in Colossus's shared log as soon as they are acknowledged by Bigtable, providing increased durability.

Importantly, data is never stored in Bigtable nodes themselves; each node has pointers to a set of tablets that are stored on Colossus. As a result:

• Rebalancing tablets from one node to another is very fast, because the actual data is not copied. Bigtable simply updates the pointers for each node.
• Recovery from the failure of a Bigtable node is very fast, because only metadata needs to be migrated to the replacement node.
• When a Bigtable node fails, no data is lost.

See Instances, Clusters, and Nodes for more information about how to work with these fundamental building blocks.

Load balancing

Each Bigtable zone is managed by a primary process, which balances workload and data volume within clusters. This process splits busier/larger tablets in half and merges less-accessed/smaller tablets together, redistributing them between nodes as needed. If a certain tablet gets a spike of traffic, Bigtable splits the tablet in two, then moves one of the new tablets to another node. Bigtable manages all of the splitting, merging, and rebalancing automatically, saving users the effort of manually administering their tablets. Understanding Bigtable Performance provides more details about this process.

To get the best write performance from Bigtable, it's important to distribute writes as evenly as possible across nodes. One way to achieve this goal is by using row keys that do not follow a predictable order. For example, usernames tend to be distributed more or less evenly throughout the alphabet, so including a username at the start of the row key will tend to distribute writes evenly.

At the same time, it's useful to group related rows so they are adjacent to one another, which makes it much more efficient to read several rows at the same time. For example, if you're storing different types of weather data over time, your row key might be the location where the data was collected, followed by a timestamp (for example, WashingtonDC#201803061617). This type of row key would group all of the data from one location into a contiguous range of rows. For other locations, the row would start with a different identifier; with many locations collecting data at the same rate, writes would still be spread evenly across tablets.

See Choosing a row key for more details about choosing an appropriate row key for your data.

Supported data types

Bigtable treats all data as raw byte strings for most purposes. The only time Bigtable tries to determine the type is for increment operations, where the target must be a 64-bit integer encoded as an 8-byte big-endian value.

Memory and disk usage

The following sections describe how several components of Bigtable affect memory and disk usage for your instance.

Unused columns

Columns that are not used in a Bigtable row do not take up any space in that row. Each row is essentially a collection of key/value entries, where the key is a combination of the column family, column qualifier and timestamp. If a row does not include a value for a specific column, the key/value entry is simply not present.

Column qualifiers

Column qualifiers take up space in a row, since each column qualifier used in a row is stored in that row. As a result, it is often efficient to use column qualifiers as data.

Compactions

Bigtable periodically rewrites your tables to remove deleted entries, and to reorganize your data so that reads and writes are more efficient. This process is known as a compaction. There are no configuration settings for compactions—Bigtable compacts your data automatically.

Mutations and deletions

Mutations, or changes, to a row take up extra storage space, because Bigtable stores mutations sequentially and compacts them only periodically. When Bigtable compacts a table, it removes values that are no longer needed. If you update the value in a cell, both the original value and the new value will be stored on disk for some amount of time until the data is compacted.

Deletions also take up extra storage space, at least in the short term, because deletions are actually a specialized type of mutation. Until the table is compacted, a deletion uses extra storage rather than freeing up space.

Data compression

Bigtable compresses your data automatically using an intelligent algorithm. You cannot configure compression settings for your table. However, it is useful to know how to store data so that it can be compressed efficiently:

• Random data cannot be compressed as efficiently as patterned data. Patterned data includes text, such as the page you're reading right now.
• Compression works best if identical values are near each other, either in the same row or in adjoining rows. If you arrange your row keys so that rows with identical chunks of data are next to each other, the data can be compressed efficiently.
• Compress values larger than 1 MiB before storing them in Bigtable. This saves CPU cycles, server memory and network bandwidth. Bigtable automatically turns off compression for values larger than 1 MiB.

Data durability

When you use Bigtable, your data is stored on Colossus, Google's internal, highly durable file system, using storage devices in Google's data centers. You do not need to run an HDFS cluster or any other file system to use Bigtable. If your instance uses replication, Bigtable maintains one copy of your data in Colossus for each cluster in the instance. Each copy is located in a different zone or region, further improving durability.

Behind the scenes, Google uses proprietary storage methods to achieve data durability above and beyond what's provided by standard HDFS three-way replication. In addition, we create copies of your data to protect against catastrophic events and provide for disaster recovery.

Consistency model

Single-cluster Bigtable instances provide strong consistency. By default, instances that have more than one cluster provide eventual consistency, but for some use cases they can be configured to provide read-your-writes consistency or strong consistency, depending on the workload and app profile settings.

Security

Access to your Bigtable tables is controlled by your Google Cloud project and the Identity and Access Management (IAM) roles that you assign to users. For example, you can assign IAM roles that prevent individual users from reading from tables, writing to tables, or creating new instances. If someone does not have access to your project or does not have an IAM role with appropriate permissions for Bigtable, they cannot access any of your tables.

You can manage security at the project, instance, and table levels. Bigtable does not support row-level, column-level, or cell-level security restrictions.

Encryption

By default, all data stored within Google Cloud, including the data in Bigtable tables, is encrypted at rest using the same hardened key management systems that we use for our own encrypted data.

If you want more control over the keys used to encrypt your Bigtable data at rest, you can use customer-managed encryption keys (CMEK).

Backups

Bigtable backups let you save a copy of a table's schema and data, then restore from the backup to a new table at a later time. Backups can help you recover from application-level data corruption or from operator errors such as accidentally deleting a table.

Other storage and database options

Bigtable is not a relational database. It does not support SQL queries, joins, or multi-row transactions.

• If you need full SQL support for an online transaction processing (OLTP) system, consider Cloud Spanner or Cloud SQL.
• If you need interactive querying in an online analytical processing (OLAP) system, consider BigQuery.
• If you need to store highly structured objects in a document database, with support for ACID transactions and SQL-like queries, consider Firestore.
• For in-memory data storage with low latency, consider Memorystore.
• To sync data between users in realtime, consider the Firebase Realtime Database.

For more information about other database options, see the overview of database services. Google Cloud also has a variety of storage options.

What's next

• Try a Bigtable quickstart using cbt, the command-line tool for Bigtable. If you're familiar with HBase, try the HBase shell quickstart.
• Work through a Bigtable codelab.
• Learn more about Bigtable instances, clusters, and nodes.
• Learn how to create a Bigtable instance.
• Learn about the client libraries for Bigtable.
• Read the original OSDI paper about Bigtable.