Thursday, August 12, 2021

BigTable: C# code say hello | My first C# coding learning experience | 60 minutes

August 12, 2021

Here is the article. 

 Example: C# "Hello world" application

This code sample is a "hello world" application written in C#. The sample illustrates how to complete the following tasks:

  • Connect to a Cloud Bigtable instance
  • Create a new table.
  • Write data to the table.
  • Read the data back.
  • Delete the table.

This code communicates with Cloud Bigtable using the C# Admin API and C# Data API libraries in the Google Cloud Client Libraries for .NET.

To run this sample program, follow the .NET Cloud Bigtable Samples instructions on GitHub. Complete the Build and Run and Quick Start steps to create resources that you can use in your Hello World application. Make sure you edit the HelloWorld.cs file to add the names of the resources you create.

The sample application connects to Bigtable and demonstrates some simple operations.

To get started, create two client objects that you can use to connect to Bigtable. The C# Admin API's BigtableTableAdminClient helps you create and delete instances and tables. The C# Data API's BigtableClient helps you read and write table data.

bigtable/api/HelloWorld/HelloWorld.cs
// BigtableTableAdminClient API lets us create, manage and delete tables.
BigtableTableAdminClient bigtableTableAdminClient = BigtableTableAdminClient.Create();

// BigtableClient API lets us read and write to a table.
BigtableClient bigtableClient = BigtableClient.Create();

Call the admin client's CreateTable() method to generate a Table object that stores the "hello world" greetings. The table has a single column family that retains one version of each value.

bigtable/api/HelloWorld/HelloWorld.cs
// Create a table with a single column family.
Console.WriteLine($"Create new table: {tableId} with column family: {columnFamily}, instance: {instanceId}");

// Check whether a table with given TableName already exists.
if (!TableExist(bigtableTableAdminClient))
{
    bigtableTableAdminClient.CreateTable(
        new InstanceName(projectId, instanceId),
        tableId,
        new Table
        {
            Granularity = Table.Types.TimestampGranularity.Millis,
            ColumnFamilies =
            {
                {
                    columnFamily, new ColumnFamily
                    {
                        GcRule = new GcRule
                        {
                            MaxNumVersions = 1
                        }
                    }
                }
            }
        });
    // Confirm that table was created successfully.
    Console.WriteLine(TableExist(bigtableTableAdminClient)
        ? $"Table {tableId} created successfully\n"
        : $"There was a problem creating a table {tableId}");
}
else
{
    Console.WriteLine($"Table: {tableId} already exists");
}

Use the string array s_greetings[], which contains three simple greetings, as a source of data to write to the table. First, write a single row to the table using MutateRow(). Then loop through the rest of the array to build a MutateRowsRequest object that contains an entry for each greeting. Make the request to write all the entries at once with MutateRows(). Then loop through the returned response to check the status code for each entry to make sure it was written successfully.

bigtable/api/HelloWorld/HelloWorld.cs
// Initialize Google.Cloud.Bigtable.V2.TableName object.
Google.Cloud.Bigtable.Common.V2.TableName tableName = new Google.Cloud.Bigtable.Common.V2.TableName(projectId, instanceId, tableId);

// Write some rows
/* Each row has a unique row key.

       Note: This example uses sequential numeric IDs for simplicity, but
       this can result in poor performance in a production application.
       Since rows are stored in sorted order by key, sequential keys can
       result in poor distribution of operations across nodes.

       For more information about how to design a Bigtable schema for the
       best performance, see the documentation:

       https://cloud.google.com/bigtable/docs/schema-design */

Console.WriteLine($"Write some greetings to the table {tableId}");

// Insert 1 row using MutateRow()
s_greetingIndex = 0;
try
{
    bigtableClient.MutateRow(tableName, rowKeyPrefix + s_greetingIndex, MutationBuilder());
    Console.WriteLine($"\tGreeting:   -- {s_greetings[s_greetingIndex],-18}-- written successfully");
}
catch (Exception ex)
{
    Console.WriteLine($"\tFailed to write greeting: --{s_greetings[s_greetingIndex]}");
    Console.WriteLine(ex.Message);
    throw;
}

// Insert multiple rows using MutateRows()
// Build a MutateRowsRequest (contains table name and a collection of entries).
MutateRowsRequest request = new MutateRowsRequest
{
    TableNameAsTableName = tableName
};

s_mapToOriginalGreetingIndex = new List<int>();
while (++s_greetingIndex < s_greetings.Length)
{
    s_mapToOriginalGreetingIndex.Add(s_greetingIndex);
    // Build an entry for every greeting (consists of rowkey and a collection of mutations).
    string rowKey = rowKeyPrefix + s_greetingIndex;
    request.Entries.Add(Mutations.CreateEntry(rowKey, MutationBuilder()));
}

// Make the request to write multiple rows.
MutateRowsResponse response = bigtableClient.MutateRows(request);

// Check the status code of each entry to ensure that it was written successfully.
foreach (MutateRowsResponse.Types.Entry entry in response.Entries)
{
    s_greetingIndex = s_mapToOriginalGreetingIndex[(int)entry.Index];
    if (entry.Status.Code == 0)
    {
        Console.WriteLine($"\tGreeting:   -- {s_greetings[s_greetingIndex],-18}-- written successfully");
    }
    else
    {
        Console.WriteLine($"\tFailed to write greeting: --{s_greetings[s_greetingIndex]}");
        Console.WriteLine(entry.Status.Message);
    }
}

Mutation MutationBuilder() =>
    Mutations.SetCell(columnFamily, columnName, s_greetings[s_greetingIndex], new BigtableVersion(DateTime.UtcNow));

Before you read the data that you wrote, create a filter to limit the data that Bigtable returns. This filter tells Bigtable to return only the most recent version of each value, even if the table contains older cells that are eligible for garbage collection but have not yet been deleted.

bigtable/api/HelloWorld/HelloWorld.cs
RowFilter filter = RowFilters.CellsPerRowLimit(1);

Use the ReadRow() method, passing in the filter you just created, to get one version of each value in that row.

bigtable/api/HelloWorld/HelloWorld.cs
// Read from the table.
Console.WriteLine("Read the first row");

int rowIndex = 0;

// Read a specific row. Apply a filter to return latest only cell value accross entire row.
Row rowRead = bigtableClient.ReadRow(
    tableName, rowKey: rowKeyPrefix + rowIndex, filter: filter);
Console.WriteLine(
    $"\tRow key: {rowRead.Key.ToStringUtf8()} " +
    $"  -- Value: {rowRead.Families[0].Columns[0].Cells[0].Value.ToStringUtf8(),-16} " +
    $"  -- Time Stamp: {rowRead.Families[0].Columns[0].Cells[0].TimestampMicros}");

Call the ReadRows() method, passing in the filter, to get all of the rows in the table. Because you passed in the filter, Bigtable returns only one version of each value.

bigtable/api/HelloWorld/HelloWorld.cs
Console.WriteLine("Read all rows using streaming");
// stream the content of the whole table. Apply a filter to return latest only cell values accross all rows.
ReadRowsStream responseRead = bigtableClient.ReadRows(tableName, filter: filter);

Task printRead = PrintReadRowsAsync();
printRead.Wait();

async Task PrintReadRowsAsync()
{
    var responseEnumerator = responseRead.GetAsyncEnumerator(default);
    while (await responseEnumerator.MoveNextAsync())
    {
        Row row = responseEnumerator.Current;
        Console.WriteLine(
            $"\tRow key: {row.Key.ToStringUtf8()} " +
            $"  -- Value: {row.Families[0].Columns[0].Cells[0].Value.ToStringUtf8(),-16} " +
            $"  -- Time Stamp: {row.Families[0].Columns[0].Cells[0].TimestampMicros}");
    }
}

Delete the table with the DeleteTable() method.

bigtable/api/HelloWorld/HelloWorld.cs
// Clean up. Delete the table.
Console.WriteLine($"Delete table: {tableId}");

bigtableTableAdminClient.DeleteTable(name: tableName);
if (!TableExist(bigtableTableAdminClient))
{
    Console.WriteLine($"Table: {tableId} deleted successfully");
}

Here is the full code sample without comments.

bigtable/api/HelloWorld/HelloWorld.cs
// Copyright 2018 Google Inc.


using Google.Cloud.Bigtable.Admin.V2;
using Google.Cloud.Bigtable.V2;
using Grpc.Core;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace GoogleCloudSamples.Bigtable
{
    public class HelloWorld
    {
        private const string projectId = "YOUR-PROJECT-ID";

        private const string instanceId = "YOUR-INSTANCE-ID";

        private const string tableId = "Hello-Bigtable";
        private const string columnFamily = "cf";
        private const string columnName = "greeting";
        private static readonly string[] s_greetings = { "Hello World!", "Hello Bigtable!", "Hello C#!" };
        private static List<int> s_mapToOriginalGreetingIndex;
        private const string rowKeyPrefix = "greeting";
        private static int s_greetingIndex;

        private static void DoHelloWorld()
        {
            try
            {
                BigtableTableAdminClient bigtableTableAdminClient = BigtableTableAdminClient.Create();

                BigtableClient bigtableClient = BigtableClient.Create();

                Console.WriteLine($"Create new table: {tableId} with column family: {columnFamily}, instance: {instanceId}");

                if (!TableExist(bigtableTableAdminClient))
                {
                    bigtableTableAdminClient.CreateTable(
                        new InstanceName(projectId, instanceId),
                        tableId,
                        new Table
                        {
                            Granularity = Table.Types.TimestampGranularity.Millis,
                            ColumnFamilies =
                            {
                                {
                                    columnFamily, new ColumnFamily
                                    {
                                        GcRule = new GcRule
                                        {
                                            MaxNumVersions = 1
                                        }
                                    }
                                }
                            }
                        });
                    Console.WriteLine(TableExist(bigtableTableAdminClient)
                        ? $"Table {tableId} created successfully\n"
                        : $"There was a problem creating a table {tableId}");
                }
                else
                {
                    Console.WriteLine($"Table: {tableId} already exists");
                }

                Google.Cloud.Bigtable.Common.V2.TableName tableName = new Google.Cloud.Bigtable.Common.V2.TableName(projectId, instanceId, tableId);


                       Note: This example uses sequential numeric IDs for simplicity, but
                       this can result in poor performance in a production application.
                       Since rows are stored in sorted order by key, sequential keys can
                       result in poor distribution of operations across nodes.

                       For more information about how to design a Bigtable schema for the
                       best performance, see the documentation:

                       https://cloud.google.com/bigtable/docs/schema-design */

                Console.WriteLine($"Write some greetings to the table {tableId}");

                s_greetingIndex = 0;
                try
                {
                    bigtableClient.MutateRow(tableName, rowKeyPrefix + s_greetingIndex, MutationBuilder());
                    Console.WriteLine($"\tGreeting:   -- {s_greetings[s_greetingIndex],-18}-- written successfully");
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"\tFailed to write greeting: --{s_greetings[s_greetingIndex]}");
                    Console.WriteLine(ex.Message);
                    throw;
                }

                MutateRowsRequest request = new MutateRowsRequest
                {
                    TableNameAsTableName = tableName
                };

                s_mapToOriginalGreetingIndex = new List<int>();
                while (++s_greetingIndex < s_greetings.Length)
                {
                    s_mapToOriginalGreetingIndex.Add(s_greetingIndex);
                    string rowKey = rowKeyPrefix + s_greetingIndex;
                    request.Entries.Add(Mutations.CreateEntry(rowKey, MutationBuilder()));
                }

                MutateRowsResponse response = bigtableClient.MutateRows(request);

                foreach (MutateRowsResponse.Types.Entry entry in response.Entries)
                {
                    s_greetingIndex = s_mapToOriginalGreetingIndex[(int)entry.Index];
                    if (entry.Status.Code == 0)
                    {
                        Console.WriteLine($"\tGreeting:   -- {s_greetings[s_greetingIndex],-18}-- written successfully");
                    }
                    else
                    {
                        Console.WriteLine($"\tFailed to write greeting: --{s_greetings[s_greetingIndex]}");
                        Console.WriteLine(entry.Status.Message);
                    }
                }

                Mutation MutationBuilder() =>
                    Mutations.SetCell(columnFamily, columnName, s_greetings[s_greetingIndex], new BigtableVersion(DateTime.UtcNow));

                RowFilter filter = RowFilters.CellsPerRowLimit(1);

                Console.WriteLine("Read the first row");

                int rowIndex = 0;

                Row rowRead = bigtableClient.ReadRow(
                    tableName, rowKey: rowKeyPrefix + rowIndex, filter: filter);
                Console.WriteLine(
                    $"\tRow key: {rowRead.Key.ToStringUtf8()} " +
                    $"  -- Value: {rowRead.Families[0].Columns[0].Cells[0].Value.ToStringUtf8(),-16} " +
                    $"  -- Time Stamp: {rowRead.Families[0].Columns[0].Cells[0].TimestampMicros}");

                Console.WriteLine("Read all rows using streaming");
                ReadRowsStream responseRead = bigtableClient.ReadRows(tableName, filter: filter);

                Task printRead = PrintReadRowsAsync();
                printRead.Wait();

                async Task PrintReadRowsAsync()
                {
                    var responseEnumerator = responseRead.GetAsyncEnumerator(default);
                    while (await responseEnumerator.MoveNextAsync())
                    {
                        Row row = responseEnumerator.Current;
                        Console.WriteLine(
                            $"\tRow key: {row.Key.ToStringUtf8()} " +
                            $"  -- Value: {row.Families[0].Columns[0].Cells[0].Value.ToStringUtf8(),-16} " +
                            $"  -- Time Stamp: {row.Families[0].Columns[0].Cells[0].TimestampMicros}");
                    }
                }

                Console.WriteLine($"Delete table: {tableId}");

                bigtableTableAdminClient.DeleteTable(name: tableName);
                if (!TableExist(bigtableTableAdminClient))
                {
                    Console.WriteLine($"Table: {tableId} deleted successfully");
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Exception while running HelloWorld: {ex.Message}");
            }
        }

        private static bool TableExist(BigtableTableAdminClient bigtableTableAdminClient)
        {
            GetTableRequest request = new GetTableRequest
            {
                TableName = new Google.Cloud.Bigtable.Common.V2.TableName(projectId, instanceId, tableId),
                View = Table.Types.View.NameOnly
            };
            try
            {
                var tables = bigtableTableAdminClient.GetTable(request);
                return true;
            }
            catch (RpcException ex)
            {
                if (ex.StatusCode == StatusCode.NotFound)
                {
                    return false;
                }

                throw;
            }
        }

        public static int Main(string[] args)
        {
            if (projectId == "YOUR-PROJECT" + "-ID")
            {
                Console.WriteLine("Edit HelloWorld.cs and replace YOUR-PROJECT-ID with your project ID.");
                return -1;
            }
            if (instanceId == "YOUR-INSTANCE" + "-ID")
            {
                Console.WriteLine("Edit HelloWorld.cs and replace YOUR-INSTANCE-ID with your instance ID.");
                return -1;
            }

            DoHelloWorld();
            return 0;
        }
    }
}
Posted by at No comments:

Breakfast and learn: Cloud Spanner and Cloud Bigtable: Modern, Distributed Databases that Scale (Cloud Next ‘19 UK)

Aug. 12, 2021

Here is the link. 

The decision to modernize an existing application or build a new cloud-native application using fully-managed distributed databases is often not an easy one. What benefits do you receive? What do you trade off? Which database do you choose? This session provides a deep dive on two of the most powerful and innovative cloud-native database technologies: Cloud Bigtable and Cloud Spanner and discusses the right use cases for each. This session will also include demos.

Session ID: INFRA205 Speaker(s): Sharon Dashet, Database Specialist, Google Cloud 8:00, 25:30 Adam Levin, Product Marketing, Databases, Google Cloud 21:54

 

Posted by at No comments:

Wednesday, August 11, 2021

Bigtable in action (Google Cloud Next '17) | NoSQL best self-learning course

 Aug. 11, 2021

Here is the link. 

Google's billion-user services like Gmail and Google Maps depend on Bigtable to store data at massive scale and retrieve data with ultra low-latency. Today, many use cases such as IoT, finance, mapping, advertising and dealing with time-series data, face similar demands. In this video, you'll learn how to integrate Cloud Bigtable into your application architecture to solve the challenges of storing and retrieving data for these and other use cases. We'll cover the specifics of the Cloud Bigtable service and also dive into schema level design considerations. You'll also hear how customers have successfully leveraged Bigtable to solve their problems at scale and the patterns they've implemented on top of Cloud Bigtable. Missed the conference? Watch all the talks here: https://goo.gl/c1Vs3h Watch more talks about Infrastructure & Operations here: https://goo.gl/k2LOYG

My notes 

Google research in data technologies 

  1. 2002, GFS 
  2. 2004, MapReduce
  3. 2006, Bigtable
  4. 2008, Dremel, 
  5. 2010 - 2011, Colossus, Flume, Megastore
  6. 2012, Spanner
  7. Millwheel
  8. 2013, PubSub, F1
Data model & Usage 
  • NoSQL (no-join) distributed key-value store, designed to scale-out
  • has only one index (the row-key)
  • supports atomic single-row transactions
  • unwritten cells in do not take up any space

row key - 
sparse - Do not worry about wasting space 

Cells
  • every cell is versioned (default is timestamp on server)
  • garbage collection retains latest version (configurable)
  • expiration (optional) can be set at column-family level
  • periodic compaction relations unused space from cells
Writing / reading 

Writing 
  • Put
  • Increment
  • Append
  • Conditional updates
  • Bulk import
Reading 
  • Gets
  • Range scan
  • Filter
  • Full scan
  • Export
Schema design 

Rows 
  • updates are atomic - but only at the row level
  • store items related to a given entity in a single row
  • where atomic updates aren't needed and entity is large - then split it up
  • rows are sorted lexicographically by row-key
  • store related entities in adjacent rows
Row-key
  • determine a key strategy that facilitates common queries
  • choose keys that help distribute reads/writes and avoids hotspots
  • avoid solely monotonically increasing keys (timestamp or sequence)
  • a combined-key strategy is helpful
Tall vs, wide tables 

DBYO: don't build your own

abstractions and integration already exist for popular use cases:

  • JanusGraph - graph database
  • OpenTSDB - time-series database
  • Spotify/Heroic time-series database
  • GeoMesa geospatial querying 
Integration with Cloud Dataflow
Integration with Cloud Dataproc 

Server Density - Founder and CEO data availability 25:00 -> 


Sami Zuhuruddin 

https://www.linkedin.com/in/samizuh/








Posted by at No comments:

Breakfast and learn: Music Recommendations at Scale With Cloud Bigtable (Cloud Next '19)

Posted by at No comments:

Breakfast and learn: Moving from Cassandra to Auto-Scaling Bigtable at Spotify (Cloud Next '19)

August 11, 2021

Here is the link. 

In this session, we will discuss the differences between Cassandra and Cloud Bigtable, why and how Spotify migrated some of their workloads, and how they built an auto-scaler for Cloud Bigtable. Build with Google Cloud → https://bit.ly/2UhyEuj Watch more: Next '19 Databases Sessions here → https://bit.ly/Next19Databases Next ‘19 All Sessions playlist → https://bit.ly/Next19AllSessions Subscribe to the GCP Channel → https://bit.ly/GCloudPlatform

Speaker(s): Emilio Del Tessandoro, Sandhya Ghai Session ID: DBS200 product:Cloud Bigtable,Cloud Spanner; fullname:Sandhya Ghai; event: Google Cloud Next 2019; re_ty: Publish; product: Cloud - Databases - Cloud Bigtable; fullname: Emilio Del Tessandoro, Sandhya Ghai;

Posted by at No comments:

Tuesday, August 10, 2021

Road trip Whistler: Problems found on the road trip

 I took the road trip to Whistler alone. And I like to write down problems found on the trip. 

  1. Auto climate control did not work when I drove back; I need to order a used one from ebay.com;
  2. After $600 dollars auto maintenance from my neighbor, I decided to take a road trip 300 KM round trip in one day; 
  3. Google offline map did not work in my trip back to Vancouver until I drove over 50 KM; Need to find out; 
  4. I hurt my back when I hiked, I was little scared after I saw a big black bear with a group of people; I should keep my own pace and do not hurt my back. It will take weeks to recover, need to learn how to build strong muscle near my waistline; Weight training is needed;
  5. Photo and videos taken, how to make good plans. 

Posted by at No comments:

Sick day: Hurt my back | Whistler hiking

August 10, 2021

I took a sick day to find ways to recover my back. I hurt my back during Whistler hiking. 

Sick day planning

I need to make some plans for my sick day. I like to work around more instead of seating whole day. I like to go out and shop at Costco. 

I need to figure out how to restore my back muscle strength in order to stand up without using my arm to strength my body. 


Posted by at No comments:
Subscribe to: Posts (Atom)