Monday, August 19, 2019

109 Convert sorted list to binary search tree

I added some content to my sharing post, here is the link.

It is comeback practice for me to work on the algorithm. I did work on the algorithm 3 years 4 months ago.
I chose the algorithm to give the interviewee to work on in my last mock interview on interviewing.io. This is the second time I used the algorithm, my first time is over 6 months ago. But I found out that I did not really understand the algorithm very well. All I remember is that bottom up solution will have optimal time complexity O(N), N is total number of nodes in the linked list.
I wrote the solution based on the geekforgeeks.com article: https://www.geeksforgeeks.org/sorted-linked-list-to-balanced-bst/, and then I found the article through one of Leetcode discuss, https://articles.leetcode.com/convert-sorted-list-to-balanced-binary/. So I updated my code again.
What makes an optimal solution?
I reviewed the algorithm on August 9, 2019 5:07 pm. In terms of design of the algorithm, the algorithm should be able to do minimum, and also bug free.
Given a list with ascending order, how to construct a binary search tree with height difference than one?
It is true that inorder traversal the output will be in ascending order.
It is important to set root node of tree as the middle of list, otherwise it will not keep the property of binary search tree height difference less and equal than one.
First, think about every node in the tree will be iterated as root node once. So the list of the nodes should be iterated to match the iteration of tree nodes.
Second, apply inorder traversal and the list of iteration will match tree node iteration;
Third, think about how to avoid null pointer exception, using two index position of start and end position of list to help; one iteration of the list can get the length, and it will help to measure the list is empty or not.
The goal is for me to come out the optimal solution using O(N) and be able to write the code using less than five minutes.
Keywords
binary search tree
a list in ascending order
inorder traversal - left, root, right
tree height difference less and equal to one
Every node will be root node once
Move list pointer to next once
List move matches root node of tree iteration
count the length of list
Using middle = (start + end)/2, start is 0, and end is length of list - 1; in other words, start, end are two index of list.
I like to share my C# code.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace _109_convert_sorted_list_to_binary_search_tree
{
    /// <summary>
    /// review on 9/11/2018
    /// Review the blog:
    /// https://articles.leetcode.com/convert-sorted-list-to-balanced-binary/
    /// </summary>
    class Program
    {
        // Definition for singly-linked list.
        public class ListNode
        {
            public int val;
            public ListNode next;
            public ListNode(int x) { val = x; }
        }

        public class TreeNode
        {
            public int val;
            public TreeNode left;
            public TreeNode right;
            public TreeNode(int x) { val = x; }
        }

        static void Main(string[] args)
        {
            var node1 = new ListNode(1);
            var node2 = new ListNode(2);
            var node3 = new ListNode(3);
            var node4 = new ListNode(4);
            var node5 = new ListNode(5);
            var node6 = new ListNode(6);
            var node7 = new ListNode(7);

            node1.next = node2;
            node2.next = node3;
            node3.next = node4;
            node4.next = node5;
            node5.next = node6;
            node6.next = node7;

            var root = SortedListToBST(node1);
        }       

        /// <summary>
        /// The time complexity will be O(N), N is the total number of nodes in the sorted linked list.
        /// Use bottom up approach to construct the binary search tree.
        /// 
        /// </summary>
        /// <param name="head"></param>
        /// <returns></returns>
        public static TreeNode SortedListToBST(ListNode head)
        {
            if (head == null)
            {
                return null;
            }

            var iterate = head;
            int count = 0;
            while (iterate != null)
            {
                iterate = iterate.next;
                count++;
            }

            return InorderTraversalBottomUp(ref head, 0, count - 1);
        }

        /// <summary>
        /// BST inorder traversal - match the sorted linked list order
        /// In terms of constructing BST, the inorder traversal is applied. 
        /// https://articles.leetcode.com/convert-sorted-list-to-balanced-binary/
        /// </summary>
        /// <param name="start"></param>
        /// <param name="end"></param>
        /// <returns></returns>
        private static TreeNode InorderTraversalBottomUp(ref ListNode list, int start, int end)
        {
            if (start > end)
                return null;

            int middle = (start + end) / 2;

            // enforce inorder traversal - left, root, right 
            // Left
            var left = InorderTraversalBottomUp(ref list, start, middle - 1);

            // Root 
            // each node in the sorted list will be visited one by one,
            // the node will be the root node of the tree. 
            var root = new TreeNode(list.val);

            root.left = left;

            // go to next one in the sorted linked list 
            list = list.next;

            // Right
            root.right = InorderTraversalBottomUp(ref list, middle + 1, end);

            return root;
        }
    }
}
If you like my analysis and sharing, please give me an upvote. I also put all my C# algorithm together, the link is
https://github.com/jianminchen/Leetcode_Julia/tree/master/Leetcode discussion.

What a busy day!

August 19, 2019

Introduction

It is my busy day to stay in Pullman hotel. I enjoyed so many things in the hotel, and I like to write a blog about it.

My morning

I spent over two hours in the hotel gym. I like to work out like a really smart software engineer. Keep myself open to learn, workout in gym is such perfect time for me. I sweated after riding on a bike more than 22 minutes.

My lunch out

I walked 200 meters to meet Dr. Jin, who was my roommate back in 2001 in Florida. She just works there as a program manager. We chatted about one hour, I had chance to see her new car.

She works for a Johnson Johnson company. The world is so connected.

My two algorithms warm up

I like to study algorithms, both of them are tree algorithms.


Saturday, August 17, 2019

One more Amazon leadership principle

August 17, 2019

Introduction


It is time for me to review Amazon leadership principle. I like to learn one more leadership very well. My most favorite one is frugality leadership principle. I did personal finance research starting from Nov. 2018, I learned that being frugal is so important.

What is my most favorite one? 


I think that it should be "leaders should be able to develop others to be leaders".


214. Shortest Palindrome

August 17, 2019

Introduction


It is the good idea to show how to learn to solve a hard level algorithm. I believe that the learning process will help me to overcome difficulty in my daily job and help me to prepare more challenging programming task. I tried so many times to learn to solve 214 Shortest palindrome recently, more than three times I asked the algorithm in the mock interview as an interviewer.

Case study


I did write a post to share my understanding today. Here is the link.

It is so challenge to learn KMP algorithm and understand how to construct KMP table. I learned quickly by watching the video.
From 5:37 - 8:04, the explanation how to build longest prefix and suffix table is easy for me to follow,
a b c d a b c a
0 1 2 3 4 5 6 7
0 0 0 0 1 2 3 1 <- Lps Array (longest prefix suffix array)
I will add more explanation how to solve the problem later.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace _214_shortest_palindrome
{
    class Program
    {
        static void Main(string[] args)
        {
            RunKMPTableTestcase();
            var result = ShortestPalindrome("aacecaaa");
        }

        /// <summary>
        /// study video 
        /// https://www.youtube.com/watch?v=GTJr8OvyEVQ
        /// Knuth–Morris–Pratt(KMP) Pattern Matching(Substring search)
        /// 5:37 - 8:04
        /// </summary>
        public static void RunKMPTableTestcase()
        {
            // KMP table should be [0,0,0,0,1,2,3,1]
            var lps = ComputeLpsArray("abcdabca");
        }

        public static string ShortestPalindrome(string s)
        {
            var charArray = s.ToCharArray();
            Array.Reverse(charArray);
            int[] table = ComputeLpsArray(s + "." + new string(charArray));
            charArray = s.Substring(table[table.Length - 1]).ToCharArray();
            Array.Reverse(charArray);
            return new string(charArray) + s;
        }

        /// <summary>
        /// study code 
        /// https://leetcode.com/problems/shortest-palindrome/discuss/350795/C-Solution-O(n)-using-%22longest-prefix-suffix-array%22-of-KMP
        /// longest prefix and suffix array
        /// Detail explanation - 
        /// study video 
        /// https://www.youtube.com/watch?v=GTJr8OvyEVQ
        /// Knuth–Morris–Pratt(KMP) Pattern Matching(Substring search)
        /// 5:37 - 8:04
        /// a b c d a b c a
        /// 0 1 2 3 4 5 6 7
        /// 0 0 0 0 1 2 3 1
        /// </summary>
        /// <param name="str"></param>
        /// <returns></returns>
        private static int[] ComputeLpsArray(string str)
        {
            var length = str.Length;

            // default is 0 
            var lps = new int[length];

            for (int i = 1; i < length; i++)
            {
                int j = lps[i - 1];
                 
                while ((j > 0) && (str[i] != str[j]))
                {
                    j = lps[j - 1];
                }

                lps[i] = str[i] == str[j] ? j + 1 : j;
            }

            return lps;
        }
    }
}


Knuth–Morris–Pratt(KMP) Pattern Matching(Substring search)

Here is the link.

5:18 PM - 6:00 PM

I need to push myself hard to learn KMP algorithm in 14 minutes.

First 6 minutes video:

KMP Substring search
              0   1   2   3  4  5  6  7
text        a   b   c   b   c  g  l   x  --------. m

pattern   b   c   g   l   l  -> O(mn)

KMP algorithm can be completed in O(m + n) time complexity

KMP Substring search

text - abcxabcdabxabcdabcdabcy

pattern - abcdabcy


KMP Substring Search


AWS - Away teams

揭秘 AWS 内部开发和维护技术:Away Teams 概念,即为了达到最快速度,接受某些缺点


Here is the article. 

不过,互联网巨头亚马逊庞大的云计算部门AWS内部有个特别的消化系统:一个名为Away Teams的概念,这个概念是指为了达到最快速度,接受某些缺点。

一旦贵公司的工程师和技术人员成百上千,适用于团队层面的一切满足不了新的需要。生产环境处于一片混乱时,必须找到某种方法,那样那20个、50个或100个团队才能互相帮助。

敏捷、Scrum和开发运维(DevOps)等方法可以使某个特定项目保持顺利开展,并从概念环节进入到交付环节,但它们无法使众多团队的工作保持协调性。

当然,为某个平台或应用软件开发一套连贯的设计是个基本问题,组织管理项目以实施这种设计也是个基本问题。但无论你最初做得有多好,后来都需要调整。

那些团队中的每一个都是为了实现某些目标而设立的。也许它们有各自的盈亏(P&L),或目标和关键结果(OKR,谷歌就采用了著名的OKR,受英特尔使用OKR的启发)。但在现代平台中,构成整体的几乎所有服务都将相互使用。


Fireside Chat: DevOps at Amazon with Ken Exner, GM of AWS Developer Tools - AWS Online Tech Talks

Here is the link.

I like to plan up to 100 hours to study AWS related technology in short future. I really like Amazon AWS and like to learn more from those tech talks.




Friday, August 16, 2019

How to fail and look good doing it

Here is the link.

Plan for it

Define "success" (and "done", and...)
Define processes for obvious bad outcomes
Don't bother thinking of every edge case
Define a process for dealing with new cases
Decide how much failures is acceptable
Create only the bare minimum process


Step 3: Deal with it (the hard part)
Do not blame or be angry
Gather your team - discuss fixes as a group
Make a decision and delegate immediately
Communicate; take personal responsibility
Allow stakeholders to express anger to you
Do not allow the business to apportion blame* or dwell on the situation


Actionable Item


It is such a good topic. I should write one by myself as well.


301. Remove Invalid Parentheses

August 16, 2019


Introduction


It is one of algorithms I work on to try to learn 10 solutions. Today I spent some time to share those submissions first, and later I will review how to make improvements. 


My sharings



C# DFS practice in 2018, here is the link.
C# study code and apply TED principle practice back in 2017, here is the link.
C# BFS algorithm practice in 2017, here is the link.
C# BFS practice back in 2017, here is the link.






Minimum Viable Design LeanAgile system & software architecture - Matt Walters

Here is the video I like to spend time to watch.

About this presentation

Visual design
software architecture
Examples and approaches
Languages/ platforms/protocols
Specifics of just about anything


AWS re:Invent 2018: [NEW LAUNCH!] Deep Dive on Amazon RDS on VMware (DAT375)

Here is the link.

I like to watch the video and learn something from the presentation.


Neha Narkhede, CTO, Confluent: Apache Kafka Streaming Platform Explained—SpringOne Platform 2018

Here is the link.




Missing Siva Corp Inc.

August 16, 2019

Introduction


It is my personal finance research. I do have to keep myself learn and one of challenges is to open to my past history as a software programmer. I worked for a startup company back in Florida from 2006 to 2007 very short time 13 months, and this Sunday I will meet a friend Tracy who also worked for Siva over three years.

Missing Siva Corp Inc


I also like to write a nice story about Siva corporation Inc. I did have chance to meet the founder of the company back in 2001, and then I had chance to meet a few founders and friends as well. One of friends also helped me to get back to F1 student visa back in 2001 - a promissory note, and 4500 miles road trip from Florida to Vancouver Canada in 2010.

It is hard to measure 18 years in my life span. What does that really means for us to grow as a person, and professionals.

One of reasons I like to write on this topic is about my lunch with Amazon onsite interview, I was asked if I had experience to work on Java. I did share the experience I had to work on Java more than 13 months on this startup Siva, and then public company Par Tech inc.




Julia Grace - Senior Director of Engineering at Slack

Here is the link.


Kafka Summit Panel | Microsoft, Slack, Confluent, University of Cambridge (SF 2018)

Here is the link.


Neha Narkhede leads a panel discussion at Kafka Summit SF 2018 with Kevin Scott (CTO, Microsoft), Julia Grace (Head of Infrastructure Engineering, Slack), Martin Kleppman (Researcher, U. of Cambridge), Jay Kreps (co-founder and CEO, Confluent), and Neha Narkhede (co-founder and CTO at Confluent). Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

Lisa Guo, Hui Ding Keynote PyCon 2017

Here is the video I like to watch.

Ding Hui's profile is here.

Python is simple and clean, and favors pragmatism

1. Scope the problem, AKA, do simple things first.
2.

Scaling python to support user and feature growth

Python efficiency strategy

1. Build extensive tools to profile and understand perf bottleneck
2. Moving stable, critical components to C/C++, e.g., memcached access
3. Cythonization
4. Sync? New python runtime?

Ding Hui's profile is here.

Head of Instagram infrastructure team, overseeing Instagram's core infrastructure including Django/Python platform, social graph and Cassandra key-value storage systems, video infrastructure and realtime infrastructure. Responsible for driving reliability, scalability, efficiency and quality of backend infrastructure and web platform. Built out the team from 2 engineers to 70+ engineers across multiple teams. Proud of building a strong engineering culture within the team, and cultivated broad cross-functional collaborations within Facebook. Proven track record of successful delivery of multiple large scale projects. My team also plays a key role in delivering major products at scale, together with our product engineering teams, e.g., Instagram Direct, Stories, Live, IGTV.

Instagram runs the world's largest python fleet in production and we pride ourselves in the way we drive scale, efficiency and technical innovation with Python.
Instagram is also the world's top 3 user of Apache Cassandra, open source key-value storage system. We have been leading some of the 10x changes to Cassandra performance.





California meet up

August 16, 2019

Introduction


It is important for me to stay connected and also I should take initiative to invite friends to get together. I start the coding blog and also I like to learn to be a good role model for younger friends, stay motivated and bring best out of life as a friend.

My meetup 


I need to rent a car in San Francisco airport for my trip to Facebook, and I also need to meet a few of friends dinner on August 18, 2019. I set up on August 14, 2019.

All of my friends are back in Florida from 1998 to 2001. I am the elder one, 8 years old than most of them. It is so challenging to learn that I will meet those people again, some of them we used to attend events together, my mom and I as a team, and a few young couples back from 1998 to 2001.

Life is tough and career is also the tough thing to manage. But I understand that most tough things are to be parents, raise kids healthy and well-educated, good manner as well.

One of friends used to work for same company Siva back from 2001 to 2003, and I joined in 2006 and left 2007. Her linkedin profile is here, now she works for Google.

I also had a friend who works for Apple and wife works for Mercedes in California as well. They visited me in the city of Vancouver in 2017, and I did surprise them since they are managers and scientist. Compared to them, I understood that life is tough ahead for me as a single and also as a professional, Canadian, Chinese.

We meet again, I will meet Tracy again after 18  years and her husband. Tracy and I both worked for a startup company called Siva, and we both knew the owners and then we went to the same party back in 2001.

My meetup invitation


I like to show that I take some time off to plan the event, after carefully consideration, I believe that I should spend time with friends, and learn from their experience. All of them accepted my invitation.

I like to start something smaller, and also celebrate those time we spent together.






Vacation day - day five

August 16, 2019

Introduction


It is my fifth vacation day. I am staying in the city of Vancouver, and put together a plan for today and also the travel to California this Sunday.


My schedule








Wednesday, August 14, 2019

Case study: design youtube or netflix

August 14, 2019

Introduction


It is not easy to work on system design called design youtube.com. How to prepare in 30 minutes on this topic? I like to study the lecture note and then write down my thought process.

My notes


Now it is 5:44 PM. I like to go out to play tennis in less than 30 minutes. So I will spend next 30 minutes to take notes.

First, I like to write down the terms I read in the lecture note I should look into later.

video compression and replication
bandwidth 10MB/min
upload:view ratio of 1:200
1TB/s outgoing bandwidth
300GB/min (5GB/sec) bandwidth estimates
Storage estimates: 1500 GB/min (25 GB/sec)
46K/ 200 => 230 videos/sec

High level design

1. processing queue - uploaded video will be pushed to a processing queue to be de-queued later for encoding, thumbnail generation, and storage.
2. encoder - to encode each uploaded video into multiple formats
3. thumbnail generator:
4. Video and thumbnail storage:  distributed file storage
5. user database:
6. video metadata storage:


How to manage read traffic?

segregate read traffic from write traffic

Multiple copies of each video, we can distribute our read traffic on different servers.

For metadata, we can have master-slave configurations where writes will go to master first and then gets applied at all the salves.

Stalesness in data -  it might be acceptable

Introduction to Google BigTable: Best Uses, Design, and Demo

Here is the link.

Bigtable is an internal Google database system that’s so revolutionary that it kickstarted the NoSQL industry. Google went on to use Bigtable to power many of its other core services, such as Gmail and Google Maps. Finally, in 2015, it made Cloud Bigtable available as a service that its customers could use for their own applications. In this course, you will learn which of your applications could make use of Bigtable and how to take advantage of its high performance.


What is bigtable?
Internal Google database that kickstarted the NoSQL industry
Web indexes behind search engine took too long to build
Needed real-time access to petabytes of data
2006 research paper describing Bigtable
Led to HBase, Cassandra, and other NoSQL databases
Bigtable powers Gmail, Google Maps, and other services



Best uses for Bigtable
Design
. Architecture and storage model
. Schema design
. Cluster configuration
. Access control
Using bigtable

Learning objectives

Identify the best use cases for Bigtable
Describe bigtable's architecture and storage model
Optimize query performance through good schema design
Configure and monitor a Bigtable cluster




Bigtable

Here is the wiki article I plan to read.


My vacation day schedule - day 3

August 14, 2019

Introduction


It is my vacation day, the third day to take a vacation. Now it is 11:43 AM. I need to make a plan for my schedule. I like to add a few walks around the neighborhood, and also some sports activity before the dark.


My schedule


It is important for me to add some breaks to walk around the neighborhood, so that I can take enough break and study very well in-between.


System design interviews: A step by step guide

August 14, 2019

Introduction



It is not an easy task to work on called system design. I am still trying to learn the importance how to stay organized, well-structured in system design interview.

My understanding


I spent last two weeks to learn the basics, and I read the book called design large data intensive application, I watched a few videos from Facebook, Instagram. I find out that it is not too difficult for me to learn system design. I need to invest time and build good habit to study it, talk about, and blog about it.


A step by step guide

step 1: requirements clarifications
step 2: system interface definition
step 3: back-of-the-envelope estimation
step 4: define data model
step 5: high-level design
step 6: detailed design
step 7: identifying and resolving bottlenecks




Case study: Design ticketmaster

August 14, 2019

Introduction

It is a good idea for me to learn how to write down the most important things I learn from lecture learning. The system design is called Design a ticketmaster. It takes a lot of time to put together the detail about the design, the database table, columns, and then relationship, and all other details. I like to write a short case study how to manage the design in 45 minutes time range.

Case study

I am new to the system design. I started from 2016 Amazon onsite interview, and then I started to learn how to work on better.

1. Requirement - gather requirement

Functional requirement:
ticket Booking service
client purchase API
...

Non-Functional requirement:
highly concurrent; Same seat may have multiple booking request.
ticket booking, financial transactions, ACID compliant.

Design consideration

1. To simplify, no user authentication
2. No partial ticket order
...

Capacity estimation
Traffic estimation:
3 billion page view per month, 10 million tickets a month

Storage estimates:

500 cities
10 cinemas each city
2000 seats each cinema
two shows every day

Seat booking needs 50 bytes (IDs, NumberOfSeats, ShowID, MovieID, ...

to store all the data about all shows of all cinemas of all cities for a day:
500 cities * 10 cinemas * 2000 seats * 2 shows * (50 + 50) bytes = 2 GB/ day

Five year data, 3.6TB

5. System APIs

SearchMovies(keyword, city, ...
return: (JSON)

6. Database design

A few observations about the data we are going to store:
1. Each city can have multiple cinemas
2. Each cinema will have multiple halls
3. Each movie will have many shows and each show will have multiple bookings
4. A user can have multiple bookings

Tables:
Movie
Show
booking
User
Cinema
Cinema_hall
Show_seat
Payment
City
Cinema_Seat

7. High level design

clients -> load balancer -> multiple web servers -> Application servers -> cache servers and database

8. Detail component design

It is so interesting to watch the slide show how the booking is designed, and how to reserve the seat and then allow user to book, make payment.

Two daemon services:
1. One to keep track of all active reservations
2. One to remove any expired reservations from the system

It is so interesting to learn that data structure is chosen for ActiveReservationService

9. Concurrency

NO two users are able to book same seat

'Serializable' is the highest isolation level and guarantees safety from Dirty, Nonrepeatable, and Phantoms reads.

10 Fault tolerance

Service crash, read from database

11. Data partitioning

MovieID or showID,

Two service partitioning

Consistent hashing based on ShowID


Simona Halep Documentary - The Road To Roland Garros

I like to watch the video again and then I will write two blogs borrowing ideas from the video.

First two minutes

I like the court, tournament, and friend here, feel at home. - She talked about her experience.
The coach gave some comment.

Take day by day, match by match; just focus, just get motivated. I do not want to expect anything.

Coach: Play smart. ....

Simona's style - subtitle

Give one minute video first.

Talk about her strength, belief

Coach: express his opinion

very well coached from junior, well-developed. Even though she is 5 foot 5. She is smart on the court. ..

Mardrid 2018 quarterfinal - video - performance, what the performance...
Coach gave feedback -> interview -> think about match, next will be better. It is just an tournament. I will have a lot ahead.

I am not stressing about points. Busy schedule. THERE IS NO PRESSURE.

The competition

Coach: Every one is so close to the top. Rivalry, ...

WTA great place in the moment.

7:11/ 15:45
Rome 2017 Final

Stay healthy
8:30/ 15:45

Three boys with me all the time. Vibes, a lot of energy, they are young. They believe in me.

10:30/ 15:45
10:26

Tuesday, August 13, 2019

Case study: Designing Facebook messenger

August 13, 2019

Introduction


It is my favorite topic called designing Facebook messenger. I like to write a short case study so that I can track how good I am in terms of system design, my curiosity level, and my determination to push myself hard to learn beyond algorithm and data structure problem solving.

Case study

I like to study Grokking system design lecture notes first, and then write down my study notes.

I like to write down things used in the system design.

Messages handling
Pull model
Push model

How to maintain an open connection with the server?

HTTP Long Polling or WebSockets

The long polling request can timeout or can receive a disconnect from the server, in that case, the client has to open a new request.

How many chat servers we need?

Plan for 500 million connections at any time, a server handles 50K concurrent connection at any time, we would need 10K such servers.

How do you know which server holds the connection to which user?

Load balancer, map each UserID to a server to redirect the request.



Time to warm up algorithm and data structure problem solving

August 13, 2019

Introduction


It is time for me to count down hours. I only have less than 24 hours to study, and I like to plan to warm up my algorithm and data structure problem solving.

A few drills to warm up 


I like to warm up a few hours. I like to have a few drills. One drill is to go over all algorithms I practice in 2019. Another drill is to go over algorithms by categories.

I like to use this github page to warm up my algorithms.

Here is the page I like to warm up algorithms using categories.




Case study: Design typeahead suggestion

August 13, 2019

Introduction


It is my system design study and research. I like to go over this design called design typehead suggestion in next 2 - 3 hours. I like to take some notes as well.

Case study

I like to write down some thinking process to help myself to understand better about system design.

Typeahead suggestion - what is meaning of typeahead suggestion? It is like giving out hint when you types the word. In other words, it tries to predict the query based on the characters the user has entered and gives a list of suggestions to complete the query.

Functional requirement: As the user types in their query, the service should suggest 10 terms starting with whatever the user has typed.

Non-function requirements: The suggestion should appear in real-time. Within 200ms.

Basic system design and algorithm

The service should suggest next terms that will match the given prefix.

Data structure - Trie
How to find top suggestion? given prefix, how can we find the top 10 terms for the given prefix?

One simple solution can be to store the count of searches that terminated at each node.

How much time will it take to traverse its sub-tree?
How to store top suggestions with each node?
Let me think about it, and also understand the ideas in the ...
We can store top 10 suggestions at each node that we can return to the user.
We can optimize our storage by storing only references of the terminal nodes rather than storing the entire phrase.

How would we build this trie?

Build our trie bottom up - post order traversal
Each parent node will recursively call all the child nodes to calculate their top suggestions and their counts. Parent node will combine top suggestions from all of their children to determine their top suggestions.

How to update the trie?

We can have a Map-Reduce (MR) set-up to process all the logging data periodically say every hour.







Reading list from system design

August 13, 2019

Introduction


It is my system design research. Now it is 5:27 PM. I need to get ideas what to read before I start to warm up algorithm and data structure.

Reading list 


Here are the reading list I like to go over next 28 hours.

Here are some useful links for further reading:
1. Dynamo - Highly Available Key-value Store
2. Kafka - A Distributed Messaging System for Log Processing
3. Consistent Hashing - Original paper
4. Paxos - Protocol for distributed consensus
5. Concurrency Controls - Optimistic methods for concurrency controls
6. Gossip protocol - For failure detection and more.
7. Chubby - Lock service for loosely-coupled distributed systems
8. ZooKeeper - Wait-free coordination for Internet-scale systems
9. MapReduce - Simplified Data Processing on Large Clusters
10. Hadoop - A Distributed File System

Case study: Design Twitter search

August 13, 2019

Introduction


It is almost 5:00 PM. I have to push myself to go out to play one to two hours tennis, so I can come back to home office to continue to study for system design. My next study topic is called Design Twitter search.


Case study 

I like to use 100 sentences to summarize what I have learned through last 14 minutes reading.

1. Requirements and goals of the system -

Assume that Twitter has 1.5 billion total user with 800 million daily active users.
On average Twitter gets 400 million tweets every day.
The average size of a tweet is 300 bytes.
there will be 500M searches every days.
Search query will consist of multiple words combined with AND/OR.

2. Capacity estimation and constraints

Storage capacity:
  new tweets     average tweet 300 bytes
  400M           * 300                                      => 120GB/ day

Total storage per second:
            120GB/24hours/3600 sec ~= 1.38MB/second

3. System APIs

SOAP or RESP APIs - I need to learn how to define the API

Parameters
Return:

4. High level design

Clients
Application server
Index server
Storage server

Lesson learned from high level design (8/13/2019 5:24PM)

I like to write a statement here. High level design is simple, index server is included. But later on, I should estimate how many index server should be included in the architecture design, it is around

Total memory of index: 21 TB. Assuming a high-end server has 144GB of memory, we would need 152 such servers to hold our index.



Case study: Designing Twitter

August 13, 2019

Introduction


It is a good idea to study grokking system interview case - design Twitter. What I like to do is to go over the content quickly, less than one hour, and take as many notes as I can.

Design Twitter


Now it is 4:00 PM. I like to spend time from 4:00 PM to 5:00 PM on this topic.


Push technology

Here is wiki article called Push technologies.

Content

1. General use
2. Examples
 2.1 Webpush
 2.2 HTTP server push
2.3 Pushlet
2.4 Long polling
2.5 Flash XML Socket relays
2.6 Reliable Group Data Delivery (RGDD)
2.7 Push notification


Introduction - Web Development

Here is link to access Udacity course - web development.


Lock Contention - Web Development

Here is the link.





Memcached vs. Redis?

Here is the post from stackoverflow.com. I like to read the content and learn something here.


Memcached, Locking and Race Conditions

Here is the link.


Growing Reddit - Web Development

Here is the link.

Growing Reddit

-fake content

url    _______________
title  _______________
user  _______________

  submit

- fake users
- set the tone
- feel alive

- no comments
- no categorization
- no emails
- no censoring










Caching - web development

Here is the link.

Cache is hashtable

Caching

cache hit
cache miss


Caching Techniques - Web Development

Here is the link.

Caching techniques

Approach                     DB read/ pageview   DB read/ Submit   Bugs
_________________________________________________________
no caching                          every                        none
naive caching                    cache miss                 none                  yes
clear cache                        cache miss                 none
refresh cache               


Cache updating

complex inserts + speed
    vs
DB reads

the more accurate the cache, the more complex the code.






Pylons project

Here is the wiki page.

Pyramid is a minimalistic, platform-independent web framework. It is persistence agnostic and is integrated both with SQL databases via SQLAlchemy and with the Zope Object Database, as well as other NoSQLdatabases, such as CouchDB.[3]
Pyramid allows developers to define routes using regular expressions that map to objects. Like its fellow framework Zope, Pyramid also allows hierarchical object traversal, where each part of a URL is an object containing other objects, in a way that is similar to folders in a filesystem.[8]

Pylons Web Framework[edit]

Pylons Framework
Pylonsfw.png
Developer(s)Ben Bangert, James Gardner
Initial releaseSeptember 2005; 13 years ago[9]
Stable release
1.0.2[10] / July 21, 2015; 4 years ago
Written inPython
Operating systemCross-platform
TypeWeb application framework
LicenseBSD license
Websitepylonsproject.org/about-pylons-framework.html
Pylons Framework is an open-source Web application framework written in Python. It makes extensive use of the Web Server Gateway Interface standard to promote reusability and to separate functionality into distinct modules.[11] It is strongly influenced by Ruby on Rails: two of its main components, Routes and WebHelpers, are Python reimplementations of Rails features.


Web.py framework

Here is the page.


Scaling - Web Development

Here is the link.


Improving Memcache - Web Development

Here is the link.


Zookeeper - Web Development

Here is the link.

Zookeeper - locking, dynamic configuration,

replicated nodes in zookeeper




Chat with my young sister 40 minutes

August 13, 2019

Introduction


It is my personal finance research. I like to learn how to attract money, and also fully use my limited time to catch up learning, and also meet people I like. One thing I can do is to chat with my young sister, and share my viewpoints.

Chat with my young sister


It is hard for me to live with so many siblings. I have to learn how to focus on my own projects. I will copy and paste my conversation here from wechat.

1. I make a rule not to join any sibling wechat group. Since there is time difference, I will suffer sleep problem since I cannot control myself to read wechat;
2. I make decisions based on facts, the language they use, the attitude, and I tell her that I can not tolerate any abusive language etc. No is my policy, no more big family wechat group.
3. I am my own family, as a single person, I even do not own my own home in Vancouver near 10 years. I need to take care of my own business



W-8 Form Definition

I had to close Ameritrade.com stock account in United States, since I am a Canadian citizen and live in Canada. Today I chatted with my friend in SJTU 1988 graduate wechat group, and he told me that he opened the account in Ameritrade.com, and he uses W8 form.


My schedule - second day vacation

August 13, 2019

Introduction


It is my personal finance research. I like to learn how to plan a day, and also give myself chance to learn something important, helpful for me to perform on algorithm and data structure problem solving in two days.

My schedule


Now it is 11:51 AM. I already spent more than 5 hours since I woke up around 6:00 PM.

Do not get too busy. Take some time off to take multiple walks around neighborhood. Show up on tennis court, and spend 1 - 2 hours.


Amazon macie

August 13, 2019

Introduction


It is my second day vacation in the city of Vancouver. I like to enjoy the day, take some time to walk around neighborhood, play some tennis. Most of important is to learn something related to Amazon AWS technology. I am glad to spend 10  - 30 minutes to learn about Amazon macie.

Amazon macie


Here is the link.