System design | Decouple 10 principles | In Chinese

March 25, 2021

I just could not believe that I can learn so quickly in Chinese. Those articles are well-written on system design. I just spent a few minutes on wechat investment group to introduce Jet Ding and his Chinese channel, and then Google recommendation prompted me to watch more. I decided to start warmup study for system design. 

10 principles - decoupling 

Here is the article. 

第一步，解耦现有模块

将现有耦合在一起的模块进行重新的设计，设计成可以独立部署的多个模块，使用微服务框架很容易做到，成熟的示例代码都特别多，这里不再多讲。下面是我的微服务实现的一个架构设计图。

第二步，抽取公共模块

架构设计原则之一就是反向依赖，只从上往下依赖，所以，我们将公共的重复功能的模块抽取出来。必须强调一点的是，公共模块必须足够的功能单一，不能有其他业务的逻辑判断在里面。在整个模块依赖关系里，应该是一棵树状结构的关系图，而不是一个网状的关系图。

1）做好代码控制

笔者之前就碰到过这种问题，模块划分完了，当需求变更的时候，研发人员根本不管是不是公共模块，只要能快速完成任务，哪里改的快就在哪里改。因此，这个需要内部要做好代码的权限管理，不应该开放所有的提交代码的权限给所有的人。后来我就将公共模块的合并代码的权限收回了，合并代码需要先提交申请，代码review过才能合并代码。这就保证了公共模块代码的功能单一。

2）做好版本管理

公共模块被多个模块模块使用，任何代码的修改都可能会导致到正在使用的模块无法使用。这个就需要做好各个模块的版本管理，我是使用maven进行版本管理的，定义一个总的父pom项目来进行各个模块的版本管理，任何被其他模块使用的开发包都要在父pom里进行版本管理。当新的需求来了以后，需要对公共模块进行修改时，要更新模块的版本号，同时更新父pom的版本号，需要使用公共模块新功能的模块就修改父pom的版本号，不需要使用公共模块新功能的模块就不用修改父pom的版本号，这样公共模块的新老版本都能使用，即使出现问题，也只会影响到使用新版本的模块。

第三步，解耦迭代需求

现在的代码迭代速度快，同时会面对多个需求，有的需求紧急，有的需求不紧急，而且紧急程度可能随时会调整，如果将所有的需求都放在一个分支，当只想上线其中几个需求的时候发现无法将不上线需求的代码拆分出来，是不是很尴尬，即使能拆分出来，代码修改过以后又要重新进行部署测试，很费时费力，所以要针对不同的需求重新建立研发分支，这样就将不同需求的分支解耦，保证想上哪个就上哪个，需要上多个需求的就将分支合并上线。

第四步，配置解耦

为每个模块每个环境配置一个配置文件，这样就可以把不同的环境的配置解耦，不用每次上线都更新一次。但是如果需要修改数据库配置，还是需要重新部署重启应用才能解决。使用微服务的配置中心就能解决这个问题了，比如使用ZooKeeper作为SpringCloud的配置中心，修改ZooKeeper中的节点数据就可以实时更新配置并生效。

第五步，权限解耦

当采用微服务架构把原来的系统拆分成多个系统以后，你会发现原来简单的问题，现在变的复杂了，比如功能的权限控制，原来是跟业务代码放到一起，现在如果每个业务模块都有功能权限的代码，将是一件非常麻烦的事情。那么解决办法就是将权限功能迁移出来，恰巧使用SpringCloudGateway就能完成这件事情，SpringCloudGateway能够进行负载均衡，各种路由拦截，只要将原来的权限控制代码迁移到Gateway里实现以下就可以了，权限配置管理界面和代码逻辑都不用变。如果是API接口呢，就需要将安全验证等功能放在Gateway里实现就好了。

第六步，流量解耦

当你的系统访问量越来越大的时候，你会发现每次升级都是一件非常麻烦的事情，领导会跟你说这个功能忙时不能停机影响用户使用呀，只能半夜升级呀，多么痛快的事情啊。有的时候运营人员也会发现，怎么我的后台访问怎么这么慢？问题出在哪里呢？问题就出在，所有的模块都用了一个Gateway，多端同时使用了相同的流量入口，当在举行大促时，并发量非常高，带宽占用非常大，那么其他的功能也会跟着慢下来。

不能在举行大促时发券时，我线下支付一直支付不了，这是非常严重的事故了，客服电话会被打爆了。所以，必须要对流量进行拆分，各个端的流量不能相互影响，比如APP端、微信端、运营后台和商户后台等都要分配独立的Gateway，并接入独立的带宽，对于流量大的端可以使用弹性带宽，对于运营后台和商户后台就比较小的固定的带宽即可。这样就大大降低了升级时的难度，是不是再上线时就没那么紧张了？

第七步，数据解耦

系统刚上线的时候，数据量不大，所有的模块感觉都挺好的，当时间一长，系统访问量非常大的时候会发现功能怎么都变慢了，怎么mysql的cpu经常100%。那么恭喜你，你中招了，你的数据需要解耦了。

首先要模块间数据解耦，将不同模块使用独立的数据库，保证各模块之间的数据不相互影响。

其次就是冷热数据解耦，同一个模块运行时间长了以后也会积累大量的数据，为了保证系统的性能的稳定，要减少因为数据量太大造成的性能降低，需要对历史数据进行定期的迁移，对于完整数据分析汇总就在其他的库中实现。

第八步，扩容解耦

一个好的架构设计是要有好的横向扩展的能力，在不需要修改代码只通过增加硬件的方式就能提高系统的性能。SpringCloud和Dubbo的注册中心天生就能够实现动态添加模块的节点，其他模块调用能够实时发现并请求到新的模块节点上。

第九步，部署解耦

互联网开发在于能够快速的试错，当一个新的版本上线时，经常是需要先让一部分用户进行测试一下，这就是传说中的灰度发布，同一个模块先部署升级几台服务器到新版本，重启完成后流量进来以后，就可以验证当前部署的这几台服务器有没有问题，就继续部署其他的节点，如果有问题马上回滚到上一个版本。使用SpringCloudGateway的WeighRouterFilter就能实现这个功能。

第十步，动静解耦

当同一个模块的瞬间有非常高并发的时候，对，就是说的秒杀，纯粹的流量解耦还是不够，因为不能让前面的流量冲击后面真正的下单的功能，这个时候就需要更细的流量解耦，要将静态文件的访问通通抛给CDN来解决，动态和静态之间是通过定时器来出发的，时间未到之前一直刷新的是静态的文件，当时间到了之后，生成新的js文件，告诉静态页面可以访问下单功能了。

总结

在模块划分时，要遵循“一个模块，一个功能”的原则，尽可能使模块达到功能内聚。

事实上，微服务架构短期来看，并没有很明显的好处，甚至短期内会影响系统的开发进度，因为高内聚，低耦合的系统对开发设计人员提出了更高的要求。高内聚，低耦合的好处体现在系统持续发展的过程中，高内聚，低耦合的系统具有更好的重用性，维护性，扩展性，可以更高效的完成系统的维护开发，持续的支持业务的发展，而不会成为业务发展的障碍。

System design | Andrew Wu | 後端工程師想要晉級到架構師

Here is the article. 

講師介紹：

談論各種軟體開發與設計的大小事，有 20 年的大型與雲端服務的開發經驗。

喜歡研究各種技術背後的原理與實作細節，期許自己做個優秀的系統架構師。

主題以: .NET / C# / OOP / Container / Microservices / Azure / Distributed System / Cloud Native / DevOps 為主軸，同時在部落格上也持續分享相關主題的一系列文章。期許能將這些實作經驗分享到社群。

• 任職 91APP, 首席架構師 Chief Architect
• 榮獲 Microsoft MVP 微軟最有價值專家 獎項,2016 ~ 2021
• 曾任 一宇數位科技 技術長
• 擔任資策會 雲端系列課程 Azure PaaS 講師
• 擔任多次 Microsoft Azure Cafe, TechDays, TechEd, DevOpsDays Taiwan, .NET Conf 等大型研討會講師

會開啟這系列文章，以及背後的動機，其實是來自這十幾年工作上面臨的挑戰，思考下來的結果。我踏入軟體開發這行，已經超過 20 年了 (Orz..), 經歷過幾年的工程師，技術經理，技術總監，一路到技術長，典型的工程師到管理職的過程。不過我希望台灣也能發展出真正的軟體業，能真正做點不一樣的服務出來，於是最近的幾年，我開始有計劃的轉職為架構師，重新回到專注技術的領域。

這轉換的過程中，越來越體認到，只有把 code 寫好是不夠的，在擴大架構團隊的過程中，我面試了很多人，也談過了很多人選，其實其中不乏實作能力比我強的高手啊! 但是談完後回到團隊的定位，總覺得缺少了些什麼，軟體開發不應該 “只有” 把需求做出來的過程，這過程只能算是軟體 “代工”，而不是軟體 “研發” 啊。在尋找這樣特質的人才過程中，我開始思考這樣的人除了 “招募” 之外，該怎麼 “培養” ? 所以才開始有了這念頭，想要好好地把我自己過去學習的經驗，用有系統的方式告訴大家，該如何培養自己在軟體產業長久發展下去的能力。

要在這行長久發展，你得要有能力跨過 40 歲這關卡。這門檻不會只有技術能力而已，一定還有其他更有價值的能力搭配才是。”高齡工程師” 這類話題最近幾年很熱門啊，只要有留意相關社群或是媒體，不難發現每隔一段時間都會有高齡工程師去留的問題被熱烈討論。我都當過 CTO 這樣的角色了 (雖然公司規模不算大)，不過我想我開始能想像老闆的角度會考量什麼事情了。 40+ 的工程師，你除了把 code 寫得更快，工具更熟練，框架學得越多之外，你必須還要有一些年輕人取代不了的能力才行，否則老闆用 “scale out” (用在這裡好像怪怪的 XDD) 的策略，如果多找三個年輕人可能就能取代你了，那它還有可能用三倍以上的薪資聘用你嗎? 這些能力是需要漸漸培養累積的，等你 40+ 才開始做是來不及的，你必須在 30+ 摸清楚產業跟你自己職涯方向後，就要開始有計劃的學習累積才行。

 

舉個例子來說明好了。公司內部我們負責的一套系統，需要深入的了解 Rabbit MQ 設定跟管理的細節，我就以 Rabbit MQ 來當作例子，說明每個 Level 應該有的行為表現:

LV2:
我會 Rabbit MQ, 代表我知道如何安裝、設定、管理 Rabbit MQ 服務；我也知道如何透過 C# 將 Message 送入 MQ, 也知道如何將 Message 取出處理。 基本上就是知道 Rabbit MQ 是拿來幹嘛用的，同時有能力正確使用他的 SDK，基本上就具備 LV2 的要求了。

LV3:
我了解什麼情況下需要用 Message Queue 來解決非同步任務執行的問題；我知道 Rabbit MQ 的特色: Exchange / Queue 的組合與應用，何時該用 Fanout, 何時該用 Topic。基本上你對 Rabbit MQ 已經有一定的了解，如果你的團隊來了個新人，你除了自己會用之外，也有能力開始跟新人說明介紹 Rabbit MQ 的概念與操作。

LV4:
Rabbit MQ 適合用來處理 “生產者與消費者”問題，非同步任務產生後可以交給 MQ 處理，不需等待回應可以繼續後面的動作，而另一端可由 Worker 負責消化任務。MQ 後端可以善用平行處理的技巧，提高整體的處理能力，也能同時用多個 Worker 分攤處理的任務。每個 Worker 都能夠以最佳效能運作，處理完畢後才領取下一個任務，輸出穩定，不會受到瞬間大量所影響，這種情況下能達到最佳的處理效能。

How I got Google, Facebook, AMEX internships as a freshman - 10 software engineering interview tips

March 24, 2021

Here is the link. 

#facebookintern​ #googleintern​ #softwareengineering​ It's almost recruiting szn!! Here are some things I've learned after landing internships at Google, Facebook and AMEX before my sophomore year! Recruiting is tough:/// don't be discouraged by rejections — we've all been there — but I promise, if you work hard, it'll definitely pay off:) Good luck everyone and let me know if you have questions in the comments below yeeet Covered a lotta ground in this video so here are some time stamps:) 0:58​ - Recruiting Process Debrief 2:54​ - When to Apply 4:44​ - How to Study 5:20​ - Question Topics 5:48​ - Useful Practice Resources 7:11​ - Common Pitfalls when Studying 8:43​ - Common Questions 10:31​ - Difficulty 10:58​ - Free Resources 11:35​ - How to Interview (coding + talking at the same time rip - multitasking is hard af) 12:37​ - Questions to Prepare 15:35​ - How to Practice Interviewing 16:32​ - How to Create a Solid, Standout Resume 18:00​ - How to Get Responses 18:38​ - Referrals and Networking (for any women in CS - apply for Rewriting the Code https://rewritingthecode.org/​ if you want access to lots of referrals - or lowkey just ask around haha) 21:08​ - Summary! List of Free Resources: For practicing interview questions - LeetCode - https://leetcode.com/​ HackerRank - https://www.hackerrank.com/​ (problem solving) For refreshing concepts - Geeks for Geeks - https://www.geeksforgeeks.org/​ (super helpful for data structures, binary trees) For creating products - Ray Wenderlich - https://www.raywenderlich.com/​ (mobile app dev, web app dev, but tbh u can just google other tutorials as well ^ these might be a little outdated) Popular Questions: When is it better to use Python/Java? When and why would you use recursive implementation over an iterative and vise versa? What's the difference between a class and an object? How does hashing work? What's a hashing function? What's the functionality of a cache? (feel free to Google these answers) M U S I C cliche - mxmtoon please done - mxmtoon falling for u - mxmtoon, Peachy! my feelings are fatal - mxmtoon (maia, I'm ur biggest fan:))

Questrade.com TFSA: My review | My position on GRAY biotech stock | Pump and dump?

March 24, 2021

Actionable Items

1. I like to invest on GRAY stock, but my timing could be better. 
2. It is challenge for me to understand how GRAY stock will perform. 
3. I just think that there is the possibility for GRAY stock to go up and double the value in next 12 months. 
4. I think that it is important to think about value of stock, not just market price right now. 
5. I will think about more carefully about NOK position 3000 shares. 

Follow up 

June 16, 2023

1. Why I did not choose to invest on OXY stock?
2. Why no one in wechat group told me that I should invest something else? Only one of them told me that. 
3. Why I did not choose 10 stocks first, and then worked on my criteria to make selections?
4. Why I did not choose to study one of professional website and then start to have a long term investment on biggest company instead?
5. This is so wrong to go for pump and dump investment. 

The reality of working for Facebook - BBC

 March 24, 2021

Here is the link. 

March 24, 2021 | Market update | Ameritrade.com IRA account | My trades

 

Microservice architecture:

March 24, 2021

Here is the link. 

C# | First practice | Union Find algorithm | Timeout on 8/12 test case | March 24, 2021

March 24, 2021

Here is the link.

C# | First practice | Union Find algorithm | Timeout on 8/12 test case | March 24, 2021

March 24, 2021
Introduction
It is the second algorithm in Amazon mock online code screen. I like to try to solve the algorithm using union find algorithm, brute force every edge in connection argument, and then check how many disjoint set if the edge is removed.

Timeout challenge
My idea failed to pass test case 8/ 12 cases. One of ideas is to prune the algorithm to expedite the search.

public class Solution {
     /// <summary>
        /// March 24, 2021
        /// The idea is to use union find algorithm, and go over each edge, determine if 
        /// the edge is removed, then count of disjoint sets is one or not. 
        /// n - nodes in network, 1 to 10^5 
        /// </summary>
        /// <param name="n"></param>
        /// <param name="connections"></param>
        /// <returns></returns>
        public IList<IList<int>> CriticalConnections(int n, IList<IList<int>> connections)
        {
            if (n <= 0 || connections == null || connections.Count == 0)
            {
                return new List<IList<int>>(); 
            }

            var length = connections.Count;
            var paths = new List<IList<int>>(); 

            for (int i = 0; i < length; i++)
            {
                var count = unionFindDisjointSetCount(n, connections, i);
                if (count > 1)
                {
                    paths.Add(connections[i]); 
                }
            }

            return paths; 
        }

        /// <summary>
        /// Union find algorithm
        /// remove one edge - count disjoint sets 
        /// </summary>
        /// <returns></returns>
        private static int unionFindDisjointSetCount(int n, IList<IList<int>> connections, int removed)
        {
            var parents = new int[n];

            for (int i = 0; i < n; i++)
            {
                parents[i] = i; 
            }

            var length = connections.Count;
            for (int i = 0; i < length; i++)
            {
                if (i == removed)
                {
                    continue;
                }

                var edge = connections[i];
                var end1 = edge[0];
                var end2 = edge[1];
                var max = Math.Max(end1, end2);
                var min = Math.Min(end1, end2);

                parents[FindParents(parents, max)] = FindParents(parents, min);
            }

            // go over all the nodes again
            var rootNodeCount = 0; 
            for (int i = 0; i < n; i++)   // Do not mix length with n; should be n, not length
            {
                parents[i] = FindParents(parents, i);               
            }

            // check last moment 
            for (int i = 0; i < n; i++)
            {
                if (parents[i] == i)
                {
                    rootNodeCount++;
                }
            }

            return rootNodeCount; 
        }

        /// <summary>
        /// union find algorithm
        /// path compression 
        /// </summary>
        /// <param name="parents"></param>
        /// <param name="node"></param>
        /// <returns></returns>
        private static int FindParents(int[] parents, int node)
        {
            if (parents[node] == node)
            {
                return node; 
            }

            parents[node] = FindParents(parents, parents[node]);
            return parents[node];
        }
}

public class Solution {
     /// <summary>
        /// March 24, 2021
        /// The idea is to use union find algorithm, and go over each edge, determine if 
        /// the edge is removed, then count of disjoint sets is one or not. 
        /// n - nodes in network, 1 to 10^5 
        /// </summary>
        /// <param name="n"></param>
        /// <param name="connections"></param>
        /// <returns></returns>
        public IList<IList<int>> CriticalConnections(int n, IList<IList<int>> connections)
        {
            if (n <= 0 || connections == null || connections.Count == 0)
            {
                return new List<IList<int>>(); 
            }

            var length = connections.Count;
            var paths = new List<IList<int>>(); 

            for (int i = 0; i < length; i++)
            {
                var count = unionFindDisjointSetCount(n, connections, i);
                if (count > 1)
                {
                    paths.Add(connections[i]); 
                }
            }

            return paths; 
        }

        /// <summary>
        /// Union find algorithm
        /// remove one edge - count disjoint sets 
        /// </summary>
        /// <returns></returns>
        private static int unionFindDisjointSetCount(int n, IList<IList<int>> connections, int removed)
        {
            var parents = new int[n];

            for (int i = 0; i < n; i++)
            {
                parents[i] = i; 
            }

            var length = connections.Count;
            for (int i = 0; i < length; i++)
            {
                if (i == removed)
                {
                    continue;
                }

                var edge = connections[i];
                var end1 = edge[0];
                var end2 = edge[1];
                var max = Math.Max(end1, end2);
                var min = Math.Min(end1, end2);

                parents[FindParents(parents, max)] = FindParents(parents, min);
            }

            // go over all the nodes again
            var rootNodeCount = 0; 
            for (int i = 0; i < n; i++)   // Do not mix length with n; should be n, not length
            {
                parents[i] = FindParents(parents, i);               
            }

            // check last moment 
            for (int i = 0; i < n; i++)
            {
                if (parents[i] == i)
                {
                    rootNodeCount++;
                }
            }

            return rootNodeCount; 
        }

        /// <summary>
        /// union find algorithm
        /// path compression 
        /// </summary>
        /// <param name="parents"></param>
        /// <param name="node"></param>
        /// <returns></returns>
        private static int FindParents(int[] parents, int node)
        {
            if (parents[node] == node)
            {
                return node; 
            }

            parents[node] = FindParents(parents, parents[node]);
            return parents[node];
        }
}

Gracie Gold Interview on Mental Health On And Off The Ice

March 24, 2021

Here is the link. 

Gracie Gold, an olympic medal-winning figure skater, opens up about depression, isolation, and mental health during her storied career on the ice. In this JED Voices full interview, Gracie Gold talks candidly with Chamique Holdsclaw about getting help, how she approaches therapy, and why she is publicly speaking out about her experience with mental health and depression. The Jed Foundation (JED) is a nonprofit that exists to protect emotional health and prevent suicide for our nation's teens and young adults. To get help now Text "START" to 741-741 or call 1-800-273-TALK (8255) To get help or get involved visit: https://www.jedfoundation.org/

#IAm​ Michelle Kwan Story

March 24, 2021

Here is the link. 

CAPE's #IAm​ Campaign made possible by U.S. Bank celebrates AAPI role models throughout May, Asian American and Pacific Islander Heritage Month (AAPIHM). Watch the entire series at http://www.IAm-Campaign.com​.

Olympian Gracie Gold Gets Candid About Her Mental Health with Michelle Kwan | Thank You Notes

March 24, 2021

Here is the link. 

For my new series, Thank You Notes, I teamed up with Olay to challenge a group of women to surprise their role models — the women who inspired them to face anything — with a thank you note. In this episode: Michelle Kwan may be one of the best figure skaters of all time, but it’s her incredible resolve that made her such an inspiration to fellow skater Gracie Gold. In this video, Gracie tells Michelle: “You once estimated that you’ve fallen 131,000 times in your skating career. To some, that might seem like an insane amount of times to mess up, but it’s the way you always got up that inspired me.” In 2017, Gracie checked into a treatment facility to address her depression and eating disorder. While she was there, one of the many voices that reached out to offer her support was Michelle Kwan. “When you reached out to me,” Gracie tells Michelle, “It meant more than you willl ever know.” Watch the whole emotional exchange above.

Gracie Gold

 Grace Elizabeth Gold (born August 17, 1995), known as Gracie Gold,^[6] is an American figure skater. She is a 2014 Olympic team event bronze medalist, the 2014 NHK Trophy champion, the 2015 Trophée Éric Bompard champion, and a two-time U.S. national champion (2014, 2016). She is also a two-time World Team Trophy champion (2013, 2015).

On the junior level, she is the 2012 World Junior silver medalist, the 2011 JGP Estonia champion, and the 2012 U.S. junior national champion.

Gold is the first and only American woman to win an NHK Trophy title. She also holds the record for the highest short program score ever recorded by an American woman: 76.43, scored in the 2016 World Championships (historical record).

Olympic Figure Skater Gracie Gold On Comeback After Eating Disorder | TODAY

March 24, 2021

Here is the link. 

Before the 2018 Pyeongchang Olympics, figure skater Gracie Gold announced she was stepping away from the ice to deal with personal struggles. A year later, she talks to Savannah Guthrie about the pressure facing athletes and her fight to return to the rink. » Subscribe to TODAY: http://on.today.com/SubscribeToTODAY​ » Watch the latest from TODAY: http://bit.ly/LatestTODAY

How 2018 USA Olympian Karen Chen Became Figure Skating’s ‘Quiet Assassin’ | Mach | NBC News

 March 23, 2021

Here is the link.