Thursday, August 26, 2021

How to learn large distributed system very well by myself?

Aug. 26, 2021

Introduction

It is tough for me to learn large distributed system. I like to make changes, my goal is not to pass system design interview, and I like to learn and also enjoy the learning. 

Top quality learning content

I chose to learn Google cloud technology this year. I started to think more carefully, and I should read a lot of quality books on distributed system, and I should be able to understand the reasoning in a few hours if I choose to read the books. 

I decide to go over the following books. If I cannot understand the book, then I will find ways to figure out. 

  1. The datacenter as a computer | Designing warehouse - Scale machines
  2. Cassandra The definitive guide, Eben Hewitt
  3. Elasticsearch | The definitive guide | A distributed real-time search and analytics engine
  4. Hadoop Learn how to crunch big data to extract meaning from the data avalanche Beginner's guide
  5. Hadoop MapReduce cookbook
  6. HBase The definitive guide
  7. Programming Hive
  8. Kafka The definitive guide
  9. Learning Spark
  10. Scala cookbook

Book reading: 2.2 PLATFORM-LEVEL SOFTWARE | My 10 minutes reading time

 2.2 PLATFORM-LEVEL SOFTWARE

The basic software system image running in WSC server nodes isn’t much different than what one would expect on a regular enterprise server platform. Therefore we won’t go into detail on this level of the software stack.

Firmware, device drivers, or operating system modules in WSC servers can be simplified to a larger degree than in a general purpose enterprise server. Given the higher degree of homogeneity in the hardware configurations of WSC servers, we can streamline firmware and device driver development and testing since fewer combinations of devices will exist. In addition, a WSC server is deployed in a relatively well known environment, leading to possible optimizations for increased performance. For example, the majority of the networking connections from a WSC server will be to other machines within the same building, and incur lower packet losses than in long-distance internet connections. Thus, we can tune transport or messaging parameters (timeouts, window sizes, and so on) for higher communication efficiency.

Virtualization first became popular for server consolidation in enterprises but now is also popular in WSCs, especially for Infrastructure-as-a-Service (IaaS) cloud offerings [VMware]. A virtual machine provides a concise and portable interface to manage both the security and performance isolation of a customer’s application, and allows multiple guest operating systems to co-exist with limited additional complexity. The downside of VMs has always been performance, particularly for I/O-intensive workloads. In many cases today, those overheads are improving and the benefits of the VM model outweigh their costs. The simplicity of VM encapsulation also makes it easier to implement live migration (where a VM is moved to another server without needing to bring down the VM instance). This then allows the hardware or software infrastructure to be upgraded or repaired without impacting a user’s computation. Containers are an alternate popular abstraction that allow for isolation across multiple workloads on a single OS instance. Because each container shares the host OS kernel and associated binaries and libraries, they are more lightweight compared to VMs, smaller in size and much faster to start.

 


Book reading: 2.1 WAREHOUSE DATA CENTER SYSTEMS STACK | The Datacenter as a Computer | Designing Warehouse-Scale Machines | Aug. 26, 2021

  1. Platform-level software
  2. Cluster-level infrastructure
  3. Application-level software 

2.1 WAREHOUSE DATA CENTER SYSTEMS STACK

The applications that run on warehouse-scale computers (WSCs) dominate many system design

trade-off decisions. This chapter outlines some of the distinguishing characteristics of software

that runs in large internet services and the system software and tools needed for a complete computing

platform. Here are some terms used to describe the different software layers in a typical

WSC deployment.

• Platform-level software: The common firmware, kernel, operating system distribution,

and libraries expected to be present in all individual servers to abstract the hardware

of a single machine and provide a basic machine abstraction layer.

• Cluster-level infrastructure: The collection of distributed systems software that manages

resources and provides services at the cluster level. Ultimately, we consider these

services as an operating system for a data center. Examples are distributed file systems,

schedulers and remote procedure call (RPC) libraries, as well as programming models

that simplify the usage of resources at the scale of data centers, such as MapReduce

[DG08], Dryad [Isa+07], Hadoop [Hadoo], Sawzall [Pik+05], BigTable [Cha+06],

Dynamo [DeC+07], Dremel [Mel+10], Spanner [Cor+12], and Chubby [Bur06].

• Application-level software: Software that implements a specific service. It is often useful

to further divide application-level software into online services and offline computations,

since they tend to have different requirements. Examples of online services are

Google Search, Gmail, and Google Maps. Offline computations are typically used in

large-scale data analysis or as part of the pipeline that generates the data used in online

services, for example, building an index of the web or processing satellite images to

create map tiles for the online service.

• Monitoring and development software: Software that keeps track of system health and

availability by monitoring application performance, identifying system bottlenecks,

and measuring cluster health.

Apache Hadoop | My learning in 10 minutes

 

Modules

The project includes these modules:

  • Hadoop Common: The common utilities that support the other Hadoop modules.
  • Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
  • Hadoop YARN: A framework for job scheduling and cluster resource management.
  • Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

Who Uses Hadoop?

A wide variety of companies and organizations use Hadoop for both research and production. Users are encouraged to add themselves to the Hadoop PoweredBy wiki page.


Related projects

Other Hadoop-related projects at Apache include:

  • Ambari™: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner.
  • Avro™: A data serialization system.
  • Cassandra™: A scalable multi-master database with no single points of failure.
  • Chukwa™: A data collection system for managing large distributed systems.
  • HBase™: A scalable, distributed database that supports structured data storage for large tables.
  • Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.
  • Mahout™: A Scalable machine learning and data mining library.
  • Ozone™: A scalable, redundant, and distributed object store for Hadoop.
  • Pig™: A high-level data-flow language and execution framework for parallel computation.
  • Spark™: A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation.
  • Submarine: A unified AI platform which allows engineers and data scientists to run Machine Learning and Deep Learning workload in distributed cluster.
  • Tez™: A generalized data-flow programming framework, built on Hadoop YARN, which provides a powerful and flexible engine to execute an arbitrary DAG of tasks to process data for both batch and interactive use-cases. Tez is being adopted by Hive™, Pig™ and other frameworks in the Hadoop ecosystem, and also by other commercial software (e.g. ETL tools), to replace Hadoop™ MapReduce as the underlying execution engine.
  • ZooKeeper™: A high-performance coordination service for distributed applications.

I will learn large distributed system very well this summer ...

Aug. 26, 2021

Introduction

I was too busy to learn large distributed system from 2016 to 2021, and I had hard time to read a book called large intensive data application starting from 2018. Learned lessons from onsite interview with Facebook from 2018 to 2021, I decided this year to read a few books about Hadoop, I like to be able to understand a book if I choose to spend a few hours on large distributed system subject. It takes a lot of time to read a book very well. I made the decision, so I had great time to watch videos from Google I/O, and google cloud technology talks on youtube.com. 

I will learn large distributed system very well this summer ...

I watch a lot of videos on youtube.com, but I find out that it is better for me to read a few books about large distributed system first. I started to read a few papers related to bigTable, and then journey begins.

I will count how many videos I watch on youtube on this Google cloud technologies. 


Cloud Pub/Sub Overview - ep. 1

Aug. 26, 2021

Here is the link. 

In this first episode of Pub/Sub Made Easy, we help you get started by giving an overview of Cloud Pub/Sub. Learn how to ingest large amounts of data for analysis, simplify the development of event-driven microservices, and much more! Overview → https://goo.gle/31T8AEU Cloud Pub/Sub documentation → https://goo.gle/2BVmLhP Pub/Sub Made Easy playlist → https://goo.gle/2XQ3vwC Subscribe to the GCP Channel → https://goo.gle/GCP



Breakfast and learn: Real-Time Stream Analytics with Google Cloud Dataflow: Common Use Cases & Patterns (Cloud Next '18)

 Aug. 26, 2021

Here is the link.

Google Cloud Dataflow makes it easy to process and analyze real-time streaming data so that you can derive insights and react to new information in real-time. Learn how it is used in conjunction with other technologies, like PubSub, Kafka, BigQuery, Bigtable, or Datastore, to build end-to-end streaming architectures. Learn how these architectures enable diverse use cases such as real-time ingestion and ETL, real-time reporting & analytics, real-time alerting, or fraud detection. DA219 Event schedule → http://g.co/next18 Watch more Data Analytics sessions here → http://bit.ly/2KXMtcJ Next ‘18 All Sessions playlist → http://bit.ly/Allsessions Subscribe to the Google Cloud channel! → http://bit.ly/NextSub event: Google Cloud Next 2018; re_ty: Publish; product: Cloud - Data Analytics - Dataflow; fullname: Ryan McDowell;


Breakfast and learn: Caching Made Easy, with Cloud Memorystore (Cloud Next '18)

Aug. 26, 2021

Here is the link. 

Cloud Memorystore introduces a managed redis and memcache service on Google Cloud. In this session, we will walk you through the new service features. The session will also show how you can easily migrate applications using redis to Cloud Memorystore with minimal changes. IO224 Event schedule → http://g.co/next18 Watch more Infrastructure & Operations sessions here → http://bit.ly/2uEykpQ Next ‘18 All Sessions playlist → http://bit.ly/Allsessions Subscribe to the Google Cloud channel! → http://bit.ly/NextSub event: Google Cloud Next 2018; re_ty: Publish; product: Cloud - Databases - Memorystore; fullname: Gopal Ashok;


Wednesday, August 25, 2021

Andrel Jikh: 10 Best Stocks To Own For Life | Stock investment

Aug. 25, 2021

Here is the link. 

My list of the 10 best stocks to buy and hold for life. ► Get up to a $250 in Digital Currency: https://blockfi.com/andrei ► Where I Buy Bitcoin: https://gemini.sjv.io/1E3dz ► My Stock Portfolio + Stock Tracker: https://www.patreon.com/andreijikh ► Get 2 FREE stocks valued up to $1850 (when you deposit $100): https://act.webull.com/kol-us/share.h... ► MY DISCORD: https://discord.gg/9TVPxj73Bb ► Open A Roth IRA: https://m1finance.8bxp97.net/c/198055... ► Follow Me On Instagram: https://www.instagram.com/andreijikh/ ► How I Protect My Bitcoin: https://shop.ledger.com/pages/ledger-... My PO Box: Andrei Jikh 4132 S. Rainbow Blvd # 270 Las Vegas, NV 89103 2 years ago I made a video about the top 7 stocks that pay you money and in that video I talked about the best dividend stocks that you can own forever. But in today’s video I wanted to share with you my top 10 best stocks if I had $1,000 to invest with and someone forced me to pick 10 stocks and I needed to create a portfolio today. This is what I would do. 10. Amazon is now valued over 1.6 trillion dollars and if I only had $1,000 - it would be very tempting not to put all of that in this stock, but we want to be diversified so I’d put in $100 of my $1,000. Now for me, this is a stock I’m going to probably load up on relatively soon and put in $50k to $100k, it’s been a long time coming, I’ve been looking at this stock and I’m like when do I buy this thing? It doesn’t go down. So for me, this will be a very long term investment for the next 10 to 20 years from now, there is no way this company isn’t worth double or triple what it is today. And because their quarterly earnings didn’t impress the analysts - the stock finally dropped so now might be a good time to buy. 9. Johnson and Johnson. This is a pharmaceutical company. I’ve owned Johnson and Johnson since 2014, I bought shares when it was just under $100 and they’ve been a powerhouse of passive income and one of the things that interested me the most about it is that I heard it was a dividend aristocrat (a stock that has been paying dividends, passive income, to investors for at least 25 years in a row without interrupting and without stopping) Johnson and Johnson has done not just 25 years, but 58 years without a dividend cut. 8. Tesla. More than just a tech company it’s a car company, it's a tech company that will most likely play a huge role in not only the tech space but the energy space and potentially the finance space as well. Tesla owns Solar City, they install solar roof panels, they sell cars, they have powerful battery technology, they have arguably the most advanced self driving product on the market with the most integrated charging platform in the world and in the future I think Elon and Tesla will integrate cryptocurrency into their pipeline as well. According to Morningstar the fair price is $570. 7. Disney. An innovation powerhouse. Their latest product - the Disney Plus streaming platform has been a huge revenue maker for them. They got over 95 million subscribers on the Disney Plus platform in the first 14 months. In just a year and two months they were able to beat their 4 year goal. Their new 4 year goal is to hit a quarter of a billion subscribers by 2024. 6. Google. In 2015 they reported a second quarterly earnings of 17.7 billion dollars. Just recently, they reported a revenue of 61.9 billion dollars just 6 years later which is also 62% higher than the second quarter of last year when the pandemic happened. Just for context, there are 2 billion people on YouTube right now, that is 26% of the entire world’s population that uses YouTube and that number will continue to grow. 5. Facebook. Facebook owns FB marketplace, Facebook Messenger, Whatsapp, Instagram and they’re launching their own cryptocurrency called Diem and The Metaverse which is virtual reality sort of like in Ready Player One. Like them or not, Facebook has an insane amount of potential in the future even when factoring the risk of regulation. They have so much valuable data and there’s anyone that will compete with Google in the future, it will be Facebook. The share price is trading at slightly below fair value which may be a good time to buy in. 4. KO Coca Cola! A Warren Buffett favorite and Warren Buffett proves that drinking Coca Cola doesn’t affect your longevity. Like Johnson and Johnson they’ve they are a dividend aristocrat that have been paying their shareholders dividends for 58 years in a row. Their payout ratio is 74% but their dividend safety score is still 80 which is solid. Watch the video for the top 3 stocks! *None of this is meant to be construed as investment advice, it's for entertainment purposes only. Links above include affiliate commission or referrals. I'm part of an affiliate network and I receive compensation from partnering websites. The video is accurate as of the posting date but may not be accurate in the future.

RIG stock: Frackers See Light At The End Of The Tunnel In Oil Markets

Frackers See Light At The End Of The Tunnel In Oil Markets


Recovering global oil demand and drilling activity has made oilfield services firms optimistic about the sector’s prospects not only in the immediate future but also in the medium term.  

Oilfield service providers reported second-quarter earnings that beat analyst forecasts. But more importantly, companies specializing in different segments—from fracking to offshore drilling—expressed confidence that the oil services industry now faces a solid recovery in the coming quarters and even years, despite the recent Delta variant surge. 

The market also seems to believe that recovering global oil demand will boost drilling activity. Shares of oilfield services firms have rallied over the past year, driven by rising oil prices and demand. 

Schlumberger (NYSE: SLB) is up 46.5 percent in the past 12 months, Helmerich & Payne (NYSE: HP) is up 54 percent, Halliburton (NYSE: HAL) stock has gained 24 percent, Baker Hughes (NYSE: BKR) has seen shares rise by 42 percent over the past year, and Transocean (NYSE: RIG) has surged by 159 percent since August 2020.  

The rising U.S. rig count and the increase in American crude oil production also support the view that a recovery is underway. 

As of August 20, the total rig count in the U.S. was 503 active rigs, up by 249 from the same time last year—the highest rig count since April 2020, but still down sharply from the 790 active rigs in March 2020. The EIA’s estimate for U.S. crude oil production for the week ending August 13 rose by 100,000 bpd for the third week in a row to an average of 11.4 million bpd. 

Moreover, the oilfield services sector has now seen a fifth consecutive month of job growth, adding an estimated 6,082 jobs in July, the Energy Workforce & Technology Council said earlier this month in an analysis based on preliminary data from the Bureau of Labor Statistics (BLS). The sector has restored around 38,300 positions from a peak of over 115,000 pandemic-related job losses, the council estimates.  

Executives at oilfield services firms said in recent weeks that they expect a lasting, possibly a multi-year, recovery in activity and in their business. 

The world’s three largest oilfield services providers expect the vaccine-driven recovery in global oil demand to continue through the coming quarters, which they expect will lead to the next upcycle in demand for oil drilling and completions. 

“The positive activity momentum we see in North America and international markets today, combined with our expectations for future customer demand, gives us conviction for an unfolding multi-year upcycle,” Jeff Miller, Halliburton’s chairman, president, and CEO, said last month in the Q2 results release. 

“Although we recognize the risks presented by the variant strains of the COVID-19 virus, we expect spending and activity levels to gain momentum through the year as the macro environment improves, likely setting up the industry for stronger growth in 2022,” Baker Hughes chairman and CEO Lorenzo Simonelli said. 

“With oil price at elevated levels, the supply response to this demand recovery is developing broadly as anticipated. Indeed, this combination has resulted in a call on short-cycle production as well as an uptick in long-cycle project, reflected in new FIDs and encouraging recovery in both offshore developments and near-field exploration activity through the second quarter,” Schlumberger CEO Olivier Le Peuch said on the Q2 earnings call. 

Helmerich & Payne CEO John Lindsay said at the end of July, “Assuming oil prices remain stable and near current levels, we would not be surprised to see 2022 budgets for public companies drive further incremental increases in rig activity next year.”

The Permian will continue to lead the way in active rig gains, Helmerich & Payne says. 

Midland, Texas-based ProPetro Holding, which provides pressure pumping for fracking in North America, said in early August that “the pressure pumping industry is faced with an impending reinvestment cycle that will require innovative solutions to meet the needs of the market.” 

“As we navigate the COVID-19 delta and potentially other variants, volatility will remain, but oil price fundamentals appear strong for a continued recovery,” ProPetro chairman CEO Phillip Gobe said. 

“Timing remains uncertain as we’re still in the early innings of what we view as a multiyear recovery,” he added. 

Outside the United States, recovery is also underway and will accelerate in the coming months, executives say.

“Supported by the ongoing conversations with our customers, we believe we are at the beginning of a sustainable recovery for offshore drilling. Assuming oil prices remain constructive, we believe we will witness a robust offshore market recovery in 2022 and beyond,” Transocean’s CEO Jeremy Thigpen said on the Q2 earnings call earlier this month.

By Tsvetana Paraskova for Oilprice.com 

Architecture sessions - Google cloud next '19 | Large distributed system | My plan

 Aug. 25, 2021

Introduction

I like to go over as many videos as possible from the following list. I do believe that it is much better for me to learn from those videos first, I will continue to read more books related to large distributed system. 

Architecture sessions - Google cloud next '19

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24