From January 2015, she started to practice leetcode questions; she trains herself to stay focus, develops "muscle" memory when she practices those questions one by one.
2015年初, Julia开始参与做Leetcode, 开通自己第一个博客. 刷Leet code的题目, 她看了很多的代码, 每个人那学一点, 也开通Github, 发表自己的代码, 尝试写自己的一些体会.
She learns from her favorite sports – tennis, 10,000 serves practice builds up good memory for a great serve. Just keep going.
Hard work beats talent when talent fails to work hard.
Sep.16 -- The world is facing high energy prices for the foreseeable future as oil and natural gas producers resist the urge to drill again, according to Chevron CEO Mike Wirth. He spoke with Bloomberg's Alix Steel about challenges ahead for the energy sector and consumer.
05/04/2017 - Did you know we have an amazing archive facility that showcases our company's rich history spanning 135 years? You'll have the chance to tour the facility with our historian John Harper and his team during a four-part video series featuring the Chevron Archive. Learn more about Chevron’s history: https://www.chevron.com/about/history
Nicole Oakes, a crude supply and trading manager for Chevron’s Latin America team, shows us what life is like on the trading room floor. https://www.chevron.com/stories/i-am-...
2020 birthday | GME stock | $30 dollar loss | One day | Panic sale | 54 year old
Highest price of GMO was $483/ share, if I do have confidence and choose to keep those 30 shares of GME, then I can make profit around $15,000 US dollars.
2021 birthday | Review stocks | Candidates | Sweet bet | 55 year old
Although a SAN provides only block-level access, file systems built on top of SANs do provide file-level access and are known as shared-disk file systems.
Storage architectures
Storage area networks (SANs) are sometimes referred to as network behind the servers[1]: 11 and historically developed out of a centralized data storage model, but with its own data network. A SAN is, at its simplest, a dedicated network for data storage. In addition to storing data, SANs allow for the automatic backup of data, and the monitoring of the storage as well as the backup process.[2]: 16–17 A SAN is a combination of hardware and software.[2]: 9 It grew out of data-centric mainframe architectures, where clients in a network can connect to several servers that store different types of data.[2]: 11 To scale storage capacities as the volumes of data grew, direct-attached storage (DAS) was developed, where disk arrays or just a bunch of disks (JBODs) were attached to servers. In this architecture, storage devices can be added to increase storage capacity. However, the server through which the storage devices are accessed is a single point of failure, and a large part of the LAN network bandwidth is used for accessing, storing and backing up data. To solve the single point of failure issue, a direct-attached shared storage architecture was implemented, where several servers could access the same storage device.[2]: 16–17
DAS was the first network storage system and is still widely used where data storage requirements are not very high. Out of it developed the network-attached storage (NAS) architecture, where one or more dedicated file server or storage devices are made available in a LAN.[2]: 18 Therefore, the transfer of data, particularly for backup, still takes place over the existing LAN. If more than a terabyte of data was stored at any one time, LAN bandwidth became a bottleneck.[2]: 21–22 Therefore, SANs were developed, where a dedicated storage network was attached to the LAN, and terabytes of data are transferred over a dedicated high speed and bandwidth network. Within the SAN, storage devices are interconnected. Transfer of data between storage devices, such as for backup, happens behind the servers and is meant to be transparent.[2]: 22 In a NAS architecture data is transferred using the TCP and IP protocols over Ethernet. Distinct protocols were developed for SANs, such as Fibre Channel, iSCSI, Infiniband. Therefore, SANs often have their own network and storage devices, which have to be bought, installed, and configured. This makes SANs inherently more expensive than NAS architectures.[2]: 29
Learn three architecture options: Shared-nothing, shared disk, and shared-memory
Definition of shared-nothing? Think about it in a minute.
Definition: A shared-nothing architecture is simply a distributed architecture. This means the server nodes do not share either memory or disk and have their own copy of the operating system, hence they share “nothing” other than the network to communicate with other nodes
Google the following statement - SAN - storage area network, block-based protocol
Shared-disk requires disks to be globally accessible by the nodes, which requires a storage area network (SAN) that uses a block-based protocol.
block-based protocol - learn three things in 20 minutes
SAN - a storage area network (SAN) - in 20 minutes
SAN - focus on a block-based protocol - in 10 minutes
Shared-nothing has become the dominant parallel architecture for big data systems, such as MapReduce and Spark, analytics platforms, NoSQL databases and search engines [Özsu & Valduriez 2020]. The reason is simple: it is the only architecture that can provide scalability at reasonable cost, typically within a cluster of servers. In the context of cluster computing, scalability can be further characterized by the terms scale-up versus scale-out. Scale-up (also called vertical scaling) refers to adding more power (processor, memory, IO devices) to a server and thus gets limited by the maximum size of the server, e.g. 32 processors. Scale-out (also called horizontal scaling) refers to adding more servers, called “scale-out servers”, in a loosely coupled fashion, to scale almost infinitely.
In a parallel database system, the real challenge is to scale linearly with increasing workloads (see our blog post on Scalability), including more users and more data. For instance, if you double the size of your cluster, you would expect to support a workload that is twice as big. For an OLAP workload, this may mean dividing the response time of a large analytical query by two, whereas for an OLTP workload this may mean doubling the system throughput (e.g., number of transactions per minute). Note that these are very different objectives, which can be achieved with different architectures and techniques. The level of difficulty to implement basic functions (concurrency control, fault-tolerance, availability, database design and tuning, load balancing, etc.) varies from one architecture to the other.
Stonebraker proposed the term “shared-nothing” to better contrast with two other popular architectures, shared-memory and shared-disk. A shared-nothing architecture is simply a distributed architecture. This means the server nodes do not share either memory or disk and have their own copy of the operating system, hence they share “nothing” other than the network to communicate with other nodes [Valduriez 2018]. In the 1980s, shared-nothing was just emerging (with pioneers like Teradata and Tandem NonStopSQL) and shared-memory was the dominant architecture.
In shared-memory (see above), any processor (P) has access to any memory module (M) or disk unit through some interconnect. All the processors are under the control of a single operating system. One major advantage is simplicity of the programming model, which is based on shared virtual memory. Metadata (e.g. directory) and control data (e.g., lock tables) can be shared by all processors, which means that writing database software is not very different than for single processor computers. In particular, load balancing is easy since it can be achieved at runtime by allocating each new task to the processor with least load. However, the major problem with shared-memory is limited scalability and availability.
With increasingly quick processors (even with larger caches), conflicting accesses to the shared-memory increase rapidly and degrade performance. Furthermore, since the memory space is shared by all processors, a memory fault may affect most of them, thereby hurting data availability. Depending on whether physical memory is shared, two approaches are possible: Uniform Memory Access (UMA) and Non-Uniform Memory Access (NUMA) (see Figure 2). UMA is the architecture of multicore processors while NUMA is used for tightly-coupled multiprocessors. Interestingly, today’s servers are NUMA, which introduces many difficulties for database managers since they were built for UMA. In particular, seeing as all threads can access memory from an CPU, this results in a high fraction of accesses to remote NUMA memory. This results in exhausting the memory bandwidth to remote memories that is significantly smaller than the memory bandwidth of the local memory. Most database servers just do not scale up linearly in NUMA and a lot of work is needed for them to become NUMA-aware.
In a shared-disk architecture, any processor has access to any disk unit, but exclusive (nonshared) access to its main memory. Each processor–memory node is under the control of its own copy of the operating system and can communicate with other nodes through the network. Then, each node can access database pages on the shared-disk and cache them into its own memory. Shared-disk requires disks to be globally accessible by the nodes, which requires a storage area network (SAN) that uses a block-based protocol. Since different processors can access the same page in conflicting update modes, global cache consistency is needed. This is typically achieved using a distributed lock manager, which is complex to implement and introduces significant contention. Shared-disk has three main advantages: simple and cheap administration, high availability, and good load balance. Database administrators do not need to deal with complex data partitioning, and the failure of a node only affects its cached data, while the data on disk is still available to the other nodes. Furthermore, load balancing is easy because any request can be processed by any node. The main disadvantages are cost (because of the cost of the SAN) and limited scalability, caused by the potential bottleneck and overhead of cache coherence protocols for large databases. In the case of OLTP workloads, shared-disk has remained the preferred option as it makes load balancing easy and efficient. However, their scalability is heavily limited by the distributed locking that causes severe contention and limits the scalability to a few nodes.
Today, a shared-nothing architecture (see Figure 4) is cost-effective as servers can be off-the-shelf components (multicore processor, memory and disk) connected by a regular Ethernet network or a low-latency network such as Infiniband or Myrinet. To maximize performance (yet at additional cost), nodes could also be NUMA multiprocessors, thus leading to a hierarchical parallel database system [Bouganim et al. 1996]. By favoring the smooth incremental growth of the system by the addition of new nodes, shared-nothing provides excellent scalability, unlike shared-memory and shared-disk. However, it requires careful partitioning of the data on multiple disk nodes. Furthermore, the addition of new nodes in the system presumably requires reorganizing and repartitioning of the database to deal with the load balancing issues. Finally, fault-tolerance is more difficult than with shared-disk, seeing as a failed node will make its data on disk unavailable, thus requiring data replication. It is due to its scalability advantage that shared-nothing has been first adopted for OLAP workloads, in particular data warehousing, as it is easier to parallelize read-only queries. For the same reason, it has been adopted for big data systems, which are typically read-intensive.
A final question is: can shared-nothing be used to support big write-intensive transactional workloads as well? NoSQL (see our blog post on NoSQL), in particular key-value systems, have excellent horizontal scalability, i.e., scaling over a cluster of nodes. However, since they did not manage to scale transactional management, they gave up on transactional consistency; choosing to focus instead on scalability. However, it is possible to provide both scalability for data management and transactional management (see our blog post on the CAP theorem). Yet, it is a hardcore problem that only a few systems in the NewSQL category have managed to solve (see our blog post on NewSQL).
MAIN TAKEAWAYS
There are three main parallel database architectures: shared-memory, shared-disk and shared nothing. Shared-memory is used for in-memory databases, and shared-disk for small clusters. Shared nothing is becoming the dominant technology since it works anywhere, from on-premise cluster to private or public cloud. The main challenge is to attain linear scalability and transactional (ACID) consistency. Initially, key-value NoSQL systems have been able to achieve linear scalability, but by sacrificing transactional consistency. More recently, a few NewSQL systems have been able to achieve both linear scalability and transactional consistency.
Josh Brown of Ritholtz Wealth Management and Pete Najarian of MarketRebellion.com join 'Halftime Report' to discuss where they're finding opportunities to invest in a downturn market.
Damien Courvalin, Goldman Sachs head of energy research, joins 'Power Lunch' to discuss why he raised his year-end price target for oil and walks through the state of the energy market. For access to live and exclusive video from CNBC subscribe to CNBC PRO: https://cnb.cx/2NGeIvi
#oilprices#Brentcrude#gasprices
Yahoo Finance's Brian Sozzi and Julie Hyman spoke with Goldman Sachs Head of Energy Research Damien Courvalin about the outlook for oil and gas prices.
Don't Miss: Valley of Hype: The Culture That Built Elizabeth Holmes
WATCH HERE:
https://youtu.be/Sb179GLPNYE
Oct.04 -- Goldman Sachs Head of Energy Research Damien Courvalin says declining oil inventories, stalled U.S.- Iran negotiations, and OPEC+ are behind the bank's bullish $90 a barrel year-end forecast. "Inventories are about to fall to their lowest in 10 years. And in our view that requires another leg higher" Courvalin says on "Bloomberg Markets."
Oct.04 -- Amrita Sen, director and founder of Energy Aspects, discusses the current energy crunch across the globe, looks ahead to the OPEC+ meeting later and gives her outlook for energy prices. She speaks on “Bloomberg Daybreak: Europe.”
Sep.27 -- Amrita Sen, director of research at Energy Aspects, discusses the factors pushing natural gas and oil prices higher. She speaks on "Bloomberg Markets."
High-performance transaction system applications typically insert rows in a
History table to provide an activity trace; at the same time the transaction system generates log
records for purposes of system recovery. Both types of generated information can benefit from
efficient indexing. An example in a well-known setting is the TPC-A benchmark application,
modified to support efficient queries on the History for account activity for specific accounts.
This requires an index by account-id on the fast-growing History table. Unfortunately, standard disk-based index structures such as the B-tree will effectively double the I/O cost of the
transaction to maintain an index such as this in real time, increasing the total system cost up to
fifty percent. Clearly a method for maintaining a real-time index at low cost is desirable. The
Log-Structured Merge-tree (LSM-tree) is a disk-based data structure designed to provide
low-cost indexing for a file experiencing a high rate of record inserts (and deletes) over an
extended period. The LSM-tree uses an algorithm that defers and batches index changes, cascading the changes from a memory-based component through one or more disk components in an
efficient manner reminiscent of merge sort. During this process all index values are continuously accessible to retrievals (aside from very short locking periods), either through the
memory component or one of the disk components. The algorithm has greatly reduced disk arm
movements compared to a traditional access methods such as B-trees, and will improve cost performance in domains where disk arm costs for inserts with traditional access methods
overwhelm storage media costs. The LSM-tree approach also generalizes to operations other
than insert and delete. However, indexed finds requiring immediate response will lose I/O efficiency in some cases, so the LSM-tree is most useful in applications where index inserts are
more common than finds that retrieve the entries. This seems to be a common property for
History tables and log files, for example. The conclusions of Section 6 compare the hybrid use
of memory and disk components in the LSM-tree access method with the commonly understood
advantage of the hybrid method to buffer disk pages in memory.
Statement: the key-value pair model provided by distributed B-trees or distributed hash tables is too limiting
a shared-nothing [33] architecture - https://www.linkedin.com/pulse/case-shared-nothing-ricardo-jimenez-peris/
the Log-Structured Merge Tree [26] stores updates to index data
Read the paper related to the Log-Structured Merge Tree
Related work
The Boxwood project [24] has components that overlap
in some ways with Chubby, GFS, and Bigtable, since it
provides for distributed agreement, locking, distributed
chunk storage, and distributed B-tree storage. In each
case where there is overlap, it appears that the Boxwood’s component is targeted at a somewhat lower level
than the corresponding Google service. The Boxwood
project’s goal is to provide infrastructure for building
higher-level services such as file systems or databases,
while the goal of Bigtable is to directly support client
applications that wish to store data.
Many recent projects have tackled the problem of providing distributed storage or higher-level services over
wide area networks, often at “Internet scale.” This includes work on distributed hash tables that began with
projects such as CAN [29], Chord [32], Tapestry [37],
and Pastry [30]. These systems address concerns that do
not arise for Bigtable, such as highly variable bandwidth,
untrusted participants, or frequent reconfiguration; decentralized control and Byzantine fault tolerance are not
Bigtable goals.
In terms of the distributed data storage model that one
might provide to application developers, we believe the
key-value pair model provided by distributed B-trees or
distributed hash tables is too limiting. Key-value pairs
are a useful building block, but they should not be the
only building block one provides to developers. The
model we chose is richer than simple key-value pairs,
and supports sparse semi-structured data. Nonetheless,
it is still simple enough that it lends itself to a very efficient flat-file representation, and it is transparent enough
(via locality groups) to allow our users to tune important
behaviors of the system.
Several database vendors have developed parallel
databases that can store large volumes of data. Oracle’s
Real Application Cluster database [27] uses shared disks
to store data (Bigtable uses GFS) and a distributed lock
manager (Bigtable uses Chubby). IBM’s DB2 Parallel
Edition [4] is based on a shared-nothing [33] architecture
similar to Bigtable. Each DB2 server is responsible for
a subset of the rows in a table which it stores in a local
relational database. Both products provide a complete
relational model with transactions.
Bigtable locality groups realize similar compression
and disk read performance benefits observed for other
systems that organize data on disk using column-based
rather than row-based storage, including C-Store [1, 34]
and commercial products such as Sybase IQ [15, 36],
SenSage [31], KDB+ [22], and the ColumnBM storage
layer in MonetDB/X100 [38]. Another system that does
vertical and horizontal data partioning into flat files and
achieves good data compression ratios is AT&T’s Daytona database [19]. Locality groups do not support CPU cache-level optimizations, such as those described by
Ailamaki [2].
The manner in which Bigtable uses memtables and
SSTables to store updates to tablets is analogous to the
way that the Log-Structured Merge Tree [26] stores updates to index data. In both systems, sorted data is
buffered in memory before being written to disk, and
reads must merge data from memory and disk.
C-Store and Bigtable share many characteristics: both
systems use a shared-nothing architecture and have two
different data structures, one for recent writes, and one
for storing long-lived data, with a mechanism for moving data from one form to the other. The systems differ significantly in their API: C-Store behaves like a
relational database, whereas Bigtable provides a lower
level read and write interface and is designed to support
many thousands of such operations per second per server.
C-Store is also a “read-optimized relational DBMS”,
whereas Bigtable provides good performance on both
read-intensive and write-intensive applications.
Bigtable’s load balancer has to solve some of the same
kinds of load and memory balancing problems faced by
shared-nothing databases (e.g., [11, 35]). Our problem is
somewhat simpler: (1) we do not consider the possibility
of multiple copies of the same data, possibly in alternate
forms due to views or indices; (2) we let the user tell us
what data belongs in memory and what data should stay
on disk, rather than trying to determine this dynamically;
(3) we have no complex queries to execute or optimize.
I like to learn from the following writing in a book:
In 2003, Google published a paper titled “The Google File System”. This scalable distributed file system, abbreviated as GFS, uses a cluster of commodity hardware to store huge amounts of data. The filesystem handled data replication between nodes so that losing a storage server would have no effect on data availability. It was also optimized for streaming reads so that data could be read for processing later on.
Shortly afterward, another paper by Google was published, titled “MapReduce: Simplified Data Processing on Large Clusters”. MapReduce was the missing piece to the GFS architecture, as it made use of the vast number of CPUs each commodity server in the GFS cluster provides. MapReduce plus GFS forms the backbone for processing massive amounts of data, including the entire search index Google owns.
What is missing, though, is the ability to access data randomly and in close to real-time (meaning good enough to drive a web service, for example). Another drawback of the GFS design is that it is good with a few very, very large files, but not as good with millions of tiny files, because the data retained in memory by the master node is ultimately bound to the number of files. The more files, the higher the pressure on the memory of the master.
So, Google was trying to find a solution that could drive interactive applications, such as Mail or Analytics, while making use of the same infrastructure and relying on GFS for replication and data availability. The data stored should be composed of much smaller entities, and the system would transparently take care of aggregating the small records into very large storage files and offer some sort of indexing that allows the user to retrieve data with a minimal number of disk seeks. Finally, it should be able to store the entire web crawl and work with MapReduce to build the entire search index in a timely manner.
Being aware of the shortcomings of RDBMSes at scale (see “Seek Versus Transfer” on page 315 for a discussion of one fundamental issue), the engineers approached this problem differently: forfeit relational features and use a simple API that has basic create, read, update, and delete (or CRUD) operations, plus a scan function to iterate over larger key ranges or entire tables. The culmination of these efforts was published in 2006 in a paper titled “Bigtable: A Distributed Storage System for Structured Data”, two excerpts from which follow:
Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers.
It is highly recommended that everyone interested in HBase read that paper. It describes a lot of reasoning behind the design of Bigtable and, ultimately, HBase. We will, however, go through the basic concepts, since they apply directly to the rest of this book.
HBase is implementing the Bigtable storage architecture very faithfully so that we can explain everything using HBase. Appendix F provides an overview of where the two systems differ.
It is a lonely journey to build a successful career, I have to work on my advancement continuously. With great learning experience, I have over 10 onsite interviews from Facebook (3), Amazon (4), Microsoft (2), Google (1) from 2008 to 2021. I learned so much from those interview experience. Last two years, I started to learn how to invest on stock market, it is tough since I did not make any gain on my Questrade.com TFSA accounts. Last 11 years, I did not purchase any real estate in Canada.
Case study | Lack of confidence | My impulsive decision making | Six months efforts
Recently I sold my 20,000 shares of GTE stock at price 0.91/ share, after Amazon onsite, in less than 2 weeks, the price went up to 1.10/ share. I guessed that I must be lack of confidence on my own research.
I have to watch out myself to check my confidence level. I have to learn how to work with myself better. Train myself better to handle stress of investment.
Work on something else | Read more large distributed system books | Invest and work on equity research
I have a wonderful life in the city of Vancouver, stay healthy and enjoy simple life. Being frugal, I choose to stay at a tiny bedroom rented, drive a 2001 Honda accord, in July I chose to ask my neighbor to help maintenance jobs in order to cut cost, a series of jobs including coolant flush, air conditioning fix, gasket replacement, climate control unit order in ebay and replacement, fix of front body dent etc., in total of $660, and work on equity research, and being a lonely investor on stock market.
I just keep learning. So much fun in terms of 55 year old, I did run 10 K race first time less than a month ago, and also take care of my frozen shoulder, try to warm up my shoulder every day before going to work.
I also like to solve more algorithms on Leetcode.com. And also I like to post more videos on Instagram, and learn how to use my Google 4a 5G more often to record good videos and I like to be creative, learn more on Instagram.com.
