Google cloud: Dataflow | 20 minutes reading

 Dataflow

Unified stream and batch data processing that's serverless, fast, and cost-effective.

• Fully managed data processing service

• Automated provisioning and management of processing resources

• Horizontal autoscaling of worker resources to maximize resource utilization

• OSS community-driven innovation with Apache Beam SDK

• Reliable and consistent exactly-once processing

Streaming data analytics with speed

Dataflow enables fast, simplified streaming data pipeline development with lower data latency.

Simplify operations and management

Allow teams to focus on programming instead of managing server clusters as Dataflow’s serverless approach removes operational overhead from data engineering workloads.

Reduce total cost of ownership

Resource auto scaling paired with cost-optimized batch processing capabilities means Dataflow offers virtually limitless capacity to manage your seasonal and spiky workloads without overspending.

KEY FEATURES

Key features

Autoscaling of resources and dynamic work rebalancing

Minimize pipeline latency, maximize resource utilization, and reduce processing cost per data record with data-aware resource autoscaling. Data inputs are partitioned automatically and constantly rebalanced to even out worker resource utilization and reduce the effect of “hot keys” on pipeline performance.

Flexible scheduling and pricing for batch processing

For processing with flexibility in job scheduling time, such as overnight jobs, flexible resource scheduling (FlexRS) offers a lower price for batch processing. These flexible jobs are placed into a queue with a guarantee that they will be retrieved for execution within a six-hour window.

Ready-to-use real-time AI patterns

Enabled through ready-to-use patterns, Dataflow’s real-time AI capabilities allow for real-time reactions with near-human intelligence to large torrents of events. Customers can build intelligent solutions ranging from predictive analytics and anomaly detection to real-time personalization and other advanced analytics use cases. 

All features

Vertical autoscaling - new in Dataflow Prime	Dynamically adjusts the compute capacity allocated to each worker based on utilization. Vertical autoscaling works hand in hand with horizontal autoscaling to seamlessly scale workers to best fit the needs of the pipeline.
Right fitting - new in Dataflow Prime	Right fitting creates stage-specific pools of resources that are optimized for each stage to reduce resource wastage.
Smart diagnostics - new in Dataflow Prime	A suite of features including 1) SLO-based data pipeline management, 2) Job visualization capabilities that provide users a visual way to inspect their job graph and identify bottlenecks, 3) Automatic recommendations to identify and tune performance and availability problems.
Streaming Engine	Streaming Engine separates compute from state storage and moves parts of pipeline execution out of the worker VMs and into the Dataflow service back end, significantly improving autoscaling and data latency.
Horizontal autoscaling	Horizontal autoscaling lets the Dataflow service automatically choose the appropriate number of worker instances required to run your job. The Dataflow service may also dynamically reallocate more workers or fewer workers during runtime to account for the characteristics of your job.
Dataflow Shuffle	Service-based Dataflow Shuffle moves the shuffle operation, used for grouping and joining data, out of the worker VMs and into the Dataflow service back end for batch pipelines. Batch pipelines scale seamlessly, without any tuning required, into hundreds of terabytes.
Dataflow SQL	Dataflow SQL lets you use your SQL skills to develop streaming Dataflow pipelines right from the BigQuery web UI. You can join streaming data from Pub/Sub with files in Cloud Storage or tables in BigQuery, write results into BigQuery, and build real-time dashboards using Google Sheets or other BI tools.
Flexible Resource Scheduling (FlexRS)	Dataflow FlexRS reduces batch processing costs by using advanced scheduling techniques, the Dataflow Shuffle service, and a combination of preemptible virtual machine (VM) instances and regular VMs.
Dataflow templates	Dataflow templates allow you to easily share your pipelines with team members and across your organization or take advantage of many Google-provided templates to implement simple but useful data processing tasks. This includes Change Data Capture templates for streaming analytics use cases. With Flex Templates, you can create a template out of any Dataflow pipeline.
Notebooks integration	Iteratively build pipelines from the ground up with Vertex AI Notebooks and deploy with the Dataflow runner. Author Apache Beam pipelines step by step by inspecting pipeline graphs in a read-eval-print-loop (REPL) workflow. Available through Google’s Vertex AI, Notebooks allows you to write pipelines in an intuitive environment with the latest data science and machine learning frameworks.
Real-time change data capture	Synchronize or replicate data reliably and with minimal latency across heterogeneous data sources to power streaming analytics. Extensible Dataflow templates integrate with Datastream to replicate data from Cloud Storage into BigQuery, PostgreSQL, or Cloud Spanner. Apache Beam’s Debezium connector gives an open source option to ingest data changes from MySQL, PostgreSQL, SQL Server, and Db2.
Inline monitoring	Dataflow inline monitoring lets you directly access job metrics to help with troubleshooting batch and streaming pipelines. You can access monitoring charts at both the step and worker level visibility and set alerts for conditions such as stale data and high system latency.
Customer-managed encryption keys	You can create a batch or streaming pipeline that is protected with a customer-managed encryption key (CMEK) or access CMEK-protected data in sources and sinks.
Dataflow VPC Service Controls	Dataflow’s integration with VPC Service Controls provides additional security for your data processing environment by improving your ability to mitigate the risk of data exfiltration.
Private IPs	Turning off public IPs allows you to better secure your data processing infrastructure. By not using public IP addresses for your Dataflow workers, you also lower the number of public IP addresses you consume against your Google Cloud project quota.

Google cloud platform deep-dive | 15 minutes summary of Google cloud Bigtable | Misha Brukman, Google

 Sept. 24, 2021

Here is the link. 

Googler Misha Brukman and Leon Stein from eBay discuss how Cloud Bigtable handles eBay's global catalog with billions of listings, scaling to hundreds of terabytes with many millions of reads and writes, and gigabytes of data transferred in and out every second. Missed the conference? Watch all the talks here: https://goo.gl/c1Vs3h Watch more talks about Application Development here: https://goo.gl/YFgZpl

15 minutes Google cloud technology by the engineer Googler Misha Brukman:

Google cloud platform deep-dive - 15 minutes summary of Google cloud Bigtable

Misha Brukman, Google

8:40 - 23:00/ 54:18  

17:41

• NoSQL, sparse wide-column database
• Single index: the row key
• Atomic single-row transactions

wide-column: column family, unlimited columns. Data model, three dimensional spreadsheet

Bytes - uninterpreted data type 

Time series schema - Heroic, Spotify - Heroic 

wide-column databases 

HgraphDB - ? JanusGraph - IBM, Expero, new project -> open source wide-column database 

bigtable - stateless - Bigtable and

Fishing: Stave River Fishing

 

Stave River Fishing Location & Species

The Stave River is located in Mission, British Columbia. It’s a short length river that starts at the Ruskin Dam and dumps into the main stem of the Fraser River. Albeit a small river the Stave holds a decent number of species like Chum Salmon, Coho Salmon, Steelhead, native Rainbow Trout, Cutthroat, Whitefish and Northern Pikeminnow to mention a few. This river becomes extremely busy in October and November when the Salmon begin to enter the system. This often creates close proximity fishing and highly pressured fish.

In 2019 BC Hydro opened the new picnic site on the east side of the river (near the Ruskin Dam). This picnic has a gate that opens at 7:00 am and provides amble parking and river accessibility. Beside the picnic area and parallel to the river there is the Ruskin Spawning Channel and there is NO FISHING ALLOWED in this section of the river. However, it’s a great place for families and anglers to watch the Salmon spawn.

Bringing billions of eBay listings together using Cloud Bigtable to support new shopping experiences

Sept. 24, 2021

Here is the link. 

Googler Misha Brukman and Leon Stein from eBay discuss how Cloud Bigtable handles eBay's global catalog with billions of listings, scaling to hundreds of terabytes with many millions of reads and writes, and gigabytes of data transferred in and out every second. Missed the conference? Watch all the talks here: https://goo.gl/c1Vs3h Watch more talks about Application Development here: https://goo.gl/YFgZpl

10 minutes Google cloud technology by the engineer Googler Misha Brukman:

Google cloud platform deep-dive - 15 minutes summary of Google cloud Bigtable

Misha Brukman, Google

8:40 - 23:00/ 54:18  

17:41

• NoSQL, sparse wide-column database
• Single index: the row key
• Atomic single-row transactions

wide-column: column family, unlimited columns. Data model, three dimensional spreadsheet

Bytes - uninterpreted data type 

Time series schema - Heroic, Spotify - Heroic 

wide-column databases 

HgraphDB - ? JanusGraph - IBM, Expero, new project -> open source wide-column database 

bigtable - stateless - Bigtable and

Visualizing Cloud Bigtable Access Patterns at Twitter for Optimizing Analytics (Cloud Next '18)

Sept. 24, 2021

Here is the link. 

In this session, Steve Niemitz from Twitter will present how they use Cloud Bigtable to power their advertiser analytics systems. In the process of designing and implementing the system, Steve and his team iteratively optimized their Cloud Bigtable row key schema using intuition about the data set. Misha Brukman and Wim De Pauw will present Cloud Bigtable, its storage and scalability model, and how Twitter visualized access patterns to Cloud Bigtable at scale, without performance impact to the Cloud Bigtable cluster. Event schedule → http://g.co/next18 Watch more Application Development sessions here → http://bit.ly/2zMcTJc Next ‘18 All Sessions playlist → http://bit.ly/Allsessions Subscribe to the Google Cloud channel! → http://bit.ly/NextSub event: Google Cloud Next 2018; re_ty: Publish; product: Cloud - Databases - Cloud Bigtable;

Coal price: My short research

 【煤炭价格暴涨70%，中国早有准备！】

今年来，煤炭市场异常火爆，煤价也在持续上涨。据媒体报道，由于主要动力煤产国的供应仍处于受限，加上海运运费不减，海运动力煤的价格持续上涨，而市场煤炭价格也创下10年以来新高。

除此之外，据统计数据显示，2021年1月份-5月份，中国煤炭的价格分别有不同幅度的上涨。其中，动力煤价格均值上涨45.5%，炼焦煤价格上涨12.6%，无烟煤价格上涨10.8%。亚洲基准价纽卡斯尔动力煤离岸价格已经超过了140美元/吨，同比增长超过70%。价格暴涨抑制了中国煤炭的进口规模。面对即将进入的用电高峰期，煤炭市场供需是否还能保持平衡？

一、全国夏季用电高峰来临，煤炭储备压力大

1、多地用电负荷创新高

随着夏季的到来，全国电力资源的压力再次出现，气象学家说今年的夏季可能比往年更热。

进入7月以来全国陆续出现高温天气。江苏、浙江、广东陆续出现用电高峰。7月6日，江苏电网用电负荷达到10031万千瓦，这是夏季梅雨期期间，江苏电网用电负荷首次突破1亿千瓦。7月8日，浙江省最高用电负荷达9681万千瓦，再创新高。7月8日广东省也迎来用电高峰，广东省最高统调负荷达13156万千瓦，再创历史新高。

华泰期货分析师王海涛说：“中国南方一直很热，每天的电力负荷不断突破新高。一些地区再次对电力进行配给，并对使用煤炭发出警告。”

2、发改委投放1000万吨煤平衡市场

7月15日，从国家发展改革委经济运行调节局了解到，今年以来，国家根据煤炭供需形势需要，先后4次向市场投放超过500万吨国家煤炭储备。为做好迎峰度夏煤炭供应保障工作，根据监测分析情况，国家发改委已提前制定储备投放预案，本次准备投放规模超过1000万吨，主要分布在全国各地的几十个储煤基地和有关港口，能够根据需要随时投放市场。

此次抛储1000万吨煤，实际对于全国电厂来说，可以缓解一天半的消耗量。对于全国全年煤炭消费近40亿吨的规模来说，抛储数量并不算大。但是，这一举措非常有象征意义，表明国家层面对于火电厂电煤紧缺情况，高度关注，后续恐有更多应对措施陆续出台。

二、煤炭企业业绩翻倍，后市会如何？

煤炭市场利好，多家煤炭上市公司半年报也陆续出炉，业绩均有大幅增长。

中煤能源(01898)：预计上半年实现净利润71.9亿元至79.5亿元，同比将增加48.8亿元至56.40亿元，增长211.3%至244.2%。

盘江煤：生产原煤1267万吨，同比增长35%；焦炭374万吨，同比增长1.4%；营业收入238亿元，同比增长40%；利润5.28亿元，同比增长4300%；税金13亿元，同比增长40%。

随着国家的调控和产能补充，相关上涨因素影响逐渐减弱，价格上涨局面将得到抑制。同时也是给供应紧张的中国煤炭市场吃下一颗“定心丸”。

Coal price: My short research | GTE and RIG stock investment

能耗双控｜电煤价格飙升逾1000元/吨 火电度电亏损超1毛：是新能源的春天还是冬天

来源:球哥看风  作者:李彩球  2021/9/23 14:11:41

 

北极星火力发电网讯:就在前几天，中央电视台报道了电煤价格问题，5500大卡以上的电煤价格已经突破1000元/吨，最高达到1080元/吨，是电煤价格正常水平的2倍以上，在这个电煤价格下，火电的亏损已经成为不争的事实，作为京津冀电力从不亏损的的京能电力，今年也给出了亏损的半年报。公司2021年上半年预计归属于上市公司股东的净利润约-3亿元。，而业绩预亏主要是由于煤炭成本同（环）比大幅上涨，影响公司利润同比下降。

（来源：球哥看风 ID：colorballwind2020 作者：李彩球）

火电压力山大-度电亏损0.12元：风电煤价格破1000元/吨

根据中央电视台的报道，陕西榆能采购的电煤价格已经达到1080元/吨，每度电亏损0.12元，每个月发电10亿度，亏损1.2亿元，冬天即将到来，水电面临枯水期，而风电和光伏的发电比例按照国家能源局的要求也只有11%，火电的电源比例仍高达50%以上，火电面临巨大能耗双控压力。

同时，水泥的熟料成本已经增加80%以上，基本上水泥价格也是上涨一倍，在生态严控的态势下，本来水泥的矿产开采已经让水泥企业苦不堪言，加上限产，水泥价格肯定是一路狂奔。

能耗双控：大宗原材料限产导致价格攀升

这几天，能耗双控政策的出台，让大宗原材料行业明显受伤：

工业硅：云南确保工业硅企业9-12月份月均产量不高于8月产量的10%（即削减90%产量）。根据2020年前三季度中国工业硅产量地区分布情况来看，云南地区工业硅产量占总产量的21%，仅次于新疆，因为工业硅的大幅减产或将影响多晶硅的价格调整！自8月以来，硅料价格已五连涨， 目前， 国内单晶复投料价格区间在206-217元/kg。

环氧树脂：9月7日江苏省工信厅召开企业专项节能监察电视电话会议，江苏省环氧氯丙烷部分企业开始陆续降负或停车，区域内环氧氯丙烷多数企业企业受到影响，江苏主要环氯生产企业均属于减排重点对象，江苏各大树脂企业也在此行列当中。环氯最新参考价在20300元/吨左右，再创高价记录，环氯价格的大涨，进一步压缩下游环氧树脂的利润空间，截至9月15日上午，黄山固体树脂参考价为32000元/吨，华东液体树脂参考价为36500元/吨以上，高出正常年份价格50%以上。

作为大宗原材料的一个典型，价格攀升已经成为行业不争事实，问题是现在的大宗原材料几乎到了一日一价的地步，让人如何不担心未来的造价成本。

光伏组件价格或重回2元/w：风电主机2元/w能撑多久

作为光伏和风电的两类大宗原材料，价格在能耗双控的高压态势下，本已给行业看到价格回落的希望再次带来失望，光伏组件价格或重回2元/W时代，而对刚刚降至在2元/w的风电主机而言，这无疑是雪上加霜。从8月份的钢材，水泥及有色金属的产能对比上月下降的幅度超多10%以上，钢材价格，水泥价格继续维持高位，原因何在呢？

降低煤炭产能是降能耗的主要措施。作为钢铁行业的焦炭价格已经持续高升，从6月份的2500元/吨飙升至4100元/吨。这无疑加大了钢铁以及冶金行业的原材料成本，也直接拉动了钢铁原材料的价格上涨，上周螺纹钢的现货价格已经超过5400元/吨。如果类似环氧树脂、钢材等主要原材料价格在能耗双控的高压形势下，下半年持续减产和控制煤炭产能，这些大宗原材料的价格或将继续维持高位，对光伏还是风电的基础设施成本是一个巨大的挑战。

当然，能耗双控给大家带来的市场的空间扩大的未来，而在巨大的市场空间下，新能源电力如何在能耗双控的大好形势下发展。

大力发展新能源和原材料价格上涨：是新能源的春天还是冬天

不得不说，能耗双控带来的能源转型确实是一定时期的阵痛，火电面临一定时间的亏损，新能源短期内还不能大幅度的补位火电的发电量，从这一点看，新能源的春天来临。同时，我们也要清醒的看到，电力是关系经济发展和民生的重要资源，冬天即将来临，尤其是北方的热电联产，电煤的价格高位态势如果不能回落，电力企业的亏损压力很难短期解除。而能耗双控带来的连锁反应，大宗原材料限产，原材料价格持续上涨，都会给风电和光伏的发展带来巨大的压力，这可能是新能源电源开发和建设的第一个冬天-成本或大幅上扬。

在当前的技术水平和电价形势下，风电的单位千瓦造价已经接近电源企业开发电源项目收益率的临界点，如果大宗原材料价格不能得到有效控制，或许刚刚下降的成本会带来反弹，项目是否能够继续维持企业收益率水平，能否具备投资价值，反过来会对新能源的开发市场带来消极影响。

从风电或光伏的成本降低的途径来看，一个是降低原材料价格，一个是减少原材料用量，第一个问题属于我们今天讨论的范畴，这个需要行业进行控制，这是外部问题，或者说是政策环境问题。第二个问题是企业自身问题，这是内部问题，需要用技术创新来解决，这也是行业企业的竞争力。

节能减排是大势所趋，作为企业要顺势而为，尤其是风电装备行业企业，要抓住这个机遇，通过自身的技术创新解决能耗双控带来的新难题，否则极有可能被历史淘汰。

Coal price problem: 发一度电，赔一毛钱 | My short research | GTE stock 20,000 share investment

 

“煤电顶牛”矛盾再升级

来源： 中国能源报

2021-09-10 22:00

当前煤电企业之所以普遍面临“经营危机”，表面看是源于“煤电顶牛”这一老问题——煤企大赚、电企大亏，但问题的本质出在电价机制没有理顺。电厂买煤卖电，是典型的“中间商”，本可以将成本顺利地疏导出去，但当前的电价形成机制，阻碍了成本的疏导，进而一次又一次地让煤电企业陷入集体亏损的困局。

发一度电，赔一毛钱——这是京津唐地区某燃煤发电企业持续多日的经营异象。知情人士甚至指出，如果电价低位锁死、煤价持续飞涨的情况继续下去，当地燃煤电厂“可能一个多月就会被彻底拖垮”。

在此背景下，大唐国际、北京国电电力、京能电力、华能集团华北分公司等11家燃煤发电企业曾在1个月前联名向北京市城市管理委员会上书，请求重新签订北京地区电力直接交易2021年10—12月的年度长协合同。截至记者发稿，能否重签合同尚无定论。

在这份名为《关于重新签约北京地区电力直接交易2021年10-12月年度长协合同的请示》（以下简称“《请示》文件”）的函件中，11家企业联名京津唐电网燃煤电厂成本已超过盈亏平衡点，与基准电价严重倒挂，燃煤电厂亏损面达到100％，煤炭库存普遍偏低，煤量煤质无法保障，发电能力受阻，严重影响电力交易的正常开展和电力稳定供应，企业经营状况极度困难，部分企业已出现了资金链断裂。

据记者了解，京津唐地区只是全国煤电行业“经营危机”的一个缩影。

煤价翻倍增长

“2008年左右也出现过电煤持续涨价的情况，但当时京津唐地区煤电机组的利用小时数还是比较高的，虽然那时候煤价也比较高，但持续时间短，至少卖电赚回来的电费还足够买煤，有时候还能剩点。但这次不一样，入不敷出，如果再不调整，可能整个煤电行业要崩溃了。”华北电网电力调度处原处长梁明亮说，按照当前秦皇岛港5500大卡燃煤价格约885元/吨计算，折算到7000大卡标煤，价格为1126元/吨；2020年全国平均供电煤耗为307克/千瓦时，依此测算，仅燃料成本就达到0.3456元/千瓦时。“现在的基准电价在0.35—0.36元/千瓦时左右。这还没考虑电煤运输到电厂的费用，肯定是发一度赔一度。”

某煤电企业相关负责人告诉记者，在去年底签约时，煤价水平只有600多元/吨。“低的时候甚至是300—400元/吨。谁能想到现在几乎是在翻倍涨价。”

煤价随行就市、水涨船高，电价却被牢牢锁死。《请示》文件显示，北京地区电力直接交易价格平均降幅已达到0.06—0.11元/千瓦时，京津唐燃煤电厂在煤价突涨且持续高位运行等市场发生严重异常的情况下，已无力完成2020年12月签约的北京地区2021年10—12月电力直接交易和2021年3月签约的北京地区2021年10—12月电力直接交易。

中国电力企业联合会规划发展部副主任、燃料分会副秘书长叶春指出，今年上半年，中国沿海电煤采购价格指数（CECI）曹妃甸指数5500大卡现货成交价已超过1000元/吨。事实上，2016年实施煤炭供给侧结构性改革以来，煤炭供需形势扭转，电煤价格一路攀升，而煤电企业经营形势则日渐严峻，中电联多次通过各种渠道上报国家相关部委反映经营困难。“在政策性降电价、燃料价格上涨、电力市场交易规模扩大等多重因素影响下，煤电企业生存空间一压再压。”

电价机制漏洞凸显

“无力完成”就可以重签合同吗？在长沙理工大学教授叶泽看来，重签合同的诉求不合“规”却合“理”。“市场交易合同是严肃的经济合同，受法律保护，不能因为一方利益受损或者亏损就更改合同。如果这样，市场经济根本无法正常运转。但煤电企业的确严重亏损，而且燃料成本的上涨确实也不应该完全由发电企业承担。”

中国社科院财经战略研究院副研究员冯永晟指出，现行的电力市场建设并不完善。“我国长协的特殊之处在于一口价锁死，国外的长协一般会有价格调整公式，提前约定好哪些成本可以传导到电价中去，按什么方式传导。以我国目前的情况，更应该关注市场本身在价格传导顺畅性、风险管理完善程度等方面存在的问题。”

叶泽进一步指出，自2020年1月1日起，我国全面取消煤电价格联动机制，实行多年的“标杆上网电价机制”改为“基准价+上下浮动”的市场化机制。其中，基准价按各地此前燃煤发电标杆上网电价确定，浮动范围为上浮不超过10%、下浮原则上不超过15%，具体电价由发电企业、售电公司、电力用户等通过协商或竞价确定。“这个机制顺畅运转的前提是煤价保持相对稳定，一旦煤价大幅波动，新机制的不合理性就会充分暴露出来。比如今年的煤价大幅上涨，即使按10%的上浮比率确定交易价格，也不能传导煤价成本的上涨。因此，新机制在设计上有明显的漏洞。”

梁明亮也坦言，此前由于煤炭产能充裕，煤电矛盾尚有“周期”可言，“但这次就是长期缺煤，煤炭企业‘咬’着高价，电厂基本是国有企业，不能停机，再贵也得买。”

仍需政策治本

“若煤电厂全面、长期亏损，企业就面临破产的风险。”梁明亮直言，为避免亏损乃至破产，煤电企业必然会设法少发电或者停机，“最直接的影响便是缺电”。

事实也的确如此，叶春指出：“以2020年11月为例，我国浙江、湖南的用电量增速分别为8.8%和9.1%，而火电发电量增速仅为5.1%和2.4%，供需明显错配。2021年以来，部分省市未进入迎峰度夏期就频繁出现拉闸限电现象，电力供应紧缺信号凸显。”

不仅如此，冯永晟强调，煤电行业的生存窘境如果无法破解，也必将影响可再生能源的发展，进而影响碳达峰、碳中和目标的实现。“煤电是支撑新能源继续快速发展的主力资源，也是支持储能发展的战略资源。如果煤电因全面、长期亏损而过快、过度地退出，新能源又很难保障电力系统的稳定运行，最终将严重制约新能源发展目标的达成。河还没过，就不要先拆桥。不但不要拆，还要把桥架到对岸。”

叶泽认为，当前煤电企业的生存发展不取决于市场，仍取决于政策。“主管部门要基于市场经济规则，为煤电企业生存发展优化完善现行政策及市场体系和交易机制。当前的电力系统是离不开煤电的，主管部门不能对煤电行业的经营困难不管不顾。”

评论 | 理顺价格机制才能消解煤电困境

文 | 中国能源报评论员

继2008—2011年间煤电企业大面积亏损后，2017年至今煤电行业再陷泥潭。不同的是，一向“富裕”的京津唐地区煤电出现“发一度电、赔一毛钱”的情形还是首次。一个不可否认的事实是，煤电是当前及未来一段时期内我国电力系统的“压舱石”，实现碳达峰、碳中和目标离不开煤电企业的保驾护航。由此观之，煤电厂当下普遍存在的长期巨亏问题，相关主管部门绝不能置之不理。

在碳中和的背景下，谈到“高碳”的煤电，自然绕不开能源低碳转型的话题。近年来，我国能源结构大幅优化，成就斐然：非化石能源消费比重从2015年的12.1%提高到2019年的15.3%，提前一年完成“十三五”规划目标；“十三五”以来，非化石能源发电量增量占到全社会用电量增量的52.3%，已成为名副其实的主力军；碳中和目标提出后，我国非化石能源发展更加势不可挡——截至7月底，全国发电装机容量22.7亿千瓦，其中非化石能源装机容量已达10.3亿千瓦，同比大增18.0%，且在未来相当长一段时期内仍将保持强劲的增长势头。

与非化石能源规模飙涨相对应的，是煤电装机占比的逐年下降，目前已降至50%以下。但能源转型不是简单的数学题，而是一个盘根错节、千头万绪的系统性课题。煤电比重的降低，绝不意味着煤电地位的下降。从某种程度上说，随着非化石能源装机的突飞猛进，煤电在当前电力系统中愈发不可或缺。

但值得注意的是，近年来，煤电行业面临重重困难，内有燃料价格大幅上涨、利用小时巨幅下降、综合电价随市场交易持续下滑的压力，外有降碳催生的巨大环保压力。煤电行业如何定位和发展，已不只是煤电行业从业者自身需要关注的话题，更是关乎碳达峰、碳中和目标能否如期实现的重大难题。

煤电在我国电力装机中比重最大，碳排放量也占据“大头”，深度参与能源转型是势在必行的事。但煤电不仅是被改革的对象，更是改革的重要参与者。

一方面，保障国家能源安全的现实需求决定了煤电行业必须“活下去”。我国的能源资源禀赋特点是“缺油少气铀不多，有水富煤多风光”，特别是在目前原油对外依存度超70%、天然气对外依存度超40%的背景下，煤炭是目前保证我国能源安全的不二选择，这也意味着煤电的关键地位短期内不可能动摇。

另一方面，可再生能源大规模并网也需要煤电行业“活得好”。“风光”具有间歇性、波动性的天性，如何安全、稳定并网是当前建设新型电力系统最大的问题。在其他调峰资源远未成熟的当下，如果没有煤电机组平抑海量新能源接入电网后产生的剧烈波动，可再生能源的充分消纳和电网的稳定输配电将是天方夜谭，“构建以新能源为主体的新型电力系统”的目标，恐怕也将变成一句空话。

当前煤电企业之所以普遍面临“经营危机”，表面看是源于“煤电顶牛”这一老问题——煤企大赚、电企大亏，但问题的本质出在电价机制没有理顺。电厂买煤卖电，是典型的“中间商”，本可以将成本顺利地疏导出去，但当前的电价形成机制，阻碍了成本的疏导，进而一次又一次地让煤电企业陷入集体亏损的困局。

“惟改革者进，惟创新者强，惟改革创新者胜。”理顺价格机制才是消解煤电困境的关键所在。任由煤电这个城门不断“失火”，最终殃及的“池鱼”将是降碳大计。

本文作者：赵紫原、姚金楠，来源：中国能源报，原文标题：《“煤电顶牛”矛盾再升级》

DATA & ANALYTICS - Build smart applications with your new superpower: cloud machine learning

Sept. 24, 2021

Here is the link. 

Recorded on Mar 24 2016 at GCP NEXT 2016 in San Francisco. Visual effects rendering is a computationally intensive process where one second of screen-time can require thousands of cores and terabytes of frame data. Learn how Academy Award-winning and recognized studios take advantage of cloud economics and Google's on-demand computing to realize their creative visions and expand this digital medium for storytelling. Speakers: Julia Ferraioli, Google & David Zuckerman, Wix

Building a Global Data Presence with Cloud Bigtable (Cloud Next '19)

Sept. 24, 2021

Here is the link. 

Google Cloud enables any business to create a low-latency data presence in many places at once, leveraging a state-of-the-art global network. With Cloud Bigtable’s recently launched multi-region replication offering, a single instance of data — from terabytes to petabytes — can be accessed within or between five different continents, across up to four regions. This solution enables both low-latency data access around the world as well as a fast-failover disaster recovery solution for critical data. Learn how Google’s software-defined WAN works and how Cloud Bigtable uses it for global replication. Build with Google Cloud → https://bit.ly/2UlXMAh Watch more: Next '19 Databases Sessions here → https://bit.ly/Next19Databases Next ‘19 All Sessions playlist → https://bit.ly/Next19AllSessions Subscribe to the GCP Channel → https://bit.ly/GCloudPlatform Speaker(s): Carter Page, Subhasree Mandal , Douglas Mcerlean Session ID: DBS307 event: Google Cloud Next 2019; re_ty: Publish; product: Cloud - Databases - Cloud Bigtable; fullname: Carter Page, Subhasree Mandal , Douglas Mcerlean; event: Google Cloud Next 2019;

GTE stock: 37% gain in one month up to Sept. 24, 2021

Sept. 24, 2021

I noticed that GTE stock went down more than 30%, and I decided to buy more at low price. But I thought about learning more about one more business, so I chose to purchase RIG stock instead, $15,000 dollars. 

Investing on stocks is not just to check short term return. I like to allow myself make mistake, really learn and ask myself more questions to learn through real experience. 

This week in energy market: Sept. 24, 2021

 Sept. 24, 2021

This week in energy markets:

1. Brent crude: $78 a barrel, 3-year high - GTE printing $$$
2. Asian coal: $185 a tonne, 13-year high
3. German 1-year power: $108 MWh, record high
4. Europe ant gas: -$26 per mBtu, record high

1439. Find the Kth Smallest Sum of a Matrix With Sorted Rows

Leetcode discuss: 127. Word Ladder

Sept. 22, 2021

Here is the link.

C# | BFS | Deadloop - TLE concern | Warmup before onsite

Sept. 23, 2021
Introduction
It is challenge for me to adjust myself to bring up best performance for my onsite interview. I chose to practice Leetcode algorithms contains "word".

BFS algorithm | deadloop - avoid loop in search | backtracking is not necessary | Two loops - All chars from 'a' to 'z'

I tried a few times in order to make the code pass online judge. Avoid loop in BFS is a must. It is challenge to make it correct to count depth of words, startWord = "cat", endWord = "cat", word dictionary is ["cat"], the result should be 2. I will look into the issue later.

The following code passes online judge.

public class Solution {
    public int LadderLength(string beginWord, string endWord, IList<string> wordList) {
       if(beginWord == null || beginWord.Length == 0 || wordList == null || wordList.Count == 0) 
           return 0; 
        
       var hashSet = new HashSet<string>(wordList); 
        if(!hashSet.Contains(endWord))
            return 0;
        
        // apply BFS search
        var queue = new Queue<string>();
        queue.Enqueue(beginWord);
        var depth = 1; 
        while(queue.Count > 0)
        {
            depth++;
            
            var count = queue.Count;
            for(int i = 0 ; i < count; i++)
            {
                var visit = queue.Dequeue(); 
                
                for(int j = 0; j < visit.Length; j++)
                {
                    var replaced = visit.ToCharArray();
                    for(int k = 0; k < 26; k++)
                    {                        
                        replaced[j] = (char) ('a' + k);
                        var search = new string(replaced);
                        
                        if(!hashSet.Contains(search) || search.CompareTo(visit) == 0)
                        {                            
                            continue;                        
                        }            
                        
                        if(search.CompareTo(endWord) == 0)
                            return depth;
                        
                        hashSet.Remove(search); // avoid deadloop - tested! It is a must!
                        
                        queue.Enqueue(search);                                                   
                    }
                }
            }                      
        }
        
        return 0; 
    }
}

I am a new person | Take breaks | Meet friends on running court | Sept. 22, 2021 | Onsite first day

Sept. 23, 2021

Introduction

It is hard for me to manage so many past discussion posts of algorithms and also system design interview study. I came cross so many blogs in my blog, how I should prepare for my onsite for system design and algorithm. 

I am a new person | Take breaks | Mental health

I did take a 20 minutes walk and tried to relax myself during study. I made some plans for lunch and snacks. It is not easy to handle stress and it is important not to bring stress into the onsite interviews as well. 

I also took multiple breaks during study. I think that it is better for me to learn how to communicate through real interviews.

I have to think more carefully. After the onsite, I chose to go out to run in Burnaby central middle school. We had a gather-up training, and I ran less than 2 miles. I learn how to discipline myself through running training. I need to get my weight under 180 lb in order for me to run better and easily. 

I also like to take some notes to summarize what I could have done better through those two interviews. 

1. Aiming optimal solution in terms of algorithm problem solving. Still need to ask more clarification questions, I should take actions quickly and work harder, and work on a few examples, and explain what to work on. 
2. System design, I need to work on the structure of interview better, summary the architecture in better ways. What to cover or not cover, and also do not bring too many topics into conversation, focus on architecture. 
3. Focus on technical topics, I did not mention anything about horizontal scaling, CAP theorem, and microservice architecture, and CQSG - command query segregation pattern etc. 
4. Ask dive-deep questions about large distributed system - my goal, better question to ask, for example, "If there are 10 books to cover large distributed system and master the basics, what those 10 books will be?".
5. Need to take more breaks after the first and first two interviews. I did not have too many ideas, but I did not eat a lot since I only ate enough, it was best case to handle my two interviews.

Actionable items

Follow up Sept. 23, 2021 6:24 PM

1. What is your best advice to learn large distributed system? For example, if there is a book you read and you learn tremendously, what is the book? Do you think that there are 10 books to read in your quick thoughts?
2. ...

Amazon web services | Amazon System Design Preparation (SIP) | replay three times at least

Sept. 21, 2021

Here is the link. 

This video tackles a system design example question and how candidates should approach, analyze and solve such technical questions. This is video 3 out of a 3 part series for Software Development Engineer Interview Preparation (SIP). https://www.amazon.jobs/en

Ask questions is important - ask how many users - 6 millions, dialog with interviewers, assuming ...

Focus on your strength - start from frontend if you are front end developer 

1. ask scale, performance, about API
2. talk, key components 
3.  make good conversation - 

Online book store - interviewer may interrupt you, work on more detail

DB

Customer - name, ID, 

Order - Trans ID, payment method, date 

Books - ID, author, more if needed

API - key aspect - how to construct? Domain / resource / ...

Second play - 

1. manager - Samir Kopal 
2. System design - key components - design a system 
3. Do not jump into solving problem | ask question, solve it with a team | this is what I am thinking | key components 
4. Online book store 
5. Book - ebooks, regular books, help you scale - 6 M users, 500 TPS - 500 transaction per second - total in a day - 24 * 60 * 60 * 500 = ...
6. Focus on your strength - walk up from database to front end if you are a database developer

Systems Design Interview Concepts (for software engineers / full-stack web)

Sept. 21, 2021

Here is the link. 

In this video, we discuss load balancing, CDNs, database replication, sharding, caching layers, database schema and indexes, distributed filesystems, and other ways to scale an application. You don't want to scale your design too early or needlessly, but consider where the bottlenecks are. Sometimes you will need database sharding for instance, while other times you may not need a database at all (ie., if the data does not need to persist to disk). Note: the concepts will differ somewhat for a frontend or mobile systems design, which are more focused on the client-side architecture and software design patterns.

Topics:

1. Load balancer | roudn robin, hash IP address, more 
2. Cache - Redis, Cassandra, memCache | CDN 
3. database replication -> slave database | replication | consistency 
4. horizontal sharding | consistent hashing | master table for sharding algorithm
5. NoSQL database - key value pair, hybrid - ?
6. API design - Jason, protocol buffer, make it fast | Ask clarification | simplicity 

Replay | My second play notes

1. Senior, Facebook | two system design interviews 
2. Framework, design pattern, scalability 
3. Top concepts - Load balancer - throughput, latency, ... NGINX | DNS load balancing | Round robin | web server not go down
4. Caching - memcached - memcached, redis, 
5. CDN - pool technique - fetch and cache | push - high frontend cost 
6. Distributed file system - 

Replay third time | My notes 

1. Framework, design pattern, a lot of them relate to scalability 
2. CDN - image, javascript file, CSS, all others | Pull technique | first visit slow | high front cost 
3. Distributed file system - Amazon S3 | database index | query | ...

Rush to review 100 algorithms: 10:38 PM - 11:38 PM, 6:30 AM - 9:30 AM | Leetcode algorithm

Sept. 21, 2021

Introduction

I like to make plans to review my Leetcode practice over 650 algorithms in next 15 hours. 

Leetcode algorithms | My past practice

Here is the link of page. 

Adrian Cockcroft

 Adrian Cockcroft is the VP of Cloud Architecture Strategy at Amazon Web Services. He was previously a Technology Fellow at Battery Ventures, and before that a cloud architect at Netflix, where he was instrumental in their migration to a “cloud first” architecture^[1]. Prior to Netflix he held distinguished engineer roles at eBay, where he was a founding member of eBay Research Labs^[2], and Sun Microsystems. He speaks frequently on architecture and open source^[3], and writes about technology trends on Medium and Twitter.

An open-source advocate, Cockcroft is on the board of the Cloud Native Computing Foundation^[4], and a member of the Prototype Project Team at OS-Climate.