作者:大睿
大睿,DBA,喜好减肥,瘦了30多斤,负责公司数据库集群的治理和保护。
本文起源:原创投稿
*爱可生开源社区出品,原创内容未经受权不得随便应用,转载请分割小编并注明起源。
物化视图应用to
的形式写入到存储表中,即如下:
CREATE MATERIALIZED VIEW[IF NOT EXISTS][db.]table_name[ON CLUSTER] TO[db.]nameASSELECT ...
指定了存储的表,所以物化视图的创立也不须要指定 engine ,在查问中,查物化视图和查理论的存储表失去一样的数据,因为都是来自于同一份存储数据。
物化视图是计算每次写入原表的数据,通过聚合之后写入到指标表。比方,有依照 1s 一次记录的明细表,同时须要依照分钟级做数据的聚合统计pv(相似的须要),则能够通过创立物化视图的形式将聚合后的数据写到 1min 的表中(这种感觉有点像触发器)
范例
1s记录的明细表
CREATE TABLE dba_test.t_1s( `ctime` DateTime64(0), `pv` Int64)ENGINE = MergeTreePARTITION BY toDate(ctime)ORDER BY ctimeSETTINGS index_granularity = 8192
1min 记录的聚合数据
CREATE TABLE dba_test.t_1m( `ctime` DateTime64(0), `pv` Int64)ENGINE = SummingMergeTreePARTITION BY toDate(ctime)ORDER BY ctimeSETTINGS index_granularity = 8192
物化视图t_1m_mv,查问条件是从1s的表(t_1s),依照分钟级(toStartOfMinute)聚合查问后果,从新写入到1min的表(t_1m)中
物化视图
CREATE MATERIALIZED VIEW dba_test.t_1m_mv TO dba_test.t_1m( `toStartOfMinute(ctime)` DateTime, `pv` Int64) ASSELECT toStartOfMinute(ctime), sum(pv) AS pvFROM dba_test.t_1sGROUP BY ctime
写入测试
dba-clickhouse-001 :) insert into t_1s values('2022-01-01 00:10:01',1),('2022-01-01 00:10:01',1),('2022-01-01 00:20:01',2),('2022-01-01 00:20:01',2),('2022-01-01 00:30:01',3);INSERT INTO t_1s VALUESQuery id: 0bf16844-0123-4e25-a3d4-f9b5a5c8db37Ok.5 rows in set. Elapsed: 0.003 sec.dba-clickhouse-001 :) select * from t_1s;SELECT *FROM t_1sQuery id: cb442100-37a6-4de7-b6f3-f80f084710dc┌───────────────ctime─┬─pv─┐│ 2022-01-01 00:10:01 │ 1 ││ 2022-01-01 00:10:01 │ 1 ││ 2022-01-01 00:20:01 │ 2 ││ 2022-01-01 00:20:01 │ 2 ││ 2022-01-01 00:30:01 │ 3 │└─────────────────────┴────┘5 rows in set. Elapsed: 0.002 sec.dba-clickhouse-001 :) select * from t_1m;SELECT *FROM t_1mQuery id: f9d2d05d-8ad7-44a4-b66a-ea8c3c758f1f┌───────────────ctime─┬─pv─┐│ 1970-01-01 08:00:00 │ 9 │└─────────────────────┴────┘1 rows in set. Elapsed: 0.002 sec.
插入的工夫居然是1970-01-01 08:00:00
开始验证是否是查问语句有误
查看物化视图中的查问后果是否合乎预期
dba-clickhouse-001 :) SELECT:-] toStartOfMinute(ctime),:-] sum(pv) AS pv:-] FROM dba_test.t_1s:-] GROUP BY ctime;SELECT toStartOfMinute(ctime), sum(pv) AS pvFROM dba_test.t_1sGROUP BY ctimeQuery id: 1ecaf07e-c766-40b7-bfa2-0f87ee54abad┌─toStartOfMinute(ctime)─┬─pv─┐│ 2022-01-01 00:20:00 │ 4 ││ 2022-01-01 00:30:00 │ 3 ││ 2022-01-01 00:10:00 │ 2 │└────────────────────────┴────┘3 rows in set. Elapsed: 0.002 sec.
查问后果合乎预期
间接通过insert ...select...
形式确认下插入数据是否合乎预期
dba-clickhouse-001 :) insert into t_1m SELECT:-] toStartOfMinute(ctime),:-] sum(pv) AS pv:-] FROM dba_test.t_1s:-] GROUP BY ctime;INSERT INTO t_1m SELECT toStartOfMinute(ctime), sum(pv) AS pvFROM dba_test.t_1sGROUP BY ctimeQuery id: 5db8279a-ffb1-4174-843c-80cee48b448cOk.0 rows in set. Elapsed: 0.002 sec.dba-clickhouse-001 :) select * from t_1m;SELECT *FROM t_1mQuery id: acd79ea7-dc82-49f1-bb71-430a05895f19┌───────────────ctime─┬─pv─┐│ 1970-01-01 08:00:00 │ 9 │└─────────────────────┴────┘┌───────────────ctime─┬─pv─┐│ 2022-01-01 00:10:00 │ 2 ││ 2022-01-01 00:20:00 │ 4 ││ 2022-01-01 00:30:00 │ 3 │└─────────────────────┴────┘4 rows in set. Elapsed: 0.002 sec.
直接插入,数据正确,工夫没有被转化。
能够确认物化视图的查问局部是没有问题,那只能是在写入的时候呈现了问题,换个思路去想一下,工夫戳的开始工夫是1970-01-01 00:00:00,而这里插入的工夫是1970-01-01 08:00:00多了8小时,也就是说因为时区的起因导致工夫推延了。那会不会是因为插入的数据不标准,或者是“空”被转化了呢。
验证
dba-clickhouse-001 :) insert into t_1m values('',100);INSERT INTO t_1m VALUESQuery id: af1785ef-dca1-467b-84c6-27f9da6547f6Ok.1 rows in set. Elapsed: 0.002 sec.dba-clickhouse-001 :) select * from t_1m;SELECT *FROM t_1mQuery id: 34db2057-7274-4859-898e-6132f8df4465┌───────────────ctime─┬─pv─┐│ 1970-01-01 08:00:00 │ 9 │└─────────────────────┴────┘┌───────────────ctime─┬─pv─┐│ 2022-01-01 00:10:00 │ 2 ││ 2022-01-01 00:20:00 │ 4 ││ 2022-01-01 00:30:00 │ 3 │└─────────────────────┴────┘┌───────────────ctime─┬──pv─┐│ 1970-01-01 08:00:00 │ 100 │└─────────────────────┴─────┘5 rows in set. Elapsed: 0.002 sec.
果然,当插入的数据为空的时候,工夫被重置了。
比照下物化视图和指标的聚合表的构造
dba-clickhouse-001 :) desc t_1m;DESCRIBE TABLE t_1mQuery id: 96c6a5ca-e42a-47e1-8212-cbcfefa6ffa4┌─name──┬─type──────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐│ ctime │ DateTime64(0) │ │ │ │ │ ││ pv │ Int64 │ │ │ │ │ │└───────┴───────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘2 rows in set. Elapsed: 0.001 sec.dba-clickhouse-001 :) desc t_1m_mv;DESCRIBE TABLE t_1m_mvQuery id: a258f6b5-f195-4386-a9bb-4ec86e7e9bd1┌─name───────────────────┬─type─────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐│ toStartOfMinute(ctime) │ DateTime │ │ │ │ │ ││ pv │ Int64 │ │ │ │ │ │└────────────────────────┴──────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘2 rows in set. Elapsed: 0.001 sec.
聚合表工夫字段名叫ctime,物化视图的则是toStartOfMinute(ctime)
从新调整物化视图的写法,并清理t_1m表中的数据
dba-clickhouse-001 :) show create table t_1m_mv\Gstatement: CREATE MATERIALIZED VIEW dba_test.t_1m_mv TO dba_test.t_1m( `ctime` DateTime, `pv` Int64) ASSELECT toStartOfTenMinutes(ctime) AS ctime, sum(pv) AS pvFROM dba_test.t_1sGROUP BY ctimedba-clickhouse-001 :) insert into t_1s values('2022-01-01 00:10:01',1),('2022-01-01 00:10:01',1),('2022-01-01 00:20:01',2),('2022-01-01 00:20:01',2),('2022-01-01 00:30:01',3);INSERT INTO t_1s VALUESQuery id: 812d1bbd-55f3-4a8f-b9f7-bbbe93e694afOk.5 rows in set. Elapsed: 0.003 sec.dba-clickhouse-001 :) select * from t_1m;SELECT *FROM t_1mQuery id: 2d1a045a-4e53-4f94-bb6a-fe5e5d58f5c7┌───────────────ctime─┬─pv─┐│ 2022-01-01 00:10:00 │ 2 ││ 2022-01-01 00:20:00 │ 4 ││ 2022-01-01 00:30:00 │ 3 │└─────────────────────┴────┘3 rows in set. Elapsed: 0.002 sec.
论断
物化视图的字段(t_1m_mv)要与指标表(t_1m)的字段名对齐
(表白不是很谨严,大略是下面的意思)