Clickhouse技术分享: 分区裁剪

发表于 2021-07-15 更新于 2024-03-06 分类于技术文章阅读次数：本文字数： 3.5k 阅读时长 ≈ 3 分钟

本文代码截图, 基于2021-07-19的master版本

DataPart存储

CREATE TABLE default.lineorder_local
(
    `LO_ORDERKEY` UInt32,
    `LO_LINENUMBER` UInt8,
    `LO_CUSTKEY` UInt32,
    `LO_PARTKEY` UInt32,
    `LO_SUPPKEY` UInt32,
    `LO_ORDERDATE` Date,
    `LO_ORDERPRIORITY` LowCardinality(String),
    `LO_SHIPPRIORITY` UInt8,
    `LO_QUANTITY` UInt8,
    `LO_EXTENDEDPRICE` UInt32,
    `LO_ORDTOTALPRICE` UInt32,
    `LO_DISCOUNT` UInt8,
    `LO_REVENUE` UInt32,
    `LO_SUPPLYCOST` UInt32,
    `LO_TAX` UInt8,
    `LO_COMMITDATE` Date,
    `LO_SHIPMODE` LowCardinality(String)
)
ENGINE = MergeTree
PARTITION BY toYear(LO_ORDERDATE)
ORDER BY (LO_ORDERDATE, LO_ORDERKEY)
SETTINGS index_granularity = 8192;

选了Clickhouse社区官方的SSB测试套的lineorder表为例, 他以toYear(LO_ORDERDATE)为分区键, 此时插入一些数据, 在Clickhouse DataPart级别的目录下, 会出现以下文件

其中跟分区相关的有两个: partition.dat和minmax_LO_ORDERDATE.idx

partition.dat存储的是Partition的具体数值, 对应Clickhouse的源码

struct MergeTreePartition
{
    Row value;
  public:
    void load(const MergeTreeData & storage, const DiskPtr & disk, const String & part_path);
    void store(const MergeTreeData & storage, const DiskPtr & disk, const String & part_path, MergeTreeDataPartChecksums & checksums) const;
}

就是我们在system.parts表中看到的partition数值, 其中value的类型是Row, 对应表达式计算后的结果.

其中load函数是读取dat文件的数值到row中, store是存储row数据写入到文件里.

minmax_LO_ORDERDATE.idx存储的是对应字段LO_ORDERDATE的最大值和最小值范围

注意是LO_ORDERDATE的值, 而非toYear(LO_ORDERDATE)的值

对的代码在IMergeTreeDataPart的内部struct

struct MinMaxIndex
{
    /// A direct product of ranges for each key column. See Storages/MergeTree/KeyCondition.cpp for details.
    std::vector<Range> hyperrectangle;
    bool initialized = false;

public:
    MinMaxIndex() = default;

    /// For month-based partitioning.
    MinMaxIndex(DayNum min_date, DayNum max_date)
        : hyperrectangle(1, Range(min_date, true, max_date, true))
        , initialized(true)
    {
    }

    void load(const MergeTreeData & data, const DiskPtr & disk_, const String & part_path);
    void store(const MergeTreeData & data, const DiskPtr & disk_, const String & part_path, Checksums & checksums) const;
    void store(const Names & column_names, const DataTypes & data_types, const DiskPtr & disk_, const String & part_path, Checksums & checksums) const;

    void update(const Block & block, const Names & column_names);
    void merge(const MinMaxIndex & other);
};

与partition.dat类似, 他们都有load和store方法, 但是MinMaxIndex额外有update和merge, 因为DDL的SQL并不会改变partition, 但一个Delete where的语句有可能改变了上下界.

举例总结

举个例子来说, 如果一组数据, 只有3个值, 其中 LO_ORDERDATE列为: 1992-01-01,1992-06-02和 1992-12-01, 那么partition.dat存储的值就是toYear(LO_ORDERDATE)的数值, 也就是1992; 而minmax_LO_ORDERDATE.idx存储的是1992-01-01和1992-12-01, 表示这个区间的上下确界.