Skip to content

Latest commit

 

History

History
52 lines (36 loc) · 3.62 KB

tidb-lightning-overview.md

File metadata and controls

52 lines (36 loc) · 3.62 KB
Raw
title summary aliases
TiDB Lightning Overview
Learn about Lightning and the whole architecture.
/docs/dev/tidb-lightning/tidb-lightning-overview/
/docs/dev/reference/tools/tidb-lightning/overview/
/docs/dev/tidb-lightning/tidb-lightning-tidb-backend/
/docs/dev/reference/tools/tidb-lightning/tidb-backend/
/tidb/dev/tidb-lightning-tidb-backend
/docs/dev/loader-overview/
/docs/dev/reference/tools/loader/
/docs/dev/load-misuse-handling/
/docs/dev/reference/tools/error-case-handling/load-misuse-handling/
/tidb/dev/load-misuse-handling
/tidb/dev/loader-overview/
/tidb/dev/tidb-lightning-backends

TiDB Lightning Overview

TiDB Lightning is a tool used for importing data at TB scale to TiDB clusters. It is often used for initial data import to TiDB clusters.

TiDB Lightning supports the following file formats:

TiDB Lightning can read data from the following sources:

Note:

Compared with TiDB Lightning, the IMPORT INTO statement can be directly executed on TiDB nodes, supports automated distributed task scheduling and TiDB Global Sort, and offers significant improvements in deployment, resource utilization, task configuration convenience, ease of invocation and integration, high availability, and scalability. It is recommended that you consider using IMPORT INTO instead of TiDB Lightning in appropriate scenarios.

TiDB Lightning architecture

Architecture of TiDB Lightning tool set

TiDB Lightning supports two import modes, configured by backend. The import mode determines the way data is imported into TiDB.

  • Physical Import Mode: TiDB Lightning first encodes data into key-value pairs and stores them in a local temporary directory, then uploads these key-value pairs to each TiKV node, and finally calls the TiKV Ingest interface to insert data into TiKV's RocksDB. If you need to perform initial import, consider the physical import mode, which has higher import speed. The backend for the physical import mode is local.

  • Logical Import Mode: TiDB Lightning first encodes the data into SQL statements and then runs these SQL statements directly for data import. If the cluster to be imported is in production, or if the target table to be imported already contains data, use the logical import mode. The backend for the logical import mode is tidb.

Import mode Physical Import Mode Logical Import Mode
Backend local tidb
Speed Fast (100~500 GiB/hour) Low (10~50 GiB/hour)
Resource consumption High Low
Network bandwidth consumption High Low
ACID compliance during import No Yes
Target tables Must be empty Can contain data
TiDB cluster version >= 4.0.0 All
Whether the TiDB cluster can provide service during import Limited service Yes

The preceding performance data is used to compare the import performance difference between the two modes. The actual import speed is affected by various factors such as hardware configuration, table schema, and the number of indexes.