Create an ETL pipeline
To create an ETL pipeline, follow the steps below.
To go to the ETL pipeline template, select the "ETL pipeline" under the Data management menu.
On the Data management>ETL pipeline page, click the [+ ETL pipeline] button.
The ETL pipeline page displays with the Details tab preselected. The Details tab contains the details about the ETL pipeline and the data to be exchanged.
To name the ETL pipeline, in the Details tab, enter a unique ETL pipeline name.
Optionally enter a description for the ETL pipeline.
The Execution Log Entity is the Entity where the logs of the execution details will be stored. The format for this log entity is predefined and cannot be edited. Users can create a new Entity or select an existing entity from the drop-down menu.
To select an execution log, select the log from the list of values in the Details tab.
To create an execution log, in the Details tab:
Click the [+] button.
Enter execution log name.
To save the execution log, click the [Save] button.
The Data source section defines the details necessary for the Langstack application to connect to the data source. This is the location from where the data is read. The data source can be an Entity or a Connector.
To define the data source as an Entity, under the Details>Data source>Entity tab, select the required Entity from the list of values.
To select an existing Connector, under the Details>Data source>Connector tab, select the required connector from the list.
To define a new Connector, click the [+] button under Data Source>Connector tab and enter the relevant details. (click here to follow the steps to create a connector)
The Data destination section defines the details necessary for the Langstack system to connect to the data destination. The data destination can be an Entity or a Connector.
To define the data destination as an Entity, under the Details>Data destination>Entity tab, select the required Entity from the list of values.
To select an existing Connector, under the Details>Data destination>Connector tab, select the required Connector from the list.
To define a new Connector, click the [+] button under Data destination>Connector tab and enter the relevant details. (click here to follow the steps to create a connector)
For settings related to multiple executions, time duration settings, Linked App, and Execution Status sections, click here.
To align the Data source and Data destination fields, define the relevant settings in the Data Format tab as needed.
To save the ETL pipeline, click the [Save] button.
Last updated
