Reader
Last updated
Last updated
Records can be read from Langstack Entities or external data sources (using Connectors) through the settings defined in the ETL pipeline>Data Format>Reader tab.
If the Data source is a connector:
Based on the Connector type selected in the Details tab, users can define the reader stream as CSV Stream, Database Stream, JSON Stream, FixedWidth Stream, or Text Stream.
When the steam is selected, and the relevant settings are defined, the reader fields are copied to the Source fields. Additional fields can be added by clicking the [+ Field] button.
The fields in this section are then aligned with destination fields using Field Mapping. The Reader fields can be all of the fields defined in the source or a few fields depending on the information required.
Users can define match keys in the Filter Match key section. A match key can be added by clicking the [+ Match key] button.
The match key is then defined by selecting a field from the data source and the relevant source.
If the Data source is an entity:
The Reader tab displays the Source fields as the three columns of the Entity:_modified_date, _created_date, and _id.
The Filter Match key section is displayed with the following fields as per their execution sequence:
Order By: Users can define ordering the records as ascending or descending for the match key.
Skip: the number of records that are skipped before adding the records to the result.
Limit: Users can define the limit for the number of records in the result. This number can be either a constant value or looked up from a variable or a function. If no value is specified, it means there is no limit on the number of records in the results.
A match key can be added by clicking the [+ Match key] button.
