Google Drive connector as a destination

The ETL pipeline, to transfer customer data from a Langstack entity to a csv file on Google Drive, is created as follows.

To see step-by-step instructions of how to create an ETL pipeline, click here.

  • An ETL pipeline named โ€œTestEntitytoGDriveโ€ is created.

  • To connect to the data source, the necessary information is added in the Data source section:

    1. The โ€œEntityโ€ tab is selected.

    2. The โ€œTest_Customers_Entityโ€ entity is selected from the drop-down menu.

  • To connect to the data destination, the necessary information is added in the Data destination section:

    1. The โ€œConnectorโ€ tab is selected.

    2. For this example, โ€œGDrive123โ€ is selected from the drop-down menu. The connector can be added by selecting a connector from the drop-down menu or a new connector can be created by clicking the [+] button.

    3. To go to the settings, click the โ€œEdit the settingsโ€ arrow.

  • To disallow multiple simultaneous runs of the ETL pipeline, the toggle button is left enabled for โ€œskip execution while in progressโ€. Enabling this toggle button defines that the execution of this ETL pipeline will be skipped when there is one already in progress.

  • The default selection for ETL pipeline execution is โ€œImmediateโ€.

  • To align the source fields with destination fields, the settings for the reader and writer format are defined in the โ€œData Formatโ€ tab. The โ€œReaderโ€ tab is selected by default. For this example, the fields are added as per the image below.

  • To update the settings for how the data should be written, select the โ€œWriterโ€ tab.

    1. The writer stream is โ€œCSV Stream.โ€

    2. To add the table name, click on the โ€œEdit the settingsโ€ arrow.

  • To add details necessary to write the records, the settings in this section โ€œGoogle drive CSV format details:โ€ are defined as follows:

    1. In the field โ€œFile path or File URL or Folder path or Folder URLโ€ the Google drive folder name is entered "GDriveTest".

    2. The โ€œFile nameโ€ is entered as โ€œcustomersDataโ€.

    3. The โ€œCharacterSetโ€ is selected as โ€œUnicode(UTF-8)โ€.

    4. The โ€œLanguageโ€ is selected as โ€œEnglishโ€.

    5. The โ€œStart reading CSV from lineโ€ is defined as โ€œ1โ€.

    6. The โ€œseparatorโ€ is selected as โ€œCommaโ€.

  • In the โ€œSample dataโ€, the column names of the entity are pasted: โ€œCustomer_ID,First,Last,Countryโ€.

  • Once the sample data is added, the writer fields are displayed:

    • The column names are copied to Field Mapping by clicking โ€œCopy to Field Mappingโ€.

    • Click on "Accept & Collapse" to save the information.

  • The โ€œWriterโ€ mode is โ€œAppendโ€.

  • In the Field Mapping section, all the โ€œMapped Fieldsโ€ are aligned.

  • When the ETL pipeline is executed (after Save and Publish), the records will be added to the destination.

Last updated