Managing Metadata

In this section, we will understand how we are managing the Metadata in the Talend studio for a data integration platform.

The Metadata folder is used to store the reusable information on Files, databases, or the systems, which is needed to create our jobs in data integration Talend studio.

This section includes the process of creating and managing the multiple Metadata items in the Repository panel, which can be used in all our job designing.

Go to the Repository panel, and click on the Metadata, as we can see in the below image:

In Metadata, we can set up the following connection in the data integration Talend Studio:

DB connection
- MySQL DB connection
- JDBC DB connection
File schema
- File delimited
- File positional
- File Regex
- File XML
- File Excel
- File ldif
- File JSON
LDAP
Azure Storage
Google Drive
Marketo
Salesforce
Snowflake
Generic schemas
Talend MDM
Rules Management
Web Service
FTP

We will learn mostly used Metadata connections and schemas to give you a complete understanding of managing the Metadata in Talend studio for a data integration platform.

Db Connection

We must be reading and writing data or both with the help of Databases. Talend studio provides a database components to do these tasks.

In Talend studio's Metadata section, we can connect various types of databases like MySQL, JDBC, SAS, Oracle, SAP Hana, Hive, Teradata, Redshift, Amazon Aurora, Microsoft SQL Server, and so on.

File Schemas

To read and write data into the delimited Files, we have different File schemas like File Delimited, Excel, XML, Positional, Regex, LDIF, and JSON with having different components in Repository's Metadata, which we can use while creating a Job in Talend studio.

File Schemas	Description
File delimited	It is a text File, which is used to store data where each line has fields separated by the delimiter.
Excel	To read and write data to a reliable Excel spreadsheet File, we will centralize the connection to the File, along with its data structure.
XML	It is an Extensible Markup Language, which is used to store and transport the data. After centralized the Metadata of XML input and output File, we can directly use the Metadata in our jobs.
Positional	It is a File that has a fixed maximum length and records. The File positional Metadata is used to define the properties of different components like tFileInputPositional ,tFileOutputPositional, and tFileInputMSPositional components.
Regex	This kind of File schemas is made of regular expressions, like log Files.
LDIF	The LDIF File is a directory File which is expressed by attribute.
JSON	It is known as JavaScript Object Notation that is used to serialize and transmit the structured data over a network connection. And it also transfers the data between the server and the web application.

LDAP

LDAP stands for Lightweight Directory Access Protocol that is used to access and manage the directory information, and also reads and edits the directories over the IP networks.

Azure Storage

Azure Storage is a service, which allows us to create a disk in the Disk storage for our virtual machine. The disk can be accessed from one virtual machine only.

Marketo

It is a marketing automation software based on SaaS. It is made to help the companies to automate and measure marketing engagement, tasks, and workflows.

We can also connect multiple data sources in the Repository's Metadata in Talend Studio, and we can explore many characteristics of our data sources.

Talend allows us to use these data sources in our jobs, by dragging an object from the Repository panel to the design workspace window.

Snowflake

The Snowflake is used to store and analyze the data with the help of cloud-based hardware and software. And it is the only data platform built for the cloud where we can store all our data.

Generic schemas

If none of the specific Metadata matches our need or if we do not have any source File to take the schema form, we can create a generic schema in the Talend studio.

And it is used where we do not want to limit the usage of schemas, which is related to any File type or database.

Talend MDM

The Talend MDM is used to combine the real-time data, applications, and integration processes with the embedded data quality to share across on-premises, cloud, and mobile apps.

Web Service

The data integration's Talend studio allows many components to call multiple types of web services like tWebServiceInput, tWebservice [advanced component based on tWebServiceInput], tSOAP [for SOAP Web services] and tREST [for REST Web services].