Harvesting On Premise Data Sources

Harvesting is a process that extracts technical metadata from your data source into your data catalog. This tutorial provides the steps to harvest from an on premise Oracle Database data source that is connected to Oracle Cloud Infrastructure using a Virtual Cloud Network (VCN).

In this tutorial, you:

  1. Create the polices needed to harvest on premise data sources.
  2. Obtain the on premise database access details.
  3. Create a private endpoint in Data Catalog.
  4. Attach the private endpoint to your data catalog.
  5. Create a data asset.
  6. Add a connection for the data asset.
  7. Harvest the data asset.

For additional information, see configuring a private network.

Before You Begin

To successfully perform this tutorial, you must have the following:

Before you can harvest an on premise data source, you must connect your on premise data source to Oracle Cloud Infrastructure.

Connecting an On Premise Data Source to Oracle Cloud Infrastructure

1. Creating Access Policies for Network Resouces
2. Creating a Virtual Cloud Network
3. Creating a Dynamic Routing Gateway
4. Attaching the DRG to the VCN
5. Updating the Route Table
6. Creating a DHCP Option
7. (Optional) Creating a NAT Gateway
8. Creating Security Rules
9. Creating a Private Subnet

1. Creating Access Policies

To configure Data Catalog to access the private network of a data source, you need access to networking and data catalog resources.

If you already have access to perform all Data Catalog and Networking operations in your required compartments, you may skip this step.

To create the policy needed to configure a private network in data catalog, perform the following steps:

  1. Open the Console navigation menu and then select Policies under Identity.
  2. Click Create Policy.
  3. In the Create Policy panel, enter a unique name for the policy. The name must be unique across all policies in your tenancy. You cannot change this later. For example, data-catalog-private-endpoint-policy.
  4. Next, enter a description such as Grant permissions to create private endpoint and then select Keep Policy Current.
  5. Under Policy Statements field, enter the following policy rule.
    allow group data-catalog-users to manage data-catalog-private-endpoints in tenancy
    Note

    This policy allows users in the data-catalog-users group to perform all data catalog private endpoint operations in any compartment in the tenancy.
  6. Click + Another Statement.
  7. Enter the following policy rule.
    allow group data-catalog-users to manage virtual-network-family in tenancy
    Note

    This policy allows users in the data-catalog-users group to perform all network related operations in any compartment in the tenancy.
  8. Click Create.
You have successfully created the policies to access the required resources for configuring a private network in Data Catalog.

2. Obtaining Data Source Details

You need the private network and database connection information for the on premise Oracle Database you want to harvest.

Obtain the following details for the on premise Oracle Database from your administrator:

  • For configuring the private network, you need the VCN and subnet name and the URL for the Oracle Database.
  • For creating the data asset, you need the Oracle Database host, port, and database service name or SID.
  • For adding a connection, you need the database login credentials.

3. Creating a Private Endpoint

You create a Data Catalog private endpoint to configure the network access details for the on premise Oracle Database data source you want to harvest.

To create a private endpoint in Data Catalog, perform the following steps:

  1. From the navigation menu in the Console, under Data and AI, click Data Catalog.
  2. Click Private Endpoints. The Private Endpoints page displays.
  3. Click Create Private Endpoint. A Create Private Endpoint panel displays.
  4. Ensure you have permission to work in the selected compartment, and enter a name for the private endpoint. For example, XYZ Private Endpoint.
  5. Select the VCN and subnet that is used to connect your on premise Oracle Database to Oracle Cloud Infrastructure.
  6. Enter the DNS zone for the Oracle Database. Use a comma to enter more than one data source DNS zone.
  7. Click Create.
The private endpoint is being created. The create process can take a couple of minutes. When the private endpoint is created successfully, the private endpoint is in ACTIVE status.

If the private endpoint status changes to FAILED, ensure you have the created the access policies and set up your private network correctly.

4. Attaching a Private Endpoint

You attach a private endpoint to a data catalog to allow data assets to be created for data sources available in the private network.

To attach a private endpoint to a data catalog, perform the following steps:

  1. Click Data Catalogs.
  2. Click the Actions icon (three dots) for the data catalog where you want to attach the private endpoint and select Attach Private Endpoint.
  3. Select the private endpoint you created in the previous step and click Attach.
    Image shows the Attach Private Endpoint dialog in the Console
The data catalog status changes to Updating and the private endpoint is being attached. After the private endpoint is attached successfully, the status of the data catalog changes to Active.

5. Creating a Data Asset

You are now ready to register your on premise Oracle Database data source with Data Catalog as a data asset .

To create an Oracle Database data asset, perform the following steps:

  1. Click the data catalog instance where you attached the private endpoint in the previous step.
  2. From your data catalog Home tab, click Create Data Asset from the Quick Actions tile.
  3. In the Create Data Asset panel, enter a Name to uniquely identify your data asset. Optionally, enter a description.
  4. From the Type drop-down list, select Oracle Database.
  5. In the Host field, enter the database hostname.
  6. In the Port field, enter the database port.
  7. In the Database field, enter the database service name or SID.
  8. Select the Use private endpoint checkbox.
  9. Click Create.
You have successfully created your Oracle Database data asset.

6. Adding a Connection

After creating the Oracle Database data asset, you add a connection for the data asset.

To add a connection for your Oracle Database data asset, perform the following steps:

  1. From the data catalog Home tab, click Data Assets to access the Data Assets page.
  2. In the Data Assets list, select the Oracle Database data asset you created previously.
  3. In the Summary tab on the data asset details page, under Connection Information, click Add Connection.
  4. In the Add Connection panel, enter a unique name for your connection. Optionally, enter a short description.
  5. Select JDBC for Type.
  6. Enter the Username and Password to access the Oracle Database.
  7. Click Test Connection. A notification displays indicating if the test connection was successful or failed.
  8. Click Add.
You have successfully created a connection for your Oracle Database data asset.

7. Harvesting the Data Asset

You are now ready to harvest your Oracle Database data asset.

To harvest your Oracle Database data asset, perform the following steps:

  1. Click Harvest on the data asset details page for the data asset.
  2. The Select Connection page displays and the default connection is selected. Click Next.
  3. The Select Data Entities page displays. View and add all the data entities you want to harvest from the Available Oracle Schema section.
    1. Click the add icon for each data entity you want to include in the harvest job.
    2. Click Add All to select all the entities for harvesting.
    3. Use the Filter Oracle Schema box to find a data entity from the available data entities.
    4. Use the page navigation icons to browse all the data entities.
    5. Click the remove icon for any selected data entity that you want to remove from the harvest job.
    6. If you need to start over, click Remove All and then start over.
    After you have reviewed the data entities you want to harvest from the Selected Oracle Schema / Data Entities section, click Next.
  4. The Create Job page displays. In the Job Name field, enter a unique name to identify the harvest job.
  5. Optionally, enter a Description.
  6. Select Run job now and then click Create Job.
  7. The job to harvest your Oracle Database data asset is created successfully and the Jobs tab displays. Click the job name to view job details.
Your data asset is harvested successfully and you can review the harvest job details.