Configuring a Private Network

You can configure your Data Flow Application to access data sources hosted in private networks.

By configuring an Application to access a private network, you can:

  • Access Oracle Cloud Infrastructure data sources that are only accessible privately.
  • Access on-premise data sources that are connected to an Oracle Cloud Infrastructure Virtual Cloud Network (VCN) using VPN Connect or FastConnect.
Important

You can access on premise or private data sources in Data Flow using only the

DNS name of the resources the Run needs to access. You can't access a data source using the direct IP address. Also, it is currently not supported, nor if the resource you want to access is an Oracle database behind SCAN proxy.

To allow your Application to access a private network, you must:

  1. Have met the prerequisites for creating, editing, managing and using private endpoints.
  2. Create a private endpoint for your Application to use, if one doesn't already exist.
  3. Attach the private endpoint to your Application.
  4. Use the private endpoint while you Run an Application.

Prerequisites

Oracle Cloud Infrastructure lets you configure private access for your resources using private endpoints.

Data Flow uses private endpoints to access the private network where your data sources are hosted. You must have the required set up to use the Data Flow private endpoints.

Also, to create, update, or delete private endpoints in Oracle Cloud Infrastructure, you need to obtain certain permissions in Oracle Cloud Infrastructure Identity and Access Management (IAM) for the relevant compartments in your tenancy. The following table lists the required permissions for virtual networking resources in Oracle Cloud Infrastructure for the private endpoint operations.

Operation Required Access on Underlying Resources
Create a private endpoint

For the private endpoint compartment:

  • Create VNIC (VNIC_CREATE)
  • Delete VNIC (VNIC_DELETE)
  • (Optional) Update members in a network security group (NETWORK_SECURITY_GROUP_UPDATE_MEMBERS)
  • (Optional) Associate a network security group (VNIC_ASSOCIATE_NETWORK_SECURITY_GROUP)

For the subnet compartment:

  • Attach subnet (SUBNET_ATTACH)
  • Detach subnet (SUBNET_DETACH)
Update a private endpoint

For the private endpoint compartment:

  • Update VNIC (VNIC_UPDATE)
  • (Optional) Update members in a network security group (NETWORK_SECURITY_GROUP_UPDATE_MEMBERS)
  • (Optional) Associate a network security group (VNIC_ASSOCIATE_NETWORK_SECURITY_GROUP)
Delete a private endpoint

For the private endpoint compartment:

  • Delete VNIC (VNIC_DELETE)
  • (Optional) Update members in a network security group, if using one (NETWORK_SECURITY_GROUP_UPDATE_MEMBERS)

For the subnet compartment:

  • Detach subnet (SUBNET_DETACH)
Important

If you have the manage work requests permission, you can view the logs and error messages that are encountered while working with private endpoints.

Creating a Private Endpoint

Oracle Cloud Infrastructure lets you create private endpoints within your service so that you can access resources that are only available using private IPs. In Data Flow, you create a private endpoint to configure the private network where your data source is hosted.

Before you create a private endpoint in Data Flow, you must have the prerequisites and the following details:

  • The name of the Virtual Cloud Network (VCN) used to access your data source.
  • The name of the subnet in the VCN.
  • The list of DNS zones used to resolve the Fully Qualified Domain Names (FQDNs) of the data sources that you want to harvest.
    Important

    Specify the domain names or the FQDNs of the data sources in this list. For private autonomous databases, use the FQDN of the database as the DNS zone. For custom data sources running on Oracle Cloud Infrastructure compute Virtual Machines (VMs), you can specify the FQDN of the VM, the domain name of the subnet in which the VM is provisioned, or the domain name of the VCN.

Here's how you create a private endpoint:

  1. In the Console, open the navigation menu, and then under Data and AI, click Data Flow.
  2. From the Data Flow service page, click Private Endpoints.
  3. Click Create Private Endpoint.
  4. Enter a name to identify the private endpoint.
  5. Select the VCN that is created to provide private access to your data source. You can change the compartment where the VCN is.
  6. Select the Subnet that has the private endpoint from which you access your data source. You can change the compartment where the subnet is.
  7. Enter the DNS zones to resolve. The DNS zone is the set of domain name suffixes that are in the allowed list for network address translation by the private endpoint infrastructure. You can enter up to 30 DNS zones. They must be comma delimited.
  8. (Optional) Enter the number of Hosts to Access.
  9. (Optional) Add a Network Security Group from the drop-down list. You can change the compartment where the Network Security Group is.
  10. (Optional) If you have more than one Network Security Group, click Add Another Security Group. Repeat until you've added all the Network Security Groups.
  11. (Optional) Add tags to identify this private endpoint resource.
  12. Click Create.
The private endpoint is created. The create process can take a couple of minutes. When the private endpoint is created successfully, the private endpoint is in INACTIVE status.
Note

You can create up to five private endpoints in a tenancy. But only one can be used at any one time across all Runs in your tenancy that use a private endpoint. It is possible for some Runs in a tenancy to use a private endpoint and some not use a private endpoint at the same time. If you try to use one private endpoint for a new Run in a tenancy whilst another Run in the tenancy is using a different private endpoint, the new Run fails on submission.

Viewing Private Endpoints

All private endpoints created in Data Flow are listed in the Private Endpoints page. To view details for a specific private endpoint, click the private endpoint name. The private endpoint details page displays. Alternatively, click the Actions icon (three dots) for the private endpoint and select View Details.

Also, from the Private Endpoints list, you can edit, move, or delete a private endpoint.

A private endpoint can be in one of the following statuses:

Creating
The private endpoint is being created.
Active
The private endpoint is successfully created and is in use.
Inactive
The private endpoint is successfully created and is ready for use.
Updating
The private endpoint details, such as name, DNS zones, or compartment, are being updated.
Deleting
The private endpoint is being deleted.
Deleted
The private endpoint is successfully deleted.
Failed
The private endpoint was not created, updated, or deleted successfully. It must be deleted and re-created.

Attaching a Private Endpoint

You attach a private endpoint to a Data Flow Application to enable the Application to run using the configured private network. You can attach only one private endpoint to an Application, but you can attach a private endpoint to many Applications.

Before you attach a private endpoint to an Application, you must create a private endpoint.

Here's how you attach a private endpoint to an existing Application.

  1. Edit an Application and, click Show Advanced Options.
  2. In the Choose Network Access section, select the Secure Access to Private Subnet radio button.
  3. Select the private endpoint that you want to attach to the Application from the drop-down list. Click Change Compartment if it is in a different compartment to the Application.
  4. Click Save Changes.

Using a Private Endpoint

When you create an Application in the Data Flow service console, you can Attach a Private Endpoint to it that you have created.

Only users in the dataflow-admin group can create Runs that can either, activate a private endpoint configuration, or switch the network configuration back to Internet. After a Run activates a private endpoint, this private endpoint remains active until changed by a user from the dataflow-admin group with the appropriate privileges. See Set Up Administration for the right set of privileges. A user in the dataflow-users group can launch Runs only if the Application is configured to use the active private endpoint.
Note

When correctly configured, private endpoints can access a mix of private resources on the VCN plus Internet resources. Provide a list of these resources in the DNS Zones section when configuring a private endpoint.

Editing a Private Endpoint

Here's how you edit the private endpoint name and DNS zones:

  1. From the Data Flow page in the Console, click Private Endpoints.
  2. From the Private Endpoints list, click the Actions icon (three dots) for the private endpoint you want to edit and select View Details. Alternatively, you can click the name of the private endpoint to open the private endpoint details page.
  3. Click Edit.
  4. Modify the Name or External DNS Zones to Resolve.
  5. Click Save Changes.

Moving a Private Endpoint

You can move the private endpoint resource from the compartment you created it in to a different compartment.

Here's how you move a private endpoint to a different compartment:

  1. From the Data Flow page in the Console, click Private Endpoints.
  2. From the Private Endpoints list, click the Actions icon (three dots) for the private endpoint you want to move and select Move Resource.
  3. Select the new compartment for the private endpoint resource.
  4. Click Move Resource.
A notification displays indicating that the private endpoint resource is moved to the new compartment successfully. You might notice the private endpoint status change to Updating. After the move is completed successfully, the private endpoint status changes back to Inactive.

Detaching a Private Endpoint

Before you can delete a private endpoint, you must detach it from the Application to which it is attached. Also, if you want to attach a different private endpoint to an Application, you must detach any private endpoint already attached to it.

Here's how you detach a private endpoint from an Application.

  1. From the Applications page in the Console, click the Actions icon (three dots) for the Application where you want to detach a private endpoint and then select Edit.
  2. In the Edit Application panel, scroll to the Network Access section, and either select a different private endpoint, click Internet Access (No Subnet).

Deleting a Private Endpoint

You can delete a private endpoint only if it is not attached to any Application.

Caution

If you try to delete a private endpoint that is still attached to an Application, you receive a warning that the private endpoint can't be deleted. You must detach the private endpoint from the Application before you can delete it.

Here's how you delete a private endpoint:

  1. From the Data Flow page in the Console, click Private Endpoints.
  2. From the Private Endpoints list, click the Actions icon (three dots) for the private endpoint you want to delete and select Delete.
  3. In the Delete Private Endpoint dialog, type DELETE to confirm that you want to delete the private endpoint and then click Delete.
A notification displays indicating that the private endpoint is deleted successfully.
Important

The Application can no longer access the data once you delete the private endpoint.