Skip to main content
Version: Next

Fivetran

Incubating

Important Capabilities

CapabilityStatusNotes
Platform InstanceEnabled by default

This plugin extracts fivetran users, connectors, destinations and sync history. This plugin is in beta and has only been tested on Snowflake connector.

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[fivetran]'

Config Details

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
fivetran_log_config 
FivetranLogConfig
Fivetran log connector destination server configurations.
fivetran_log_config.destination_platform
string
The destination platform where fivetran connector log tables are dumped.
Default: snowflake
fivetran_log_config.destination_config
DestinationConfig
If destination platform is 'snowflake', provide snowflake configuration.
fivetran_log_config.destination_config.account_id 
string
Snowflake account identifier. e.g. xy12345, xy12345.us-east-2.aws, xy12345.us-central1.gcp, xy12345.central-us.azure, xy12345.us-west-2.privatelink. Refer Account Identifiers for more details.
fivetran_log_config.destination_config.database 
string
The fivetran connector log database.
fivetran_log_config.destination_config.log_schema 
string
The fivetran connector log schema.
fivetran_log_config.destination_config.authentication_type
string
The type of authenticator to use when connecting to Snowflake. Supports "DEFAULT_AUTHENTICATOR", "OAUTH_AUTHENTICATOR", "EXTERNAL_BROWSER_AUTHENTICATOR" and "KEY_PAIR_AUTHENTICATOR".
Default: DEFAULT_AUTHENTICATOR
fivetran_log_config.destination_config.bucket_duration
Enum
Size of the time window to aggregate usage stats.
Default: DAY
fivetran_log_config.destination_config.connect_args
object
Connect args to pass to Snowflake SqlAlchemy driver
fivetran_log_config.destination_config.end_time
string(date-time)
Latest date of lineage/usage to consider. Default: Current time in UTC
fivetran_log_config.destination_config.options
object
Any options specified here will be passed to SQLAlchemy.create_engine as kwargs.
fivetran_log_config.destination_config.password
string(password)
Snowflake password.
fivetran_log_config.destination_config.private_key
string
Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n' if using key pair authentication. Encrypted version of private key will be in a form of '-----BEGIN ENCRYPTED PRIVATE KEY-----\nencrypted-private-key\n-----END ECNCRYPTED PRIVATE KEY-----\n' See: https://docs.snowflake.com/en/user-guide/key-pair-auth.html
fivetran_log_config.destination_config.private_key_password
string(password)
Password for your private key. Required if using key pair authentication with encrypted private key.
fivetran_log_config.destination_config.private_key_path
string
The path to the private key if using key pair authentication. Ignored if private_key is set. See: https://docs.snowflake.com/en/user-guide/key-pair-auth.html
fivetran_log_config.destination_config.role
string
Snowflake role.
fivetran_log_config.destination_config.scheme
string
Default: snowflake
fivetran_log_config.destination_config.start_time
string(date-time)
Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.
fivetran_log_config.destination_config.username
string
Snowflake username.
fivetran_log_config.destination_config.warehouse
string
Snowflake warehouse.
fivetran_log_config.destination_config.oauth_config
OAuthConfiguration
oauth configuration - https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-with-oauth
fivetran_log_config.destination_config.oauth_config.authority_url 
string
Authority url of your identity provider
fivetran_log_config.destination_config.oauth_config.client_id 
string
client id of your registered application
fivetran_log_config.destination_config.oauth_config.provider 
Enum
Identity provider for oauth.Supported providers are microsoft and okta.
fivetran_log_config.destination_config.oauth_config.client_secret
string(password)
client secret of the application if use_certificate = false
fivetran_log_config.destination_config.oauth_config.encoded_oauth_private_key
string
base64 encoded private key content if use_certificate = true
fivetran_log_config.destination_config.oauth_config.encoded_oauth_public_key
string
base64 encoded certificate content if use_certificate = true
fivetran_log_config.destination_config.oauth_config.scopes
array(string)
fivetran_log_config.destination_config.oauth_config.use_certificate
boolean
Do you want to use certificate and private key to authenticate using oauth
Default: False
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
env
string
The environment that all assets produced by this connector belong to
Default: PROD
connector_patterns
AllowDenyPattern
Regex patterns for connectors to filter in ingestion.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
connector_patterns.allow
array(string)
connector_patterns.deny
array(string)
connector_patterns.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
destination_to_platform_instance
map(str,PlatformDetail)
destination_to_platform_instance.key.platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
destination_to_platform_instance.key.env
string
The environment that all assets produced by DataHub platform ingestion source belong to
Default: PROD
sources_to_platform_instance
map(str,PlatformDetail)
sources_to_platform_instance.key.platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
sources_to_platform_instance.key.env
string
The environment that all assets produced by DataHub platform ingestion source belong to
Default: PROD
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Airbyte Stateful Ingestion Config.
stateful_ingestion.enabled
boolean
The type of the ingestion state provider registered with datahub.
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Code Coordinates

  • Class Name: datahub.ingestion.source.fivetran.fivetran.FivetranSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Fivetran, feel free to ping us on our Slack.