Class/Object

io.smartdatalake.workflow.dataobject

JdbcTableDataObject

Related Docs: object JdbcTableDataObject | package dataobject

Permalink

case class JdbcTableDataObject(id: DataObjectId, createSql: Option[String] = None, preSql: Option[String] = None, postSql: Option[String] = None, schemaMin: Option[StructType] = None, table: Table, jdbcFetchSize: Int = 1000, connectionId: ConnectionId, jdbcOptions: Map[String, String] = Map(), metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry) extends TransactionalSparkTableDataObject with Product with Serializable

DataObject of type JDBC. Provides details for an action to access tables in a database through JDBC.

id

unique name of this data object

createSql

DDL-statement to be executed in prepare phase

preSql

SQL-statement to be executed before writing to table

postSql

SQL-statement to be executed after writing to table

schemaMin

An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.

table

The jdbc table to be read

jdbcFetchSize

Number of rows to be fetched together by the Jdbc driver

connectionId

Id of JdbcConnection configuration

jdbcOptions

Any jdbc options according to https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html. Note that some options above set and override some of this options explicitly.

Linear Supertypes
Serializable, Serializable, Product, Equals, TransactionalSparkTableDataObject, CanWriteDataFrame, TableDataObject, SchemaValidation, CanCreateDataFrame, DataObject, SmartDataLakeLogger, ParsableFromConfig[DataObject], SdlConfigObject, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JdbcTableDataObject
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. TransactionalSparkTableDataObject
  7. CanWriteDataFrame
  8. TableDataObject
  9. SchemaValidation
  10. CanCreateDataFrame
  11. DataObject
  12. SmartDataLakeLogger
  13. ParsableFromConfig
  14. SdlConfigObject
  15. AnyRef
  16. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JdbcTableDataObject(id: DataObjectId, createSql: Option[String] = None, preSql: Option[String] = None, postSql: Option[String] = None, schemaMin: Option[StructType] = None, table: Table, jdbcFetchSize: Int = 1000, connectionId: ConnectionId, jdbcOptions: Map[String, String] = Map(), metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

    Permalink

    id

    unique name of this data object

    createSql

    DDL-statement to be executed in prepare phase

    preSql

    SQL-statement to be executed before writing to table

    postSql

    SQL-statement to be executed after writing to table

    schemaMin

    An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.

    table

    The jdbc table to be read

    jdbcFetchSize

    Number of rows to be fetched together by the Jdbc driver

    connectionId

    Id of JdbcConnection configuration

    jdbcOptions

    Any jdbc options according to https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html. Note that some options above set and override some of this options explicitly.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. val connectionId: ConnectionId

    Permalink

    Id of JdbcConnection configuration

  7. def createReadSchema(writeSchema: StructType)(implicit session: SparkSession): StructType

    Permalink

    Creates the read schema based on a given write schema.

    Creates the read schema based on a given write schema. Normally this is the same, but some DataObjects can remove & add columns on read (e.g. KafkaTopicDataObject, SparkFileDataObject) In this cases we have to break the DataFrame lineage und create a dummy DataFrame in init phase.

    Definition Classes
    CanCreateDataFrame
  8. val createSql: Option[String]

    Permalink

    DDL-statement to be executed in prepare phase

  9. def dropTable(implicit session: SparkSession): Unit

    Permalink
    Definition Classes
    JdbcTableDataObject → TableDataObject
  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  11. def factory: FromConfigFactory[DataObject]

    Permalink

    Returns the factory that can parse this type (that is, type CO).

    Returns the factory that can parse this type (that is, type CO).

    Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

    returns

    the factory (object) for this class.

    Definition Classes
    JdbcTableDataObject → ParsableFromConfig
  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  14. def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    Permalink

    Handle class cast exception when getting objects from instance registry

    Handle class cast exception when getting objects from instance registry

    Attributes
    protected
    Definition Classes
    DataObject
  15. def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    DataObject
  16. def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession): DataFrame

    Permalink
    Definition Classes
    JdbcTableDataObject → CanCreateDataFrame
  17. def getPKduplicates(implicit session: SparkSession): DataFrame

    Permalink
    Definition Classes
    TableDataObject
  18. def getPKnulls(implicit session: SparkSession): DataFrame

    Permalink
    Definition Classes
    TableDataObject
  19. def getPKviolators(implicit session: SparkSession): DataFrame

    Permalink
    Definition Classes
    TableDataObject
  20. val id: DataObjectId

    Permalink

    unique name of this data object

    unique name of this data object

    Definition Classes
    JdbcTableDataObject → DataObject → SdlConfigObject
  21. def init(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession): Unit

    Permalink

    Initialize callback before writing data out to disk/sinks.

    Initialize callback before writing data out to disk/sinks.

    Definition Classes
    CanWriteDataFrame
  22. implicit val instanceRegistry: InstanceRegistry

    Permalink
  23. def isDbExisting(implicit session: SparkSession): Boolean

    Permalink
    Definition Classes
    JdbcTableDataObject → TableDataObject
  24. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  25. def isPKcandidateKey(implicit session: SparkSession): Boolean

    Permalink
    Definition Classes
    TableDataObject
  26. def isTableExisting(implicit session: SparkSession): Boolean

    Permalink
    Definition Classes
    JdbcTableDataObject → TableDataObject
  27. val jdbcFetchSize: Int

    Permalink

    Number of rows to be fetched together by the Jdbc driver

  28. val jdbcOptions: Map[String, String]

    Permalink

    Any jdbc options according to https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html.

    Any jdbc options according to https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html. Note that some options above set and override some of this options explicitly.

  29. lazy val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
  30. val metadata: Option[DataObjectMetadata]

    Permalink

    Additional metadata for the DataObject

    Additional metadata for the DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  31. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  32. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  33. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  34. def postRead(implicit session: SparkSession): Unit

    Permalink

    Runs operations after reading from DataObject

    Runs operations after reading from DataObject

    Definition Classes
    DataObject
  35. val postSql: Option[String]

    Permalink

    SQL-statement to be executed after writing to table

  36. def postWrite(implicit session: SparkSession): Unit

    Permalink

    Runs operations after writing to DataObject

    Runs operations after writing to DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  37. def preRead(implicit session: SparkSession): Unit

    Permalink

    Runs operations before reading from DataObject

    Runs operations before reading from DataObject

    Definition Classes
    DataObject
  38. val preSql: Option[String]

    Permalink

    SQL-statement to be executed before writing to table

  39. def preWrite(implicit session: SparkSession): Unit

    Permalink

    Runs operations before writing to DataObject

    Runs operations before writing to DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  40. def prepare(implicit session: SparkSession): Unit

    Permalink

    Prepare & test DataObject's prerequisits

    Prepare & test DataObject's prerequisits

    This runs during the "prepare" operation of the DAG.

    Definition Classes
    JdbcTableDataObject → DataObject
  41. val schemaMin: Option[StructType]

    Permalink

    An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.

    An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.

    Definition Classes
    JdbcTableDataObject → SchemaValidation
  42. def streamingOptions: Map[String, String]

    Permalink
    Definition Classes
    CanWriteDataFrame
  43. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  44. var table: Table

    Permalink

    The jdbc table to be read

    The jdbc table to be read

    Definition Classes
    JdbcTableDataObject → TableDataObject
  45. var tableSchema: StructType

    Permalink
    Definition Classes
    TableDataObject
  46. def toStringShort: String

    Permalink
    Definition Classes
    DataObject
  47. def validateSchemaMin(df: DataFrame): Unit

    Permalink

    Validate the schema of a given Spark Data Frame df against schemaMin.

    Validate the schema of a given Spark Data Frame df against schemaMin.

    df

    The data frame to validate.

    Definition Classes
    SchemaValidation
    Exceptions thrown

    SchemaViolationException is the schemaMin does not validate.

  48. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  49. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  50. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  51. def writeDataFrame(df: DataFrame, partitionValues: Seq[PartitionValues])(implicit session: SparkSession): Unit

    Permalink
    Definition Classes
    JdbcTableDataObject → CanWriteDataFrame
  52. def writeStreamingDataFrame(df: DataFrame, trigger: Trigger, options: Map[String, String], checkpointLocation: String, queryName: String, outputMode: OutputMode = OutputMode.Append)(implicit session: SparkSession): StreamingQuery

    Permalink

    Write Spark structured streaming DataFrame The default implementation uses foreachBatch and this traits writeDataFrame method to write the DataFrame.

    Write Spark structured streaming DataFrame The default implementation uses foreachBatch and this traits writeDataFrame method to write the DataFrame. Some DataObjects will override this with specific implementations (Kafka).

    df

    The Streaming DataFrame to write

    trigger

    Trigger frequency for stream

    checkpointLocation

    location for checkpoints of streaming query

    Definition Classes
    CanWriteDataFrame

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from TransactionalSparkTableDataObject

Inherited from CanWriteDataFrame

Inherited from TableDataObject

Inherited from SchemaValidation

Inherited from CanCreateDataFrame

Inherited from DataObject

Inherited from SmartDataLakeLogger

Inherited from ParsableFromConfig[DataObject]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped