Class/Object

io.smartdatalake.workflow

FileSubFeed

Related Docs: object FileSubFeed | package workflow

Permalink

case class FileSubFeed(fileRefs: Option[Seq[FileRef]], dataObjectId: DataObjectId, partitionValues: Seq[PartitionValues], isDAGStart: Boolean = false, processedInputFileRefs: Option[Seq[FileRef]] = None) extends SubFeed with Product with Serializable

A FileSubFeed is used to transport references to files between Actions.

fileRefs

path to files to be processed

dataObjectId

id of the DataObject this SubFeed corresponds to

partitionValues

Values of Partitions transported by this SubFeed

processedInputFileRefs

used to remember processed input FileRef's for post processing (e.g. delete after read)

Linear Supertypes
Serializable, Serializable, Product, Equals, SubFeed, DAGResult, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FileSubFeed
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. SubFeed
  7. DAGResult
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FileSubFeed(fileRefs: Option[Seq[FileRef]], dataObjectId: DataObjectId, partitionValues: Seq[PartitionValues], isDAGStart: Boolean = false, processedInputFileRefs: Option[Seq[FileRef]] = None)

    Permalink

    fileRefs

    path to files to be processed

    dataObjectId

    id of the DataObject this SubFeed corresponds to

    partitionValues

    Values of Partitions transported by this SubFeed

    processedInputFileRefs

    used to remember processed input FileRef's for post processing (e.g. delete after read)

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def breakLineage(implicit session: SparkSession, context: ActionPipelineContext): FileSubFeed

    Permalink

    Break lineage.

    Break lineage. This means to discard an existing DataFrame or List of FileRefs, so that it is requested again from the DataObject. This is usable to break long DataFrame Lineages over multiple Actions and instead reread the data from an intermediate table

    Definition Classes
    FileSubFeedSubFeed
  6. def clearDAGStart(): FileSubFeed

    Permalink
    Definition Classes
    FileSubFeedSubFeed
  7. def clearPartitionValues(): FileSubFeed

    Permalink
    Definition Classes
    FileSubFeedSubFeed
  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. val dataObjectId: DataObjectId

    Permalink

    id of the DataObject this SubFeed corresponds to

    id of the DataObject this SubFeed corresponds to

    Definition Classes
    FileSubFeedSubFeed
  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  11. val fileRefs: Option[Seq[FileRef]]

    Permalink

    path to files to be processed

  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  14. val isDAGStart: Boolean

    Permalink
    Definition Classes
    FileSubFeedSubFeed
  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  17. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  18. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. val partitionValues: Seq[PartitionValues]

    Permalink

    Values of Partitions transported by this SubFeed

    Values of Partitions transported by this SubFeed

    Definition Classes
    FileSubFeedSubFeed
  20. val processedInputFileRefs: Option[Seq[FileRef]]

    Permalink

    used to remember processed input FileRef's for post processing (e.g.

    used to remember processed input FileRef's for post processing (e.g. delete after read)

  21. def resultId: String

    Permalink
    Definition Classes
    SubFeed → DAGResult
  22. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  23. def updatePartitionValues(partitions: Seq[String]): FileSubFeed

    Permalink
    Definition Classes
    FileSubFeedSubFeed
  24. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SubFeed

Inherited from DAGResult

Inherited from AnyRef

Inherited from Any

Ungrouped