When batch-loading data from a source system into your Data Mart or Data Warehouse, you usually look for ways to reduce the to-be extracted data set to minimize extraction time and resource consumption. Ideally, there is a way to only extract changed records, for example, using a change date column. However, this does not always fly. For example, change date columns are not always reliable or deletions are not always tracked in the source table. An alternative would be to switch from batch to log or trigger replication on the source, but you don’t always have this luxury. So, now you have to load the entire source data set, but still you want to identify the change data, to only process delta’s in downstream data flows. How can you identify such change data using batch loading?
