Live File Processor Agent File – EBatch

<< Click to Display Table of Contents >>

Navigation:  Live > Agents > File Agents > File Processor > Settings > File >

Live File Processor Agent File – EBatch

Live File Processor Agent's data are assigned an ebatch number when loaded into the database. Thus the data's source file is known, and if the data are bad, it can be removed from the database using the EDD Rollback option in EDP.

 

Note: This functionality requires the Live Schema (schema.LIVE.xme) for EQuIS 7.20.2 Build or later, which adds the ebatch columns to the DT_LOGGER tables.

 

 

Setting

Type

Description

ebatchEddFile

Composite String

The ST_EDD_BATCH.EDD_FILE value.

 

Default Value: {folderWatcherPath}/{fileNameWithExtension}

ebatchEddType

Composite String

The ST_EDD_BATCH.EDD_TYPE value.

 

Default Value: Live {processor}

ebatchReuse

Boolean

If True and an ST_EDD_BATCH table row exists where EDD_FILE = ebatchEddFile (and ROLLBACK_DATE is null), then this row is used; else, a new ST_EDD_BATCH table row is added.

 

If False then a new ST_EDD_BATCH table row is added (each time the file changes, regardless of if an ST_EDD_BATCH table row exists where EDD_FILE = ebatchEddFile).

 

Default Value: True

 

 

Example 1 – Path and File Name

 

The ebatchEddFile's default value is {folderWatcherPath}/{fileNameWithExtension}, and:

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

HOBO

folderWatcherPath

C:\EarthSoft\Live\HOBO

 

If a file is named rainfall-2020-05-26.csv, ST_EDD_BATCH.EDD_FILE is C:\EarthSoft\Live\HOBO\rainfall-2020-05-26.csv.

 

The same ebatch number is used for a file (path and name), regardless of how often the file gets data appended to it.

 

 

Example 2 – Path and Part of File Name

 

A date and time, or other info (e.g., everything after the first dash), can be removed from the file name. An example is provided in the table below.

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

HOBO

ebatchEddFile

{folderWatcherPath}/{fileNameWithoutExtension => ebatchEddFileRegexReplacements}.{fileExtension}

HOBO

ebatchEddFileRegexReplacements

^(?'a'[^-.]*).* => ${a}-*

HOBO

folderWatcherPath

C:\EarthSoft\Live\HOBO

 

If a file is named rainfall-2020-05-26.csv, ST_EDD_BATCH.EDD_FILE is C:\EarthSoft\Live\HOBO\rainfall-*.csv.

 

The same ebatch number is used for all files named like rainfall-*.csv.

 

 

Example 3 – Path and File Name and Date

 

A date time variable (e.g., the file's last write date) can be added to the file name. An example is provided in the table below.

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

HOBO

ebatchEddFile

{folderWatcherPath}/{fileNameWithoutExtension}-{fileLastWriteTimeUtc:yyyy-MM-dd}.{fileExtension}

HOBO

folderWatcherPath

C:\EarthSoft\Live\HOBO

 

If a file is named rainfall.csv, ST_EDD_BATCH.EDD_FILE is C:\EarthSoft\Live\HOBO\rainfall-2020-05-26.csv.

 

A new ebatch number is used for file rainfall.csv for each day.

 

 

Example 4 – Not Used

 

If ebatchEddFile is defined and NULL or empty, ebatch is not used.

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

Default

ebatchEddFile


 

 

EDD Rollback

 

The EDD Rollback option in EDP can be used to remove the data, but it fails if all the data for a selected ebatch number cannot be removed. A temporary workaround may be to use an SQL statement (which can be executed using SQL Server Management Studio) to delete the DT_LOGGER_DATUM (e.g., delete from DT_LOGGER_DATUM where EBATCH = 1234), and then delete the DT_LOGGER, DT_LOGGER_SERIES, etc., rows manually.