My Incremental Load Logic using context variable(with maxid) in Talend
I faced one problem for making my value as variable. below is the scenario.
I am doing my incremental load job and this will be based on one auto incremented column. I need to get the data from my source based on last inserted id in my target database ( After full load execution ). For this I need to get the max value for that id from target database and then i need to use that value in my select statement with greater than condition in Source Database.
To Resolve this I have created Context variable .
Then get the max id from source database and stored in variable.
Used this variable in my Select Statement for getting only Delta(new) data
Steps:
I faced one problem for making my value as variable. below is the scenario.
I am doing my incremental load job and this will be based on one auto incremented column. I need to get the data from my source based on last inserted id in my target database ( After full load execution ). For this I need to get the max value for that id from target database and then i need to use that value in my select statement with greater than condition in Source Database.
To Resolve this I have created Context variable .
Then get the max id from source database and stored in variable.
Used this variable in my Select Statement for getting only Delta(new) data
Steps:
- Created contextvariable
- Created MysqlInput job to query for max row as maxidfield
- Created tJavaRow job to assign the variable context.setProperty("contextvariable",row1.maxidfield)
- Then Run, MySqlInput job with an amended query: "SELECT ... FROM .... where id > "+(String)((String)context.getProperty("contextvariable"))
