How to Optimize tuniqueRow Component in Talend

Java heap space error at tUniqRow in Talend

• To Optimise this component ,Try  to get the unique records by using min number columns in uniq condition.
• var-char(max) ,text data type columns in unique condition will decrease the performance of the component If you are using tuniqrow component with huge data we may get java heap space Exception. 
  (By storing the data on disk while getting unique record from huge data)
  Below are the steps to resolve the issue.  
  
  • Click on tunique
  • Go to component window
  • Select you unique condition columns
  • Go to Advanced Settings
  • Click on Use of Disk Option
  • Then select Directory For temp files. (This Directory Structure must be already available in the System. If it is not available the it wouldn't create any directory. This will give you Error  )
  •  Then Execute the jobBut this will be slow with compared to the speed of the job with out Use of Disk Option for small amount of data. 

How to Optimize Tmap Component in Talend

Optimize Tmap component in Talend


If Lookup table has few rows:

• Open tmap select lookup model as Load Once .This will load lookup data one time before mail flow starts.
• This Lookup data will be stored in memory then main flow execution is very fast with comparing with lookup data in memory.

IF lookup table data is very large:
Talend cannont store the lookup table data in memory. you will get java heap space Execption. To resolve this issue follow the below steps

• open tmap
• go to lookup table
• click on tmap settings
• select the value for store temp data property to True 
• click on ok
• In Tmap properties basic settings  set Temp Data Directory path by browsing folder
• Go to Advance settings
• set  max buffer size(nb of row) to some value based on  lookup condition data type

In my jobs lookup stressful for  2 mill records with 3 GB of memory  of integer data type.