Showing posts with label DI. Show all posts
Showing posts with label DI. Show all posts

Thursday, March 19, 2015

Pentaho DI arguments, parameters, and variables

Arguments
A PDI argument is a named, user-supplied, single-value input given as a command line argument (running a transformation or job manually from Pan or Kitchen, or as part of a script). Each transformation or job can have a maximum of 10 arguments.

Parameters

Parameters are like local variables; they are reusable inputs that apply only to the specific transformation that they are defined in. When defining a parameter, you can assign it a default value to use in the event that one is not fetched for it. This feature makes it unique among dynamic input types in PDI.

Note: If there is a name collision between a parameter and a variable, the parameter will take precedence.

To define a parameter, right-click on the transformation workspace and select Transformation settings from
the context menu (or just press Ctrl-T), then click on the Parameters tab.

Variables
A variable in PDI is a piece of user-supplied information that can be used dynamically and programmatically in a variety of different scopes. A variable can be local to a single step, or be available to the entire JVM that PDI is running in.

PDI variables can be used in steps in both jobs and transformations. You define variables with the Set Variable step in a transformation, by hand through the kettle.properties file, or through the Set Environment Variables dialogue in the Edit menu.

TheGet Variable step can explicitly retrieve a value from a variable, or you can use it in any PDI text field that has the diamond dollar sign icon next to it by using a metadata string in either the Unix or Windows formats:

${VARIABLE}
%%VARIABLE%%

Source: Pentaho user guide

Wednesday, March 18, 2015

Talend tFuzzyMatch component example

https://help.talend.com/images/54/bk-components-rg-542/tFuzzyMatch.png Talend tFuzzyMatch component

Problem: Let say we have a table 'Distributor_Info' and two columns 'distributor_id' and 'address1'.



We want to find the distributor_id which have matching addresses.
I used  tFuzzyMatch to do this task.

Sample Job


Below is component settings


Unique matching check box:
Select this check box if you want to get the best match possible, in case several matches are available.

Matcing Type Drop Down button:

Select the relevant matching algorithm among:

Levenshtein: Based on the edit distance theory. It calculates the number of insertion, deletion or substitution required for an entry to match the reference entry.

Metaphone:
Based on a phonetic algorithm for indexing entries by their pronunciation. It first loads the phonetics of all entries of the lookup reference and checks all entries of the main flow against the entries of the reference flow.

Double Metaphone:
a new version of the Metaphone phonetic algorithm, that produces more accurate results than the original algorithm. It can return both a primary and a secondary code for a string. This accounts for some ambiguous cases as well as for multiple variants of surnames with common ancestry.


Output Excel File (Levenshtein with Min. & Max. Distance 0):


 
                   Output Excel File (Levenshtein with Min. 1 & Max. Distance 100):


                                           Output Excel File (Metaphone):

                                      Output Excel File (Double Metaphone):



 Reference: https://help.talend.com/display/TalendComponentsReferenceGuide54EN/tFuzzyMatch

Monday, March 2, 2015

Pentaho and Talend Data Integration Comparison





Pentaho Talend
1 Data passing form one job to to another is simple then talend






Comparatively complex
2 Direct JavaScirpt support Only Java support


Job scheduling components/step available




























Not available
4 Import/Export schema on component leve is not supported. Very good support




5 Table lookup has limited functionality tMap has multiple join,filtering,reject etc funcitonality
6 Automatic job document creation not supported Supported
7 Multiple logging levels available for job execution


















Single type of log detail available
8 Limited options for job deployment Multiple deployment options available (sh, jar, web service,esb job etc)
9 Support preview of data step by step component wise Not available
10 Bad support of multiple database connection Very good support while working with many database connections
11 Only Main data flow used (between components/steps) Main and Iteration both flow supported
12 Easy to understand main flow Comparatively difficult to understand main & Iterate flow in jobs
13 Pentaho not hangs (responding well) Talend hangs often







Will be adding some more points...come back later