Quantcast
Channel: SCN : Blog List - SAP LT Replication Server
Viewing all articles
Browse latest Browse all 38

How To filter on the initial load & parallelize replication DMIS 2011 SP06 or higher

$
0
0

Hi SLT community,

 

This How To describe a new expert function released with DMIS 2011 SP6 wthat allows to filter directly in the source for the initial load to reduce the transfer volume (if you have anyway in mind to filter on the SLT system) and also the option for the definition of a parallelized replication.

          

 

Important: DMIS 2011 SP06 or higher

 

Use Case

Use Case: Filter Initial Load

You don't want to load the entire source table but a filter on the SLT is not the optimal solution as most of the data is transferred to the SLT system and skipped here. In case you filter a significant part of a table it's better to already limit the reader module of the initial load accordingly

Ensure that you use a reading type 4/5. Without it will not work.

 

 

Use Case: Parallelize Replication

You need to parallelize the replication of a table as there are high change rates and one parallel task can not catch up delta data within expected latency time.

Process

With DMIS 2011 SP6 a new table is available where Filter for the Initial Load and / or settings to parallelize the replication can be defined. The eintries in table DMC_ACSPL_SELECT have to be defined on the SLT systems

 

Fieldname

Description

MT_ID

Mass Transfer ID

MIGR_OBJ_ALIAS

Table Name

VALIDITY

Validity of this setting
1 = use for initial load and replication
2 = use for initial load only
3 = use for replication only

ACS_PLAN_ID

Access Plan ID
Use ID 00000 for filter criteria of initial load and / or replication
use access plan ID 00001 - 99999 for parallelization settings

FIELD

Fieldname

CURRNUM

Line Number (0001 - 9999) to define multiple records for the same access plan ID

LOGICAL_OPERATOR

Logical Operator (allowed in Open SQL where clause)

VALUE

Filter / parallelization value for current field

GATING_OPERATOR

required if several records for the same field are defined
multiple fields are concatenated with AND automatically

The value has to be defined properly based on the internal format of the corresponding data type. Character-like field values have to be defined with single quotes. NUMC fields have to be defined with single quotes and leading zeros.

 

Process: Filter Initial Load & Replication

You can enter filter criteria for any field of the source table. If you enter mutliple filter criteria for one field you have to define a gating operator (AND / OR). If you define filter on multiple fields the filter from the indiviual fields will be concatenated with gating operator AND.
If you define a filter for the replication also a filter on the DB trigger is recommended as otherwise the changes that don't fit to the filter criteria will not be handled and remain in the logging table.

 

Example 1: Filter on one field for a single value
Define filter criteria for table BKPF in mass transfer ID 001. Only those records should be loaded where GJAHR is greater or equal 2013. The filter criteria shall be relevant for initial load and replication

 

MT_ID

MIGR_OBJ_ALIAS

VALIDITY

ACS_PLAN_ID

FIELD

CURRNUM

LOGICAL_OPERATOR

VALUE

GATING_OPERATOR

001

BKPF

1

00000

GJAHR

0001

>=

2013

 

 

The following where clause will be used for the select:

 

SELECTFROMBKPF WHEREGJAHR >= 2013.

 

Example 2: Filter on one fieldfor mutliple values
Define filter criteria for table BKPF in mass transfer ID 001. Only those records should be loaded where BUKRS is equal 0010 or between 1000 and 3000. The filter criteria shall be relevant for the initial load only

 

MT_ID

MIGR_OBJ_ALIAS

VALIDITY

ACS_PLAN_ID

FIELD

CURRNUM

LOGICAL_OPERATOR

VALUE

GATING_OPERATOR

001

BKPF

2

00000

BUKRS

0001

=

'0010'

OR

001

BKPF

2

00000

BUKRS

0002

>=

'1000'

AND

001

BKPF

2

00000

BUKRS

0003

<=

'3000'

 

 

The following where clause will be used for the select:

 

SELECTFROMBKPF WHEREBUKRS = '0010'OR BUKRS >= '1000'ANDBUKRS <= '3000'.

 

 

Example 3: Filter on multiple fields
Define filter criteria for table BKPF in mass transfer ID 001. Only those records should be loaded where GKAHR is greater or equal 2013 and where BUKRS is equal 0010 or between 1000 and 3000. The filter criteria shall be relevant for the initial load only

 

MT_ID

MIGR_OBJ_ALIAS

VALIDITY

ACS_PLAN_ID

FIELD

CURRNUM

LOGICAL_OPERATOR

VALUE

GATING_OPERATOR

001

BKPF

2

00000

GJAHR

0001

>=

2013

 

001

BKPF

2

00000

BUKRS

0002

=

'0010'

OR

001

BKPF

2

00000

BUKRS

0003

>=

'1000'

AND

001

BKPF

2

00000

BUKRS

0004

<=

'3000'

 

 

The following where clause will be used for the select:

 

SELECTFROMBKPF WHEREGJAHR >= 2013 AND BUKRS = '0010'OR BUKRS >= '1000'ANDBUKRS <= '3000'.

 

Process: Parallelize Replication

You can enter ranges to split the replication into parallel tasks. The number of defined ranges also defines the maximum parallel processing of the current table. How many jobs will finally procssing a table is defined by the overall number of jobs and the maximum number of parllel jobs defined for a table (can be defined in LTRC -> Data Transfer Monitor). If you define range criteria for the parallelization of the replication you need to define disjount subsets of the data to be processed by each of the replication jobs.

 

Example 1: Parallelize Replication
Split replication of table BKPF in mass transfer ID 001 into 4 ranges. The ranges are defined based on field:
Range 1: 0000 -> 2000
Range 2: 2000 -> 5000
Range 3: 5000 -> 8000
Range 4: 8000 -> 9999

 

MT_ID

MIGR_OBJ_ALIAS

VALIDITY

ACS_PLAN_ID

FIELD

CURRNUM

LOGICAL_OPERATOR

VALUE

GATING_OPERATOR

001

BKPF

3

00001

BUKRS

0001

<=

'2000'

 

001

BKPF

3

00002

BUKRS

0001

>

'2000'

AND

001

BKPF

3

00002

BUKRS

0002

<=

'5000'

 

001

BKPF

3

00003

BUKRS

0001

>

'5000'

AND

001

BKPF

3

00003

BUKRS

0002

<=

'8000'

 

001

BKPF

3

00004

BUKRS

0001

>

'8000'

 

 

The following where clause will be used for the select:

Range 1:

SELECTFROMBKPF WHEREBUKRS <= '2000'. 


Range 2:

SELECTFROMBKPF WHEREBUKRS > '2000'ANDBUKRS <= '5000'.


Range 3:

SELECTFROMBKPF WHEREBUKRS > '5000'ANDBUKRS <= '8000'.


Range 4:

SELECTFROMBKPF WHEREBUKRS > '8000'.

 

 

Example 2: Parallelize Replication & Filter
Split replication of table BKPF in mass transfer ID 001 into 4 ranges as defined in example 1. In addition only those records should be loaded where GJAHR is greater or equal 2013.

 

MT_ID

MIGR_OBJ_ALIAS

VALIDITY

ACS_PLAN_ID

FIELD

CURRNUM

LOGICAL_OPERATOR

VALUE

GATING_OPERATOR

001

BKPF

1

00000

GJAHR

0001

>=

2013

 

001

BKPF

3

00001

BUKRS

0001

<=

'2000'

 

001

BKPF

3

00002

BUKRS

0001

>

'2000'

AND

001

BKPF

3

00002

BUKRS

0002

<=

'5000'

 

001

BKPF

3

00003

BUKRS

0001

>

'5000'

AND

001

BKPF

3

00003

BUKRS

0002

<=

'8000'

 

001

BKPF

3

00004

BUKRS

0001

>

'8000'

 

 

The following where clause will be used for the select:

Range 1:

SELECTFROMBKPF WHEREGJAHR >= 2013 AND BUKRS <= '2000'.

 

Range 2:

SELECTFROMBKPF WHEREGJAHR >= 2013 AND BUKRS > '2000'ANDBUKRS <= '5000'.

 

Range 3:

SELECTFROMBKPF WHEREGJAHR >= 2013 AND BUKRS > '5000'ANDBUKRS <= '8000'.

 

Range 4:

SELECTFROMBKPF WHEREGJAHR >= 2013 AND BUKRS > '8000'.

 

 

Enjoy this new features

 

Best,
Tobias


Viewing all articles
Browse latest Browse all 38

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>