Thursday, 3 August 2017

Removing invalid and non-printable characters in HANA based BW transformation

Most customers know the problem that invalid characters can be found in data records. Data load or DSO activation for char type InfoObjects can fail due to permitted or non-printable characters in BW.

There exist several solutions for this problem. Some customer uses function module calls in ABAP transformations for example RSKC_CHAVL_OF_IOBJ_CHECK or RSKC_ALLOWED_CHAR_GET. Other implement field routines with ABAP like

REPLACE ALL OCCURRENCES OF REGEX '[^[:print:]]' IN RESULT
       WITH 'replace_char'
       IGNORING CASE.

or…

IF RESULT(1) = '!' OR return = '#'.
    RESULT(1) = 'replace_char'.
ENDIF.

Now BW/4HANA has the option to push down transformation logic to HANA database, to reduce runtime of data loads. For custom transformations to be pushed down to HANA database, start, end and expert routine type need to be created as AMDP script (ABAP Managed Database Procedure script) instead of an ABAP based routine.

For the example we created a sample file with invalid characters ‘!’ and ‘#’ in record 2, 4, 5 and 6, and a non-printable character Tapstop in record 7.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

We load the data in a field based write optimized ADSO and want it to be loaded and activated in a standard ADSO with InfoObjects and changelog.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

To remove invalid and non-printable characters with an AMDP Script in a field routine, you can follow these steps.

Step 1: Select rule type routine for the transformation rule, see (1). A popup dialog asks for processing type, see (2). Choose AMDP script to create an AMDP script based field routine.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

Step 2: an AMDP class will be generated with default method – PROCEDURE and default interface – IF_AMDP_MARKER_HDB. The BW Transformation framework opens the Eclipse based ABAP class editor.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

Step 3: Enter your code for the field routine in the body of the method PROCEDURE.

You can use the following template. The code replaces the regular expression ([^[:print:]]|[#|!]) in source field “CHARACTERS” with replacement char ” and transfer the result in the target InfoObject “/BIC/ZSPECHAR”.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

If you want to remove invalid characters at the first position only, you can use this template.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

If necessary this can be extended with upper case conversion or similar.

After removing the characters in transformation our example data will be loaded without errors and the data activation in ADSO works well.

SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

No comments:

Post a Comment