SAP HANA Backup via backint interface. Step-by-step guide for Veritas NetBackup.

Prerequisites

You want config backup via backint interface for your SAP HANA multitenant database.
This step-by-step guide provides information about Veritas NetBackup solution
N.B .If you want to use other backup tools, for example, EMS Networker, Veeam Plug-in for SAP HANA, Spectrum Protect for Enterprise Resource Planning, etc this guide will be useful for you too.

SAP Environment

◉ SAP HANA DB
◉ SAP HANA Cockpit
◉ SAP HANA Studio
◉ Veritas Solution: NetBackup Agent, NetBackup Server and  NetBackup OpsCenter

Also Read: SAP HANA Application - C_HANAIMP_12

About

SAP HANA database provides a backup interface, called Backint for SAP HANA database. This interface offers 3rd-party backup tool vendors to connect their product to the backup and recovery capabilities of the SAP HANA database. Backint for SAP HANA is fully integrated into SAP HANA database, i.e. data and log backups can be configured individually to be created and recovered in the 3rd-party backup tool.

Resolution

Step 1: Install and Config NetBackup Server (Master Server)

Create Policies

NetBackup Management -> Policies

Step 2: Veritas NetBackup Agent installation  on HANA DB-host

Substep 2.1. Run installation

Substep 2.2. During the installation process specify Veritas Master server FQDN from Step 1

Step 3: Configuration of Veritas NetBackup Agent

Substep 3.1. Create utl-files

Goto to a folder:

cd /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/

Create directories for each tenant in HDB:

mkdir <SID>

Create a policy file for each SID from initial utl-file (initSAP.utl)

cp initSAP.utl <SID>/initSAP-<SID>-log.utl

cp initSAP.utl <SID>/initSAP-<SID>-data.utl

Substep 3.2. Edit created a utl-files:

Provide information about  “policy” and “schedule” parameters  in created utl-files:

nano /<SID>/initSAP-<SID>-log.utl

Substep 3.3. Create and edit node_names.txt  file:

nano /usr/openv/ netbackup/ext/db_ext/sap/node_names.txt

provide here FQDN name of Master-server from Step1

Substep 3.4  Create a symbolic link to hdbbackint_script

ln -s /usr/openv/netbackup/bin/hdbbackint_script /usr/sap/<SID>/SYS/global/hdb/opt/ hdbbackint

Step 4. Settings in SAP HANA Cockpit / SAP HANA Studio

Login in SAP Hana Cockpit. Open SystemDB Overview

Goto section DB Administration-> Configure system properties

Settings for global.ini

[section – backup]

catalog_backup_using_backint = false

log_backup_using_backint = true

data_backup_parameter_file = /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/<SID>/initSAP-data.utl (for SYSTEMDB / and each tenant)

log_backup_parameter_file= /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/<SID>/initSAP-log.utl (for SYSTEMDB / and each tenant)

log_backup_using_backint = true (for SYSTEMDB and each tenant)

[section – persistence]

basepath_catalogbackup  = /catalogbackup

basepath_databackup = /catalogbackup

N.B. In this case we setup Data/Log backup via backint.

catalogbackup via File to the NFS-share  in mount point /catalogbackup

Settings for xsengine.ini

Add or Edit

[section – scheduler]

enabled = true

Step 5: Create Ad-Hoc backup for SYSTEMDB and each tenant

Login in SAP Hana Cockpit. Open SYSTEMDB / Tenant Overview

Goto section DB Administration-> Manage database backups -> Create backup

Backup Type: Complete

Destination Type: Backint

Status: Successful

Step 6.  Create  backup Schedule for SYSTEMDB and each tenant

Login in SAP Hana Cockpit. Open SYSTEMDB / Tenant Overview

Goto section DB Administration-> Manage database backups -> Create Schedule

Backup Type: Complete

Destination Type: Backint

Unique Schedule Name: <SID>@<SID>

Start on:  Time

Every week on:  Days of the week

Step 7.  Recover SYSTEMDB  and each tenant

Login in Hana Cockpit. Open SYSTEMDB Overview

Goto Overall Tenant Status.

Stop Tenant – Recover Tenant

Backup to be Used

Check Availability Backup – Backint

Initialize Log Area – No

Recovery – In Progress

Recovery – Done!

Step 8.Maintenance for catalogbackup.

As indicated in Step 4 in this approach:

For full and log backup is used a backint interface.

For catalogbackup is used a remote NFS folder (mount point /catalogbackup).

Keep in mind that information in catalogbackup is redundant

Each next catalogbackup-file contains information about full/log backups  + small delta

In this case, we can use such a strategy for catalog-backup files:

1. Rotation. Keep in NFS -folder only the last three catalogbackup files. Backup to tape and cleanup NFS folder

2. Cleanup catalogbackup files itself

On DB-level (Create Role/User):

CREATE ROLE HANA_CLEANING;

GRANT BACKUP ADMIN, CATALOG READ, LOG ADMIN, MONITOR ADMIN, RESOURCE ADMIN, TRACE ADMIN to HANA_CLEANING;

GRANT SELECT, DELETE ON “_SYS_STATISTICS”.”HOST_OBJECT_LOCK_STATISTICS_BASE” TO HANA_CLEANING;

GRANT SELECT, DELETE ON “_SYS_STATISTICS”.”STATISTICS_ALERTS_BASE” TO HANA_CLEANING;

GRANT SELECT, DELETE ON “_SYS_REPO”.”OBJECT_HISTORY” TO HANA_CLEANING;

CREATE USER HANACLEANER1 password <your password for HANACLEANER1 DB USER>;

GRANT HANA_CLEANING to HANACLEANER1;

ALTER USER HANACLEANER1 DISABLE PASSWORD LIFETIME;

On OS-level:

log-on as an <sid>adm os-user and add a new a key to hdbuserstore:

hdbuserstore SET HANACLEANER1KEY  hostname:3<instance>15 HANACLEANER1 <your password>

log-on as a <sid>adm os-user and check hdbuserstore:

hdbuserstore LIST HANACLEANER1KEY

Download the Python-script hanacleaner.py from Note 2399996 and copy it to the Scripts-folder:

(/usr/sap/<SID>/HDB<Instance>/exe/python_support/)

Create a shell -file hanacleaner.sh

python /usr/sap/<SID>/HDB<instance>/exe/python_support/hanacleaner.py -fl 25 -fo true -tc 30 -td true -to true -es true -os false -ar 30 -ad true -bd 45 -zb 40 -lr 3 -eh 10 -eu 10 -fo true -op /usr/sap/<SID>/HDB<instance>/hostname/trace

where:

-fl 25 – fragmentation limit DataVolume  %

-fo true – displays data volume statistics before and after defragmentation

-tc 30 – minimum retained days for trace files

-td true – displays the trace files that were deleted

-to true  – displays trace files before and after the cleanup

-es true – Execute all crucial housekeeping tasks (useful to turn off for investigations with –os=true)

-os false –  Prints all crucial housekeeping tasks (useful for debugging with –es=false)

-ar 30 – minimum number retained days of the alerts

-ad true -output deleted alerts

-bd 60 – minimum retained days of backup entries in the catalog

-zb 40 – backup logs compression size limit

-lr 3 – maximum number of free log segments per service

-eh 10 – minimum retained days for handled events

-eu 10 -minimum retained days for unhandled events

-fo true – output fragmentation displays data volume statistics before and after defragmentation

-op /usr/sap/<SID>/HDB<instance>/<hostname>/trace – output path full path of the folder where the hanacleaner logs are written

hanacleaner.sh was executed:

Create a cron job for this script:

crontab -e -u hsdadm 0 1 * * * /path/hanacleaner.sh >> /path/hanacleaner.log 2>&1

Step 9. Monitoring.

Monitoring via Veritas NetBackup OpsCenter (a web-based tool)

Monitor – > Jobs

Step 10. Troubleshooting.

For example In the HANA Cockpit / Hana Studio you see Error

Backup status: Failed

Additional Information:  [447] backup could not be completed, [110512] Backint reported ‘BACKINT did not return result for [/usr/sap/<SID>/SYS/global/hdb/backint/DB_<SID>/2019_12_02_04_30_00_databackup_0_1]’ in file ‘/var/tmp/hdbbackint_<SID>.vkrhNE’, [110203] Not all data could be w.

Investigate  backup.log / backint.log / nameserver.trc for additional information.

Log files are created and updated in specific directories. In SAP HANA  you will find the trace and log files root directory at : [ /usr/sap/<SID>/HDB<Instance number>< hostname>/trace ] This trace and log root directory contains  the SYSTEMDB trace and log files only. Each tenant database has it own directory with a trace and a log files. You can detemine this information from a cdtrace command output.

Log-in as <SID>adm

Run the following command : cdtrace

or via such logic:

Path to backup.log (for SYSTEMDB) –

/usr/sap/<sid>/HDB<instance>/<host>/trace/backup.log

/usr/sap/HSP/HDB35/sp1hspdb/trace/backup.log

Path to backint.log (for SYSTEMDB) –

/usr/sap/<sid>/HDB<instance>/<host>/trace/backint.log

Path to backint.log (for TENANTS DB) –

/usr/sap/<sid>/HDB<instance>/<host>/trace/DB_<TANANT_DB_SID>/backup.log

/usr/sap/<sid>/HDB<instance>/<host>/trace/DB_<TANANT_DB_SID>/backint.log

Investigate the Veritas log files:

On the SAP client host, put the following line in the  /usr/openv/netbackup/bp.conf  file :

    VERBOSE = 5

and create log directories :

    /usr/openv/netbackup/logs/vnetd

    /usr/openv/netbackup/logs/bpcd

    /usr/openv/netbackup/logs/backint

chmod – R 0777 /usr/openv/netbackup/logs/backint

Investigate log files in these directories.

Miscellaneous.

NetBackup Agent operation commands:

Stop Agent

cd /usr/openv/bin

./bp.kill_all

Start Agent

cd /usr/openv/bin

./bp.start_all