Upgrading HANA XSA & WebIDE + SSL Configuration

Backup

First backup all your existing XSA projects to Git and/or export the mta archives.

Obtaining the Software

It’s easy to forget exactly what’s been installed, so I have captured here the pieces I needed.

Search Words	Description	Filename
EXTAPPSER	Extended App Services – XSA server binaries	EXTAPPSER00P_88-70001316.SAR
HRTT	HANA Run Time Tools, required for WebIDE	XSACHRTT06_62-70002322.ZIP
XSACMONITOR	XSA Monitoring, required for WebIDE	XSACMONITORING06_5-70002324.ZIP
WEBIDE	SAP WEBIDE 2	XSACSAPWEBIDE03_6-80002257.ZIP

Extracting the SAR

If you extract the XSA Server (EXTAPPSER*.SAR) component to a sub-folder within the same parent directory as the install media (SAP_HANA_DATABASE) then the SAP_HANA_DATABASE\hdblcm will find EXTAPPSER.

media

|

|– SAP_HANA_DATABASE

|     |- hdblcm

|     |- hdbinst

|     |- hdbsetup

|     |- …

|

|– XSA_SERVER  (put EXTAPPSER*.SAR in here, and extract it)

|     |- hdbinst

|     |- …

Installing EXTAPPSER

Running hdblcm from SAP_HANA_DATABASE directory and it will automagically find the XSA_SERVER and other packages within the directory structure.

Below I had already installed the XSA update using option 3.

Updating the XSA Apps

With the install completed I then used the xs client commands to update the WebIDE and associated components.

First log into XSA and specifying the appropriate space.

mo-3fda111e5.mo:/usr/sap/IH2/HDB00> xs login -u XSA_ADMIN -p TopSecretPassword -a https://mo-3fda111e5.mo.sap.corp:30030 -s SAP

API_URL: https://mo-3fda111e5.mo.sap.corp:30030

USERNAME: XSA_ADMIN

Authenticating...

ORG: orgname

SPACE: SAP

API endpoint:   https://mo-3fda111e5.mo.sap.corp:30030 (API version: 1)

User:           XSA_ADMIN

Org:            orgname

Space:          SAP

I can now go ahead an install the download zip files.

xs install XSACHRTT06_62-70002322.ZIP 

xs install XSACSAPWEBIDE03_6-80002257.ZIP 

xs install XSACMONITORING06_5-70002324.ZIP 

Space Enablement

Before actually using the updated WebIDE, it is required update the di-builder via the space enablement your non SAP space.  To find out out the URL for this you can run this xs command.

mo-3fda111e5.mo:/hana/media/XSA-88> xs app di-space-enablement-ui --urls

https://mo-3fda111e5.mo.sap.corp:51022

Going to that URL,shows the space needs updating with the latest goodies.

Proxy Configuration

For me to use external sites such as github.com or deploy to cloudfoundary required a proxy.  This can done by specifying the JBP_CONFIG_JAVA_OPTS for the DI_CORE module and redeploying it. I created a .mtaext file with the following contents.

_schema-version: "3.1.0"

ID: com.sap.devx.xs2-config1

extends: com.sap.devx.webide

modules:

  - name: di-core

    parameters:

      port: 53030

      memory: 2G 

    properties:

      JBP_CONFIG_JAVA_OPTS: '[java_opts: " -Dhttps.proxyHost=proxy.wdf.sap.corp -Dhttps.proxyPort=8080 -Dhttp.proxyHost=proxy.wdf.sap.corp -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=\\"*.sap.corp|*.corp.sap|mo-3fda111e5|mo-3fda111e5.mo.sap.corp|localhost|127.0.0.1|10.*\\" -Dhttps.nonProxyHosts=\\"*.sap.corp|*.corp.sap|mo-3fda111e5|mo-3fda111e5.mo.sap.corp|localhost|127.0.0.1|10.*\\""]'

After creating the .mtaext file I then needed to re-deploy the WebIDE to pickup the proxy settings.

xs install XSACSAPWEBIDE03_6-80002257.ZIP -e My-WebIDE.mtaext -o ALLOW_SC_SAME_VERSION --delete-services

Npm (node package manager) also requires the proxy to be set, I did this as below.

 xs set-env di-local-npm-registry HTTPS_PROXY http://proxy.wdf.sap.corp:8080

 xs set-env di-local-npm-registry HTTP_PROXY http://proxy.wdf.sap.corp:8080

 xs restage di-local-npm-registry

 xs restart di-local-npm-registry

Renew trust relationship between XSA and HANA Database

If you receive the following error

error setting XS_APPLICATIONUSER failed due to invalid or expired token

To correct this, you need to execute the following from the SIDADM user

XSA renew-db-trust -u XSA_ADMIN

SSL – Private Key

Previously I had not configured a trusted SSL certificate for XSA, but this can lead to unexpected errors with the different xsa services. Each untrusted URL needs to been granted browser exceptions.

I had already setup SSL for XS, so thought it would be easy to re-use some of that configuration (it’s not that easy).

1. Export the private key from the in-built SAP keystore.  

sapgenpse export_p12 -p /usr/sap/IH2/HDB00/mo-3fda111e5.mo.sap.corp/sec/SAPSSLS.pse /hana/media/ssl/Export.p12

2.  Convert the .p12 format to pem.

openssl pkcs12 -in Export.p12 -out Export.pem -nodes

Within the newly generated export.pem there are multiple certificates and keys.  The Private Key is the required part which starts with.

—–BEGIN PRIVATE KEY—–

3. copy/paste just the private key that into a new file Private.key

4. Convert to pkcs8 (that XSA requires).  This can be converted with this command.

openssl pkcs8 -topk8 -in Private.key -out Private_pkcs8.key -nocrypt

SSL Root & Intermediate Certificates

XSA also requires a full certificate chain.  This means you need to supply the host, intermediate and root certificates all combined into a single x.509 PEM encoded file.

I was provisioning and internal SAP host using our internal Web Enrolment Service.

If you are an SAP employee or have access to the corporate network this can be found here https://sapcerts.wdf.global.corp.sap/WebEnrollment.aspx

This provides  the certificate for the XSA host, but it needs to converted and combined with the full certificate chain.

1. Download the correct root & intermediate Certificates

2. Convert to x509 PEM format

openssl x509 -in SAPGlobalRootCA.crt -outform PEM -out SAPGlobalRootCA.pem 

openssl x509 -in SAPNetCA_G2.crt -outform PEM -out SAPNetCA_G2.pem

openssl x509 -in mo-3f.txt -outform PEM -out mo-3f.pem

3. Combine certificates in correct order (host, intermediate, root)

cat mo-3f.pem >> combined.pem

cat SAPNetCA_G2.pem >> combined.pem 

cat SAPGlobalRootCA.pem >> combined.pem 

Upload to XSA

xs set-certificate mo-3fda111e5.mo.sap.corp -c combined.pem -k Private_pkcs8.key

After successfully installing the certificate you need to restart each of the XSA apps individually or the whole of XSA (which is easier). This can be dome with the hdbadm user.

mo-3fda111e5.mo:/usr/sap/IH2/HDB00> XSA restart

After which I have XSA & WebIDE running with a signed SSL certificate.

Including all the new graphical Calculation View operators like graph, anonymization, union, minus, intersect, hierarchy function, etc.

Troubleshooting

If you do experience issues the following commands are useful for identifying, potential issues.

xs logs product-installer --last 1000

xs logs webide --recent

xs logs di-core --recent

xs display-installation-logs <PID>

If the certificates get messed up you can reset those using the hdbadm user

mo-3fda111e5.mo:/usr/sap/IH2/HDB00> XSA reset-certificate

Errors

I have experienced some errors when the configuration was not correct. Often the exposed error is only the latest error and not the root cause of the problem for me I was initially seeing this generic error

xs display-installation-logs 99232 -scv

ERR com.sap.cloud.lm.sl.slp.client.communication.NotFoundException:

I also found these errors, showing that a process has not started.

#2.0#2018 12 12 12:09:31.605#Z#INFO#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.lm.sl.alm.prod.installer.process.steps.StepsUtil########pool-7-thread-2###

[PollStateStep] Unexpected error: Execution of step RestartAppStep has timed out#

#2.0#2018 12 12 12:09:31.626#Z#ERROR#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.lm.sl.alm.prod.installer.process.steps.StepsUtil########pool-7-thread-2###

[PollStateStep] Unexpected error: Execution of step RestartAppStep has timed out#

#2.0#2018 12 12 12:09:31.663#Z#ERROR#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.lm.sl.alm.prod.installer.process.steps.StepsUtil########pool-7-thread-2###

[PollStateStep] Installation of the component XSAC_SAP_WEB_IDE (sap.com) 4.3.60 failed during deployment.#

#2.0#2018 12 12 12:09:31.687#Z#ERROR#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.lm.sl.alm.prod.installer.process.steps.StepsUtil########pool-7-thread-2###

[PollStateStep] Deployment of the MTA com.sap.devx.webide 4.3.61 failed. Check SAP release information note 2510063 and analyze the issue. In case you need help, contact SAP support using the SAP component HAN-WDE-PREF#

#2.0#2018 12 12 12:09:31.704#Z#INFO#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.cloud.lm.sl.slp.steps.SLProcessStepHelper########pool-7-thread-2###

Deploy Service process instance [99232] will be aborted.#

#2.0#2018 12 12 12:09:33.837#Z#INFO#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.cloud.lm.sl.slp.steps.SLProcessStepHelper########pool-7-thread-2###

Deploy Service process instance [99232] is aborted.#

#2.0#2018 12 12 12:09:33.837#Z#ERROR#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.cloud.lm.sl.slp.steps.SLProcessStepHelper########pool-7-thread-2###

Exception caught#

java.lang.Exception: Unexpected error: Execution of step RestartAppStep has timed out

at com.sap.lm.sl.alm.prod.installer.process.steps.PollStateStep.pollStatus(PollStateStep.java:187)

at com.sap.activiti.common.impl.AbstractActivitiStepWithBridge.executeStep(AbstractActivitiStepWithBridge.java:59)

at com.sap.activiti.common.impl.AbstractActivitiStep.execute(AbstractActivitiStep.java:56)

at com.sap.cloud.lm.sl.slp.steps.AbstractSLProcessStepWithBridge.execute(AbstractSLProcessStepWithBridge.java:38)

at org.activiti.engine.impl.delegate.JavaDelegateInvocation.invoke(JavaDelegateInvocation.java:34)

at org.activiti.engine.impl.delegate.DelegateInvocation.proceed(DelegateInvocation.java:37)

at org.activiti.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:25)

at org.activiti.engine.impl.bpmn.behavior.ServiceTaskJavaDelegateActivityBehavior.execute(ServiceTaskJavaDelegateActivityBehavior.java:49)

at org.activiti.engine.impl.bpmn.behavior.ServiceTaskJavaDelegateActivityBehavior.execute(ServiceTaskJavaDelegateActivityBehavior.java:40)

at org.activiti.engine.impl.bpmn.helper.ClassDelegate.execute(ClassDelegate.java:158)

at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:60)

at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)

at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:648)

at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:643)

at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:52)

at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)

at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)

at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:648)

at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:643)

at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)

at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)

at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:648)

at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:643)

at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:49)

at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)

at org.activiti.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:35)

at org.activiti.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:85)

at org.activiti.engine.impl.persistence.entity.MessageEntity.execute(MessageEntity.java:34)

at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:88)

at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)

at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)

at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)

at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)

at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)

at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)

at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)

at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)

at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.handleMultipleJobs(ExecuteJobsRunnable.java:94)

at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:836)

#2.0#2018 12 12 12:09:33.844#Z#DEBUG#com.sap.lm.sl.alm.26989.MAIN_LOG#

######com.sap.cloud.lm.sl.slp.steps.SLProcessStepHelper########pool-7-thread-2###

Step "Poll Deploy State" finished#

Here di-core was not starting, due to a lack of allocated memory.

xs logs di-core -last 500

12/12/18 11:09:37.796 [APP/1-0] ERR #2.0#2018 12 12 11:09:37.796#+00:00#ERROR#com.sap.di.api.infrastructure.auth.HiabBuilderToCoreAuthenticationManager#

12/12/18 11:09:37.796 [APP/1-0] ERR ###4213DE1E966647AEA4B57BBB4EE6DB54###com.sap.di.api.infrastructure.auth.HiabBuilderToCoreAuthenticationManager########localhost-startStop-1#PLAIN##

12/12/18 11:09:37.796 [APP/1-0] ERR Received servlet context: org.apache.catalina.core.ApplicationContextFacade@153175d8

12/12/18 11:11:10.893 [API] ERR Please check state of app 'di-core' [Org 'orgname', Space 'SAP'] (droplet with id 1 of app 'di-core' [Org 'orgname', Space 'SAP'] created by buildpack 'sap_java_buildpack') as the portion of uptime 0.41 is below critical threshold.

12/12/18 11:11:40.878 [API] ERR Please check state of app 'di-core' [Org 'orgname', Space 'SAP'] (droplet with id 1 of app 'di-core' [Org 'orgname', Space 'SAP'] created by buildpack 'sap_java_buildpack') as the portion of uptime 0.33 is below critical threshold.

12/12/18 11:11:57.000 [RTR] OUT 10.44.103.208 - - to mo-3fda111e5.mo.sap.corp:53030 "POST /admin/builder/server/register HTTP/1.1" 503 sent 9709 in 208 by -

12/12/18 11:11:58.000 [RTR] OUT 10.44.103.208 - - to mo-3fda111e5.mo.sap.corp:53030 "POST /admin/builder/server/register HTTP/1.1" 503 sent 9709 in 207 by -

12/12/18 11:12:10.889 [API] ERR Please check state of app 'di-core' [Org 'orgname', Space 'SAP'] (droplet with id 1 of app 'di-core' [Org 'orgname', Space 'SAP'] created by buildpack 'sap_java_buildpack') as the portion of uptime 0.26 is below critical threshold.

12/12/18 11:12:40.888 [API] ERR Please check state of app 'di-core' [Org 'orgname', Space 'SAP'] (droplet with id 1 of app 'di-core' [Org 'orgname', Space 'SAP'] created by buildpack 'sap_java_buildpack') as the portion of uptime 0.21 is below critical threshold.

Rescaling the application fixed this.

xs scale di-core -m 2G -f

The root cause for this was hidden deeper in the di-core logs, with an in-correct proxy configuration.  Initially in the .mtaext I had “;” separators between my nonProxyHosts, switching to “|” fixed this.

Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"

13/12/18 08:45:06.106 [APP/5-0] SYS at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2152) ~[na:1.8.0_181]

13/12/18 08:45:06.106 [APP/5-0] SYS at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[na:1.8.0_181]

13/12/18 08:45:06.106 [APP/5-0] SYS at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) ~[na:1.8.0_181]

13/12/18 08:45:06.106 [APP/5-0] SYS at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:75) ~[spring-web-4.0.9.RELEASE.jar:4.0.9.RELEASE]

13/12/18 08:45:06.106 [APP/5-0] SYS at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.0.9.RELEASE.jar:4.0.9.RELEASE]

13/12/18 08:45:06.106 [APP/5-0] SYS at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.0.9.RELEASE.jar:4.0.9.RELEASE]