HANA 2 and two riddle errors

HANA 2 has been out for a while now – even SP01 of it is available already – so I thought it is about time to get more familiar with it.

Using my trusty “skullbox”. I installed HANA 2 and realised that I now also will have to have a HANA Cockpit 2.0 installation, to be fully able to use all of the glorious new features.

Downloading the HANA Cockpit 2.0 package is a breeze as long as you do it via a proper broadband connection.

The installation archive for the database administration tool just takes about 2.5 GB (sic!).

> ll -h
-rw-r--r-- 1 lars sdba 2.5G May 12 19:58 SAPHANACOCKPIT02P_1-70002299.SAR

Compare this with, say, the current OpenSuse Leap Linux distribution (4.7 GB) and one cannot help having certain expectations around usability, the range of features and general awesomeness.
BTW: OpenSuse can be downloaded via the torrent network – given the size of the installation binaries for typical SAP application, that would be nice to have for the SAP Service Marketplace downloads, too.
Instead, in 2017, we still have to use the “Download Manager” tool, that sequentially downloads files from one server at a time.

Once the archive is on my skullbox, the interesting part begins.
Using the graphical installer “hdblcmgui” I triggered off the installation.
Being a “consumer” type of computer user as long as the situation allows, I simply accepted all the default values concerning file paths, instance numbers or names.

Click next, click next

And off the installation went, pushing a full blown HANA 2 instance onto my machine.
“WHAT?” I hear you say… which I have to respond to with “That’s right, the HANA Cockpit 2.0″ comes with its very own HANA 2 installation.”
That approach probably makes sense if one is managing many, many instances across your organisation in a centralised fashion (yep, that’s the old business user – IT – DBA/Admin way of thinking), but in times where “DevOps” is THE thing and development teams look after their “own” DBs themselves, this looks like bloat to me.
If anything, this sure will tick up the counter for productive HANA instances…

Surprisingly, the installation of HANA 2 went rather quickly. I suspect that having an SSD in your system really pays off for this sort of workload.
When I was just on the homestretch to successful-installation-celebration, a dialogue box announced:
“Installation failed – Error occurred while executing SAP HANA Cockpit Components.”

Fortunately, the hdblcmgui tool makes it really easy to look at the log files in order to find out what was the cause for the error.

Kudos to the development colleagues for that!

This is what I found in the error log file:

[...]

18:02:19.875 - INFO: Output line 89: [26] --- Installation step: Set the HANA broker default DB ---

18:02:19.875 - INFO: Output line 90: [26] Set the default mapping to : '594863bc-d5f5-0b77-e100-00007f000002'(SYSTEMDB)

18:02:19.933 - INFO: Output line 91: [26] Map org: space: to DB: 594863bc-d5f5-0b77-e100-00007f000002

18:02:19.985 - INFO: Output line 92: [27] --- Installation step: Audit-Log Service (DB) ---

18:02:19.994 - INFO: Output line 93: [27] Creating application auditlog-db

18:02:19.994 - INFO: Output line 94: [27] Creating service auditlog-db-container

18:02:23.161 - INFO: Output line 95: [27] Uploading files for application auditlog-db

18:02:23.417 - INFO: Output line 96: [27] Staging application auditlog-db

18:02:24.454 - INFO: Output line 97: [27] 6/20/17 6:02:23.601 PM [STG/1] ERR /bin/sh: mc: line 1: syntax error: unexpected end of file

18:02:24.454 - INFO: Output line 98: [27] 6/20/17 6:02:23.601 PM [STG/1] ERR /bin/sh: error importing function definition for `mc'

18:02:24.454 - INFO: Output line 99: [27] 6/20/17 6:02:23.604 PM [STG/1] ERR bash: mc: line 1: syntax error: unexpected end of file

18:02:24.454 - INFO: Output line 100: [27] 6/20/17 6:02:23.604 PM [STG/1] ERR bash: error importing function definition for `mc'

18:02:24.454 - INFO: Output line 101: [27] 6/20/17 6:02:24.160 PM [STG/1] ERR bash: mc: line 1: syntax error: unexpected end of file

18:02:24.455 - INFO: Output line 102: [27] 6/20/17 6:02:24.161 PM [STG/1] ERR bash: error importing function definition for `mc'

[...]

First off: why is the error message (indicated by ERR) marked as a non-error INFO output line?

That is unnecessarily confusing.

Then looking at the error messages really didn’t help all that much:

/bin/sh: mc: line 1: syntax error: unexpected end of file

/bin/sh: error importing function definition for `mc'

bash: mc: line 1: syntax error: unexpected end of file

bash: error importing function definition for `mc'

“What is function ‘mc’?” you might ask and so did I.

Turns out, many moons ago, when I first installed my NUC system, I opted for some convenience in the dire straits of the Linux terminal console and installed Midnight Commander (a fantastic tool, everyone should have that!).

MidnightCommander’s program file happens to be called ‘mc‘… so, this seems to be a hot lead.

Unfortunately, I’m not the super-duper-Linux-expert, which is why I haven’t been aware of the way that Linux shells technically implement their startup.

Why would I need to know this? Well, after some lengthy research, I found that the error message was caused by a shell initialisation script stored in system folder

/etc/profile.d

Without going into too many details here, there were two symbolic links (mc.sh and mc.csh) in this folder, pointing to files from the MidnightCommander installation.

After deleting those links (you can always recreate them via `ln /usr/share/mc/mc.sh /etc/profile.d/mc.sh` and `ln /usr/share/mc/mc.csh /etc/profile.d/mc.csh`), I thought the problem was fixed and I could just continue the installation.

For some reason though, this does not seem to be possible. Instead, I uninstalled the whole half-installed HANA Cockpit setup and started from scratch.

This time around, no script failed due to the “syntax error” and the “unexpected end of file”.

Awesome, until…

Which would have been awesome, if the installation would not have run into a different error right after that:

[...]

18:22:34.688 - INFO: Output line 156: [31] [DEPLOY_SERVICE] Getting multi-target apps in org "HANACockpit" / space "SAP" as COCKPIT_ADMIN...

18:22:34.689 - INFO: Output line 157: [31] [DEPLOY_SERVICE] No multi-target apps found

18:22:34.689 - INFO: Output line 158: [31] [DEPLOY_SERVICE] 

18:22:34.697 - INFO: Output line 159: [31] Checking for existing service lcm-view-grantor

18:22:34.713 - INFO: Output line 160: [31] Creating service lcm-view-grantor

18:22:35.947 - INFO: Output line 161: [31] Checking for existing service lm-service-credentials

18:22:35.960 - INFO: Output line 162: [31] Creating service lm-service-credentials

18:22:36.024 - INFO: Output line 163: [31] Deploying application Product Installer

18:22:36.281 - INFO: Output line 164: [31] [DEPLOY_SERVICE] 

18:22:36.707 - INFO: Output line 165: [31] [DEPLOY_SERVICE] Uploading 1 files:

18:22:36.707 - INFO: Output line 166: [31] [DEPLOY_SERVICE] /hana/shared/H4C/xs/installdata/apps/product-installer/product-installer.mtar

18:22:38.243 - INFO: Output line 167: ERR Failed to upload files

18:22:38.243 - INFO: Output line 168: ERR Error occured during communication with HTTP server

18:22:38.244 - INFO: Output line 169: ERR Broken pipe (Write failed) (local port 63255 to address 127.0.0.1 (localhost), remote host unknown)

18:22:38.255 - INFO: Output line 170: [31] Removing service lcm-view-grantor

18:22:38.258 - INFO: Output line 171: [31] Service binding lcm-view-grantor not found

18:22:38.312 - INFO: Output line 172: [31] ERROR: 

18:22:38.312 - INFO: Output line 173: [31] com.sap.xs2rt.installation.util.InstallationException: 

Installing the Product Installer application failed: 

com.sap.cloud.lm.sl.slp.client.communication.CommunicationException: 

Error occured during communication with HTTP server

18:22:38.312 - INFO: Output line 174: [31] at 

[...]

Once again, error messages are labelled as non-error INFO lines, but good to know each timestamp to the millisecond…

This time around, the DEPLOY_SERVICE complains about a “Broken pipe” when writing to a port on localhost, claiming the “remote host is unknown“.

“How odd!” I thought, because “localhost” is not “remote” at all and not “unknown” at all.

A quick ping localhost proved that “localhost” was indeed known and reachable, so what could be the problem?

The key hint – which I was not able to pick up, instead I found a side comment in one of few open customer incidents – is that this is not about communication via UDP or TCP, but via HTTP.

HTTP does use cookies for a variety of purposes.

Count your dots!

The insight from above web pages was that proper cookie domains have to have at least 2 (two!) dots/periods in them.

That’s right, a fully-qualified-hostname like `skullbox.cat5` is not suitable for proper cookies.

> hostname -f

skullbox.cat5

Alright, so I changed the domain name of my system (that has been running HANA and other software in different versions for over a year now and never ever had an issue with the domain name, but hey…) to `skullbox.lab.cat5`.

That’s two dots in that name now:

> hostname -f

skullbox.lab.cat5

To be frank, this error is the one that really frustrated me.

Nowhere in the current or past HANA documentation does it say:

“Make sure to have two dots in your fully-qualified-hostname or we’ll throw random cryptic error messages at you.”

Even with the knowledge of what went wrong, it was a task to find any SAP documentation in relation to this.

The closest I found are

“In accordance with the Netscape cookie specification, cookies can be set for one domain only, and a domain must contain two or three dots (.) due to security restrictions. 

Each of the seven top level domains (.COM,.EDU,.NET,.ORG,.GOV,.MIL,.INT) must contain at least one further domain component (usually the name of the company or organization), amounting to two dots. 

Each domain with a different ending (this includes the top level domains for countries, such as UK, DE, FR, and so on) must consist of two further domain components, that is, these domains must contain at least three dots. For more information see the cookie specification.

Tip

Examples of valid domains:

<host>.sap.com Top level domain -> two domain components

<host>.portal.sap.de No top level domain -> three domain components”

Rounding off

After another round of reboot and uninstallation of the broken HANA Cockpit setup, hdblcmgui finally went through all the steps, finishing with “green lights”.

So, here we are. 157 lines into the installation of a database monitoring and administration tool that apparently is required to use HANA 2.

Given the information available (before I wrote this blog post), it took me several weeks to get it up and running and nearly two days straight when I finally decided to get to the bottom of the errors and finally have a proper HANA 2 setup.

I cannot say that I enjoyed the experience or felt that it was time well spent.