SAP HANA Fast Restart (TMPFS) is available from HANA 2.0 SPS 04. There few nice blogs which already addressing the Technical setup and theoretical information of this feature . This year we had implemented this feature in few customer HANA production systems who has very high restart times.
SAP HANA Fast Restart – TMPFS
◉ SAP HANA uses TMPFS Linux Concept to store column store in memory and this concept is know as SAP Fast Restart Option. So, when ever the HANA database is restarted column store tables are never unloaded and there in memory all the time until OS is restarted or any of the maintenance activity is performed on OS
◉ Once TMPFS is enabled the first restart will be still slower as that is the time when TMPFS is filled with column store data. Remaining restarts will be faster and only mapping of column store is done for those restarts
◉ The filesystem is automatically created when mounting a filesystem with the type tmpfs
◉ Data that is stored in tmpfs DRAM disks is volatile, and the contents are erased when the OS is shut down or restarted. Upon restart of the OS, the RAM disk volumes must be re-created, and SAP HANA columnar table data is reloaded from persistent disk volumes.
◉ Few questions came up on whether we need seperate storage for TMPFS mount points we create . Answer is No, we don’t need any seperate storage .TMPFS is part of DRAM and The filesystem is automatically created when mounting a filesystem with the type tmpfs
The Fast Restart option is a bridge technology for environments where servers with Persistent Memory are not available.
TMPFS is cost effective as we don’t need any changes from Hardware or OS side but it won’t cover any OS or hardware related outages . Where as PMEM Covers OS outage as well but need to change the hardware to Intel Optane DC persistent memory
HANA Restart Testing
With out fast restart the Column load took 1 hour 31 mins ( Extract from indexserver Trace)
[542509]{-1}[-1/-1] 2020-05-18 08:59:53.821959 i TableReload TRexApiSystem.cpp(00665) : Now reloading 854 tables (loading up to 100 tables in parallel)
Line 44637: [542509]{-1}[-1/-1] 2020-05-18 08:59:53.821959 i TableReload TRexApiSystem.cpp(00665) : Now reloading 854 tables (loading up to 100 tables in parallel)
Line 44688: [542509]{-1}[-1/-1] 2020-05-18 10:34:09.806415 i TableReload TRexApiSystem.cpp(00696) : Finished table reloading
Line 44689: [542509]{-1}[-1/-1] 2020-05-18 10:34:09.837977 i Service_Startup TREXIndexServer.cpp(02063) : Pre-/Re-Loading of column store tables finished.
With Fast restart Option Implemented it took 5 minutes. Mostkyt this time is mapping of TMPFS objects to HANA
Line 44761: [640391]{-1}[-1/-1] 2020-05-19 06:51:54.026274 i TableReload TRexApiSystem.cpp(00665) : Now reloading 854 tables (loading up to 100 tables in parallel)
Line 44761: [640391]{-1}[-1/-1] 2020-05-19 06:51:54.026274 i TableReload TRexApiSystem.cpp(00665) : Now reloading 854 tables (loading up to 100 tables in parallel)
Line 44769: [640391]{-1}[-1/-1] 2020-05-19 06:56:27.293814 i TableReload TRexApiSystem.cpp(00696) : Finished table reloading
Line 44770: [640391]{-1}[-1/-1] 2020-05-19 06:56:27.324967 i Service_Startup TREXIndexServer.cpp(02063) : Pre-/Re-Loading of column store tables finished.
We tested this with Index Server restart as well and behaviour was same .
◉ In host restart all the mount points were cleaned up and restart time took longer as normal which we noticed with out tmpfs
◉ In case of tmpfs unmount and restarted HANA restart took longer as normal restart.
◉ If any of the tmpfs is unmounted and mounted back the data in that tmpfs is cleared and for consistency aspect all other data in remaining mounts are not considered
35 Mins is still High and there are some other improvements that need to done for this system related to LOB store initialization.
SAP HANA System Replication – Fast Restart Option
1.Enabling TMPFS without breaking on going replication using NZDT approach
2.Behavior of replication with any of the primary or secondary is not changed with TMPFS enabled .
3.In Case of failover the failed node came back faster and can be configured as new secondary sooner than before TMPFS – This was main concern that was addressed for Micron with TMPFS
4.TMPFS is host specific and can be configured to one host initially and perform a rolling forward approach for implementing this on both primary and secondary with minimal downtime
5.Unload and load behavior of table is same in primary and secondary and can be managed by table_unload_action which we discuss in detail in next section
6. Make sure [inifile_checker] enable = true is set so that ini file changes are transferred to secondary
7.No delay or backlog’s observed in log shipping and log replay proces
HANA Backup/Snapshots – Fast Restart Option
Backup was taken using HANA Studio, version of studio is Version: 2.3.50
◉ Production: 2 hours 49 minutes (with out TMPFS)
◉ PoC: 2hours 36 minutes
Snapshot is taken for /HANA/data volume. Version is Snapcreator 4.3.3, vendor is NetApp.
◉ Production: 20 seconds – 1 minute (with out TMPFS)
◉ PoC: 08 seconds and 11 seconds for the 2 backups we ran from Snapcreator tool
HANA Load/Unload Behaviour – Fast Restart Option
After enabling TMPFS all the column store tables are not unloaded from memory while HANA is restarted. Now we did some validation on this behavior how it will behave when HANA performs unload or load
◉ When ever we are using command UNLOAD TABLE to unload table is getting cleared from HANA memory we had seen a clear decrease in HANA used memory and also the cache size is decreased. Later Table load took longer time
◉ This unload and load behavior can be controlled by below parameter-table_unload_action = retain: Main storage remains (default for persistent memory)-table_unload_action = delete: Main storage is cleared (default for fast restart option, see below), actual deletion doesn’t take place immediately but with next savepoint
◉ Regardless of the setting of this parameter, you can also use UNLOAD with the additional DELETE PERSISTENT MEMORY clause to completely free memory, for example UNLOAD sapsr3.edid4 DELETE PERSISTENT MEMORY; This cleared the MAIN data fragments from tmpfs
◉ If you do not want to delete PMEM blocks as part of unload then you can use
UNLOAD <TABLE_NAME> RETAIN PERSISTENT MEMORY
HANA Delta Merge – Fast Restart Option
◉ Merge with fast restart feature behaves similar as in classical HANA.
◉ The new main columns are constructed in regular DRAM. Whenever the merge of any column is completed, the data is transferred into a new tmpfs block, and the regular DRAM is freed again. Between creation of the new tmpfs block and deallocation of the regular DRAM, the total memory consumption of the new column is doubled. But this is synchronized for the various columns, such that it does happen only for one column at the same time. The tmpfs blocks of the old main are deleted at the end of the merge and are physically destroyed at the next savepoint.
◉ Performance wise there was no difference in execution times of job before and after tmpfs setup.
◉ For one of the scenario where table is having concurrent modifications like create or deletion we notice a lock for a while with lock name NVM_PROVIDER_FILE .This lock secures access to non-volatile memory (NVM). It can be observed in context of persistent memory and the fast restart option. This issue happened once and when we tested job immediate after system is restarted and not able to reproduce it. As of now if you see this lock on any of the table please switch the functionality off for that table using sql ALTER TABLE <TABLE_NAME> PERSISTENT MEMORY OFF <IMMEDIATE, DEFERRED, CASCADE> . This behavior is improved in SP05 . Also this is not a roadblock or causing any performance impact .
Other Observations
◉ Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. If you unmount a tmpfs instance, everything stored therein is lost .If a tmpfs filesystem is unmounted, its contents are discarded
◉ You can see the view M_CS* to check if table is using tmpfs .For example, the PERSISTENT_MEMORY column in M_CS_TABLES, M_CS_COLUMNS and M_CS_ALL_COLUMNS shows TRUE if real persistent memory or tmpfs is enabled also you will see bit more columns in this view
◉ While estimating size if any table in memory you have to consider new fields PERSISTENT_MEMORY_SIZE_IN_TOTAL and ESTIMATED_MAX_MEMORY_SIZE_IN_TOTAL in M_CS_TABLES
◉ There is no different behavior observed in case of a composite out of memory event.
◉ Savepoint performance was normal and not noticed any increase in durations.
No comments:
Post a Comment