SAP, Linux, Virtualization and ... Itanium
1 like28,409 views
This document proposes migrating an SAP environment from Windows Server 2003 on HP Integrity servers to Red Hat Enterprise Linux (RHEL) to enable virtualization and increased flexibility. The current environment uses HP EVA storage virtualization and includes 18 HP Integrity servers running SAP ERP and BI systems. Virtualizing with Xen on RHEL would allow consolidating systems, reducing hardware costs, and gaining benefits from migrating to Linux while maintaining platform consistency between test and production environments.
![- ntfs-3g installation (http://www.ntfs-3g.org):
cp /mnt/tmp/rhel/ntfs3g*.tar
tar xvf ntfs3g....
cd ntfs3g...
./configure
make
make install
yum install ntfsprogs
- firewall should be configured (opening needed ports, systemconfigfirewall) and
selinux (for a start and troubleshooting, “setenforce 0” and SELINUX=permissive in
/etc/selinux/config)
- Xen Dom0 configuration for several network adapters, create /etc/xen/scritps/network-
multi-bridge:
#!/bin/sh
dir=$(dirname "$0")
"$dir/networkbridge" "$@" vifnum=0 bridge=xenbr0 netdev=eth0
"$dir/networkbridge" "$@" vifnum=1 bridge=xenbr1 netdev=eth1
"$dir/networkbridge" "$@" vifnum=2 bridge=xenbr2 netdev=eth2
"$dir/networkbridge" "$@" vifnum=3 bridge=xenbr3 netdev=eth3
... and then: “chmod +x /etc/xen/scritps/networkmultibridge” and edit
/etc/xen/xend-config.sxp, putting “(networkscript networkmultibridge)” instead
of “(networkscript networkbridge)”
... also, put there (dom0-min-mem 2048) ... and dom0_mem=2G into elilo.conf to
reserve 2G for Dom0 and avoid memory ballooning (2G should be enough) – elilo.conf
example:
...
image=vmlinuz2.6.18164.10.1.el5xen
vmm=xen.gz2.6.18164.10.1.el5
label=2.6.18164.10.1.el5xen
initrd=initrd2.6.18164.10.1.el5xen.img
readonly
root=/dev/VolGroup00/LogVol00
append="dom0_mem=4224M palhalt rhgb quiet"
...
- vncserver setup (/etc/sysconfig/vncservers), vncpasswd should be set for the running
account, and initial vncserver start (“chkconfig vncserver on” for the service, too)
- hp support pack installation:
yum install netsnmp netsnmplibs
./install.sh (options: 1,2,3,8)
yum install togpegasus
service togpegasus start
domainname=group.company.com is needed in /etc/sysconfig/network - in case of
additional problems with hpsmhd start, in /etc/hosts should be put:
host-IP-address host.group.company.com host
... initial configuration “/etc/init.d/hpima sample [agent]” (or better,
reconfiguration, especially after each kernel update):
SAP, Linux, Virtualization and ... Itanium Popović Zoran](https://image.slidesharecdn.com/itaniumvirteng2linked-12643787088902-phpapp01/85/SAP-Linux-Virtualization-and-Itanium-17-320.jpg)
![/etc/init.d/hpmgmtbase reconfigure
... (and additional restart of services: tog-pegasus, hpmgmtbase, hpima, hplip, hmsmhd)
/etc/init.d/hpima reconfigure
/etc/init.d/hpima start
... kernel parameters and related settings and recommendations (by Oracle, SAP):
/etc/rc.local:
ulimit u 16384 n 65536
modprobe hangchecktimer hangcheck_tick=30 hangcheck_margin=18
/etc/sysctl.conf:
kernel.msgmni=1024
kernel.sem=1250 256000 100 1024
vm.max_map_count=300000
net.core.rmem_max=1048576
net.core.rmem_default=1048576
net.core.wmem_max=262144
net.core.wmem_default=262144
- old A/P storages should have excluesively Failover (not failback, not None) in the
prefered path in the presentation options
- multipath.conf example (starting the service and setting chkconfig also is needed):
/etc/multipath.conf:
blacklist {
devnode "^(ram|raw|loop|fd|md|dm|sr|scd|st)[09]*"
devnode "^(hd|xvd|vd)[az]*"
wwid "*"
}
# Make sure our multipath devices are enabled.
defaults {
udev_dir /dev
polling_interval 10
selector "roundrobin 0"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id g u s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
max_fds 65536
no_path_retry 12
user_friendly_name yes
}
blacklist_exceptions {
wwid "3600508b4000129f70002900002030000"
wwid "36005*"
wwid "36001*"
}
multipaths {
multipath { /* for newer A/A storages, from EVA VCS4.0 */
wwid 3600508b4000e302d0000a00001200000
SAP, Linux, Virtualization and ... Itanium Popović Zoran](https://image.slidesharecdn.com/itaniumvirteng2linked-12643787088902-phpapp01/85/SAP-Linux-Virtualization-and-Itanium-18-320.jpg)
![alias saptrnsys
}
multipath { /* for A/P storage, up to EVA VCS4.0 */
wwid 3600508b4000129f70002900000cb0000
path_grouping_policy multibus
path_checker readsector0
prio_callout /bin/true
no_path_retry queue
rr_weight priorities
alias sapbisemsys
}
...
}
devices {
device {
/* EVA 3000/5000 with new firmware, EVA 4000/6000/8000, EVA 4400 */
vendor "(COMPAQ|HP)"
product "HSV1[01][01]|HSV2[01]0|HSV300|HSV450"
getuid_callout "/sbin/scsi_id g u s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
features "0"
hardware_handler "0"
path_selector "roundrobin 0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 12
rr_min_io 100
path_checker tur
}
/* MSA */
device
{
vendor "HP"
product "MSA2[02]12fc|MSA2012i"
getuid_callout "/sbin/scsi_id g u s /block/%n"
features "0"
hardware_handler "0"
path_selector "roundrobin 0"
path_grouping_policy multibus
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
path_checker tur
}
}
- Data Protector installation (MUST be started from the PA-RISC disk, e.g. B6960-15008-
01.tar.gz or downloaded from ITRC SUM):
yum install xinetd (if not installed)
... add port 5555 in iptables firewall (/etc/sysconfig/iptables "-A INPUT -m state --state
NEW -m tcp -p tcp --dport 5555 -j ACCEPT")
./omnisetup source /root/hp/B69601500801/ server
dprotector.group.company.com install da,ma,cc
chkconfig list | grep omni
... if no service found, edit manually /etc/xinted.d/omni:
SAP, Linux, Virtualization and ... Itanium Popović Zoran](https://image.slidesharecdn.com/itaniumvirteng2linked-12643787088902-phpapp01/85/SAP-Linux-Virtualization-and-Itanium-19-320.jpg)
SAP, Linux, Virtualization and ... Itanium
- 1. SAP, Linux, Virtualization and ... ... Itanium Solution Proposal Popović Zoran, MSc Senior System Engineer 07.07.2009. 1. Introduction 2. Environment in this study with Itanium platform 3. Business Requirements 4. Solution Proposal 5. Realization options 6. Added value and benefits for here descibed business environment 7. References SAP, Linux, Virtualization and ... Itanium
- 2. Introduction First version of proposal based on a prototype system were available last summer. Many things happened in the mean time, one of which is the Red Hat's decision, by the end of last December, to discontinue support for Itanium (or IPF, Intel's Itanium Processor Family) in the announced future release RHEL 6. This was not so unexpected, and RH states that this decision is made because of the small number of Itanium systems sold in the recent period, and which doesn't tend to grow. But this story is not likely to end that fast (RH is fully supporting Itanium on RHEL 5 to the end of it's life in March 2014, while giving extended support until year 2017 through OEMs), Oracle bought Sun who has SPARC architecture and other hardware (there was a failed attempt to port Solaris to Itanium), enters the Xen advisory board (http://xenbits.xensource.com) with people involved in Open Source contributions, Oracle Enterprise Linux and Xen IA64 (Itanium) ports. RH6 is to be supported on IBMs POWER architecture, and so are other "traditional" RISC/CISC CPU architectures, x86 and x86_64 (http://kbase.redhat.com/faq/docs/DOC- 23403). There are no clear indications from other software and hardware vendors about Itanium's future (at least not the ones I deal with here having their support: HP, SAP and Microsoft), but there are current benchmarking results and research studies with predictions. Back in 2003 my company bought HP EVA 5000 storage and OpenVMS cluster with two Alpha ES47 nodes to support Oracle 9i RAC database for our legacy application, along with a rack of Xeon based Proliants (it's a kind of tradition, before them we had Alpha and VAX systems). A year after or more, it was decided that we should migrate to SAP. It was an ambitious project: we had 9-12 months from start to going live, implementing BW3.5, ERP 2004 with MM, WM (having MFC automated warehouse), FI/CO, SD, PP, QM, PM, TM, logistics and procurement, etc (maybe I have omitted something), everything that we had in the legacy Oracle application but HR, CRM and some remote locations (which were covered in later roll-out and optimization projects). We also implemented Solution Manager as project base and maintenance base, having all the necessary strict Q&A and GMP procedures. All this had to be supported with appropriate hardware and infrastructure. The first choice was (among others) was HP's offer with 10 Integrity (Itanium based) servers and HP-UX. Somewhere in the early beginnings, our project management decided to change OS platform to - Windows !!! Today, five years later, we have at least twice larger system (1TB database in size in production, number of users - up to 800 with expected growth, additional affiliate production plants, automated warehouses, end users, etc) working quite good with only minor changes (added RAM up to 16-24G in production, same number of CPUS: 2x mx2 1.1GHz per node), and some infrastructure improvements. We never had any serious unplanned downtime or system failures, performance, reliability, availability and stability was predictable (apart from some OS problems with Microsoft MSCS and one short storage outage). But, in the past year we had a large roll-out and optimization project involving ERP upgrade from ECC5.0 to ECC6.0, and it showed that project members, developers, IT staff and key users always needed (and still need) additional sandbox systems, IDES demo systems, system copies and similar for different purposes (from experimenting and unofficial testing, to training and Q&A, project preparation and operation, etc). This is showing the need for new level of flexibility which could be only obtained with virtualization. There were many options, from buying new physcial machines (very expensive even for less expensive architectures), manual consolidation (MCOS and MCOD, with or without Microsoft WSRM - Windows System Resource Manager), thin OS SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 3. (single kernel) virtualization (like VZ, Parallels Virtuozzo, but as we have strict security standards and procedures which conflict with OS patch level for VZ, this wasn't convenient even for testing), to full virtualization platforms. It turns out that with Itanium you can only have HP-UX VSE / Itanium VM or RHEL if you want official SAP support (as explained later). There is also a very interesting and sophisticated SAP solution - Adaptive Controller, but for now, Xen on RHEL is doing a very good job. I have two HP BL870c hosts with 6 active sandbox systems (homogeneous copies of ERP development, Q&A and production, BI/SEM IDES with Java instance, ERP IDES, etc) and they all work properly and very stable. From pefromance standpoint, HP-UX with Windows guests doesn't offer more at all (without AVIO drivers, similar to PV guest drivers on Xen which don't exist only for Itanium), the only thing I miss at the moment is Live Migration on RHEL/Xen (which Integrity VM supports, as it also has some other nice-to-have features). But I am able to move virtual machines to different hosts manually without a problem (multipathing and shared storage does it's work). SAP supports virtual guests, without responsibility about their performance, and database on production systems should be on physical servers if maximum performance is needed - otherwise, there aren't important reasons not to do it even with database. We use Oracle 10g at the moment (10.2.0.4) on Windows, which works fine on guest systems - but because of Windows, it has to be HVM, full virtualization. For optimum performance (and many other good reasons), the best option would be migration to RHEL, including production systems (all the tests show that it would be a smooth transition). There is a general trend about migration of SAP systems from Unix to Linux (for all the good reasons), while migration from Windows is less popular. Thing is that 60% of all recently sold Itanium servers are Unix (read: HP-UX), 35% are Linux (RHEL and SLES), and only 5% are Windows (http://www.infoworld.com/d/open-source/red-hat- will-drop-itanium-support-in-enterprise-linux-6-083). Microsoft has no intention to introduce Hyper-V on Itanium (as Citrix also doesn't currently, because the Xen code branch they bought didn't cover ia64), but it is supported in all important flavours on Windows Server 2008. Migration to a different hardware platform is not an option, just as migration to HP-UX at the moment (current number of Itanium servers does not justify that risk and expense). But, the old aging hardware should be either upgraded (but it costs – additional CPU costs almost as one BL860c, and though it's partly consequenceee of local vendor's clumsiness, it is not very reasonable), or replaced with new servers – or, used for non-production (until dead), which is aligned very good with virtualization (RHEL/Xen paravirtualization is available for old Madison cores, but HVM isn't). There are better reasons to migrate to RHEL beside this (money saving driver) – decision makers can decide to change CPU architecture at one point, but with RHEL we can already have the level of flexibility we need – far better than in any other option: HP-UX is too expensive, and Windows doesn't offer it. One important aspect is coming from GMP and other compliance issues, so we need same platform both on test (Q&A) systems and in production, and this justifies even more migration to RHEL5. So, the starting point was: SAP ERP and BI systems on Windows Server 2003 on HP Integrity servers with HP EVA virtualization solution, and the current stop is: some of those systems working on Xen fully virtualized guest machines on Windows Server 2003 and on HP Integrity blade servers, and some working on paravirtualized guest machines on RHEL5. Next stop would be more systems involved, and finally – everything migrated to RHEL, either on Xen virtual machines or bare metal. Also, I find that Integrity blade servers are far more affordable (comparable to Xeon based Proliants) than expensive mid-range cell based and other Integrity flavours. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 4. Environment in this study with Itanium platform In an enterprise I am refering to here, IPF (Itanium Processor Family) architecture is being used exclusively the needs of SAP systems, there are 18 such servers at the moment (10х HP rx4640 + 2х rx2620 with Madison cores, 2х rx640 and 2х BL860c with Itanium dual-core Montecito cores, 2х BL870c with Montvale cores), beside additional x86 HP Proliant DL360 G4/5 servers (4x without additional servers for support SAP routers and Web Dispatchers which are in Microsoft Cluster for production just as SAP central services on production and test systems, or on VMWare platform for support and other purposes). Beside VMWare as main VTI for x86/x86_64 processor families, HP EVA Storage nodes are used as form of FC SAN Storage virtualization (also using MSA2012 for VMWare, additional EVA 8400 obtained, but not yet ready for production). SAP landscape currently consists of: − ERP systems (development, test (Q&A) and production), ECC6.0 Ehp3 − BI systems (development, test (Q&A) and production), BI7.0 SP18 − CEN system (transport domain controller, CUA, partly central monitoring), NW04 − Solution Manager (MOPZ, EWA, monitoring, project roadmaps, ticketing, and other) − SAP routers (for end users, and for support and external access) − SAP Web Dispatchers (BI portal, WebGUI) − network printing servers with SAPSprint, SAP Web Console for some RF terminals − different sandbox and other systems: homogeneous system copies, traning systems, IDES systems, etc. - all working as Xen guests at the moment Equipment has life time (it is aging) Among these 18 servers, 12 servers are entering 5th year of usage: rx4640/2620 servers on which lies the main SAP landscape (it consists of ERD/Q/P, BWD/Q/P systems). All these servers are very reliable and stable – there never was any serious hardware failure or issue on production servers (or even any incident as a consequence) until this day ! But, with hardware aging, support becomes more expensive, additional components or spare parts also become very expensive, and also vendor desupport dates are getting closer and closer for equipment and different functionalities (usual practice is to have maintenance renewal periods and IT equipment amortization within 3-5 years, but sometimes it makes sense to extend it). There are at least two possible roads: continued use and planning about Itanium platform, or migration to another processor architecture (as mentioned). If such migration is planned, then all other technological aspects and existing options must be taken into account (some less demanding and ”painful”, some are not, but offer different advantages and overall total cost – just as changing the OS platform, changing the server vendor, storage vendor, etc. should be considered as well). Some kind of trade-in model or amortization were not practice up to now as they are not offered by all available vendors currently (used or vendor-supported and refurbished equipment is usually not an option in production systems in general) and probably it also will not be in respective future. There is an option to stay on Itanium platform by sole migration to Itanium blade servers which has for consequence mandatory need for additional servers based on current requirements for the main landscape, but without all the other additional systems (this is not justified and will be explained further on). SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 5. Itanium servers – unused brute force Main argument for Itanium system application is usage within OLTP systems and databases – an example: for CPU patch 19 for Oracle 10.2.0.4 based on standard README instructions, downtime lasts about half an hour (given for some internal Oracle tests for a database with 2000 views and 4000 objects, while our production system has at least twice as more), on a production system with old Itanium Madison 1.1GHz CPUs (2x mx2 CPU) it lasts about 5 minutes. But, except for biggest loads during the day (peak is usually 13:00-15:00 on work days and in the end of the month), this power mostly remains unused during the rest of the day (average CPU load on our production does not go over 10% on central instances with database and central service nodes, and up to 35% on dialog instances, having at least 400 active users among 800 users) and this is expected state. Consolidated CPU usage and other server resources usage can be realized by installing additional SAP or DB instances on the same OS instance (operating system instance, or in general OE, Operating Environment) on the same server, allowed by SAP MCOD or MCOS installations – but, all there many possible problems of coexistence of such instances during their work, usage and maintenance life-cycle. That is one of the reasons why consolidated OS environments should be isolated and that is usually done through some form of virtualization. It can be justified above all simply by using non-productive and productive systems and their total cost – but, completely virtualized production environments have became a standard experience today and also a need for many business environments (if it is possible to scale resources and estimate bottom lines, then it is just necessary to take into account the “price” for virtualization). Existing solutions for virtualization which were considered in this text: • Citrix XenServer • Hyper-V / Win2k8 • HPUX VSE/VM, vPars, nPars • Parallels Virtuozzo / OpenVZ • Novell Suse / Xen • Oracle VM • Fedora / Xen • FreeBSD / Xen • Red Hat Enterprise Linux (RHEL), and so is Centos / EPEL / xenbits • Sun Virtual Box • QEMU/KVM • VMWare (there were also other solutions which were not considered for obvious reasons as described bellow) Some solutions were eliminated from the start because those are not available for IPF and are not even planned to be supported and working on Itanium any time near (Hyper- V, VMWare, XenServer, Virtual Box, and Oracle VM up to some point) but only x86/x86_64 (the environment I am refering here to is mostly based on these platforms and Windows OS, apart from some remaining OpenVMS and Alpha systems). Following criteria are applied on remaining solutions: • only full virtualization (hardware based, HVM) or paravirtualization (PVM, more SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 6. favoured) with hypervisor is supported (solutions which provide full isolation) – HP vPars, Virtuozzo, OpenVZ represend OS / “One Kernel“ virtualization which is not supported (by SAP and other vendors) on production systems • all other official requirements given by SAP (as a software vendor for our production systems), HP (as the main hardware vendor for the business environment I am refering to) and Oracle (as the main software and database (RDBMS) vendor here) about platform support • fully working installation prototype which confirms solution feasibility in our system environment, e.g. boot from SAN After this all these constraints it has turned out that there only two possible usable solutions for Itanium: RHEL/Xen and HP-UX Integrity VM (or nPars and vPars but only on HP-UX). Their comparison with bare metal native systems follows bellow (prices represent only estimations, in EUR): license / price Support Solution options** production (EUR) 1 year 1000/ RHEL / Xen 0 partially yes socket yes, but no HP-UX Integrity VM 10000* / CPU 1000* yes PV drivers HP-UX nPars 10000* / CPU 1000* yes yes HP rx3600 (4G RAM) 2 core 14000 - yes HP rx6600 (8G RAM) 4 core 24000 - yes HP BL860c 4 core (4G RAM) 5000 500 - yes HP BL870c 8 core (8G RAM) 13000 1000 - yes HP rx7640 16 core + VSE 115000 - yes Table 1 Prices given here were extracted from pre-sales and based on old proposals (* - brute estimation, mostly low bound). One must have in mind (based on proposals from HP by the end of year 2008) that memory prices for Itanium blade servers is about 240EUR per 4G RAM quad (officially refurbished and similar options are available and have 2-3 times lower price, good enough for non-production purposes), and this is a very critical resource for virtualization – while the memory price for older (non-blade) Integrity servers is even up to 5-10 times bigger. For HP-UX it is also mandatory to have additional expenses for HP installation and maintenance support including additional preparations, training and similar activities which are not needed for RHEL (though, HP- UX systems offer high level of availability, manageability and more, they represent top of the business and industrial standard, but it is very questionable if this is really needed). RHEL Xen still does not support many advanced features** (like Live Migration, PCI passthrough, paravirtualized Windows drivers – HP-UX doesn't support those drivers too, etc) which are easily enabled on other platforms (x86, x86_64) or which HP-UX supports, not likely that all of the will be, but some might be. Furthermore, HP BL870c can support almost 4 completely isolated active guest systems (even on physically different processors, kind of CPU partitions – HP nPars can not go in granularity under the physically available resources, including CPU, Integrity VM can), and without a significant performance fall it can support up to 7-15 such guest systems SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 7. (having 16 HT cores) compared to 1-core based bare metal systems, and more if high- end performance and load is not expected. Of course, number of inactive guest systems is almost unlimited – ready to be started and used when needed at any time if resources are available, without additional reinstallation, side-effects and server setup. Business requirements Business requirements which emerge during development, testing and usage of all SAP systems are putting high demands about fast changes and mechanisms which enable such flexibility having all the existing business standards and SAP usage standards preserved (GMP and validation practice, QA, availability/reliability SLAs above all, etc). Specific requirements which arise from these are: • one or more system copies in a given time (usually ASAP), homogeneous or heterogeneous – as part of the strategy for testing and/or development (but not for Q&A needs), and by given requirement or expressed need (testing of new or risky features, system types/usages and functionalities) • change of the existing architecture / SAP Landscape – for instance, the problem of the peak loads on the Q system during testing periods (systems were never sized for these needs), Support Pipeline (to circumvent transport change and validation requirements in order to speed up cut-over after system upgrade, roll-out and similar), training and sandbox systems – last two examples show in our practice that system copies are far more efficient and usable than any other solution (aligned with storage split-mirror technologies like snapshots and snapclones) compared to unjustified client copies or exports on systems with already more than 1 TB in DB size, which can not support frequent requests for Q system data refreshments. One temporary solution is also to combine those two procedures (as used in R/3 system upgrade to ECC6.0 during year 2009), and even more efficient would be to slightly change validation practice (which in essence remains same, but using far more efficient system copies). This is aligned with SAP note 432027 - Strategy for using SAP Support Packages, which describes additional sandbox and test system usage (e.g. sandbox in Upgrade Master Guide) and as part of the official landscape. A step further (a solution and a problem) would be involvement of additional development and Q&A systems, but also using more efficient change management and automated test procedures using SAP Solution Manager and eCATT tools which we already have available at hand without additional investments (but there also many, many other useful tools) except for additional planned effort by validation and different SAP teams. Every testing of changes during any of the SAP projects is a convenient opportunity for preparation and implementation of such tools and solutions. • Changes during system patching or system upgrade which require alternate system landscapes (as during upgrade in year 2009), etc. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 8. Solution proposal Pool of Servers The strategy which enables the fastest possible response on large number of requests: certain number of servers is prepared like an “on hold“ template (firmware / hardware, FC / network, SAN storage and other needed resources, then the OS installation, Windows domain name reservation and AD infrastructure settings, antivirus / firewall / proxy, account specifications and all other OS settings, Oracle RDBMS and SAP s/w installation), and then by a well defined procedure a copy is made using HP Storageworks Business Copy technology in a shortest time according to SAP standards (homogeneous copy based on database copy, with additional post-installation activities which last not more than an hour). Server in the pool can be activated in a very short time, deactivated or renewed with fresh data (just by making database copy and post- installation tasks again). Many use both physical (native, bare metal) and virtual guest systems equally, based on specific needs and requested system types. Also, this can be mixed, for instance, as a phase in the implementation of virtual guests for productive systems – database is left on physical platform, while dialog instances (application servers) work on virtual guests, which is a well known and well used practice. In the meantime, hardware and virtualization platforms grew enough to support completely virtualized systems which can be used without special constraints (apart from the aforementioned performance factor). Oracle gives also additional opportunities with RAC (Real Application Cluster) option in terms of scalability and HA (which is additionally licensed). This option is useful also with physical servers, and with MCOD (Multiple Components on One Database - one database for many SAP systems) gives also a solution with optimal performance (it has also same downturns as for consolidated systems, where patching and maintenance introduces risks for many systems). Virtualization Xen originates from the 90-ies as a Cambridge University project, and also as VMWare today it represents one of the best known virtualization platforms. The idea of virtualization exists for decades (starting from 60-ties and IBM which introduces it as part of OS for mainframe systems). In the meantime Xen code branch was acquired by Citrix, and similar technology is also used by Microsoft Hyper-V VTI platform. Beside those, Open Source / GPL code branch of the Xen development remains (currently version 3.4 is available) which is used by many Unix/Linux distributions, among which also does RHEL (RHEL5.3/5.4 is packed with and using Xen 3.1/3.2). It is announced that RHEL 5.4 and following updates are also going to support KVM (Kernel based Virtual Machine) as the hypervisor (but not on Itanium so far), with possible improvements for paravirtualized drivers for Windows guest systems under IA64, PCI Passthrough is already available with VT-d on other Xen platforms, and other improvements. Xen can offer a very good solution with minimal investments in memory hardware on some servers, and in support subscriptions for RHEL OS (more details follow in next section). RHEL as a base platform support all important requirements which are implemented and expected from servers on Windows Server 2003 currently (e.g. SAN FC redundant multipathing for HA standards, which is also useful for a form of migration of virtual server – GFS/Cluster options are not needed so far but are available (specially for SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 9. PV systems), having guest HA mechanism yet to be tested). SAP note 962334 shows that SAP supports this platform (RHEL, Xen), but additional checks and tests would be needed for production systems. In any consolidation option this virtualization solution on Itanium platform is more optimal than any other, and it is also complementary to existing VMWare infrastructure as a proven solution for other platforms. Xen and VMWare can support all possible given requirements in the environment I am discussing here. With these technologies (together with Charon/SIMH emulators for Alpha/VMS systems, which are already tested on some VMWare guests) it is possible to support virtualization infrastructure for all platforms and systems in I was mentioning. In the following ROI analysis available in the text bellow, it was not taken for whole infrastructure with complete TCO structure where virtualization can potentially bring only bigger index (which has to be validated in each special case – for Oracle above all): • the advances of licensing – many vendors give special licensing policies for guests, all somehow licensing expenses can be lower or somehow tend to favour virtualization trends, for HP VSE is is known that it can be based on partitions (but has to be checked with vendor or partners) • simplified system maintenance, etc. First prototype – testing has already started Currently, we have a system working non-stop and being actively used for second month without complaints (after the upgrade of the production system), the SAP-TEST guest system (as a replacement for the previous ERT system in the alternate landscape, 8 VCPUs on 8 virtual cores or 2 dedicated CPUs, and with 6G RAM so far) on BL870c host (SAP-ERQ6) and no interruptions – above all, very stable and reliable, in the basic RHEL5.3 version (host was not updated or patched so far). System was successfully presented with copy of the production OLTP database, and the complete installation of the guest Windows 2003 EE SP2 system was done by the standard operational procedures. It is even possible to use some of the standard EFI tools on the guest, a successful manual migration was tested, and a copy template system was also made (on SAP5), by which all important functional requirements were fulfilled for using virtualization platform in our environment. The only problem with this solution (as mentioned about the missing features for Xen on Itanium) is a very poor disk I/O. This is the consequence of the needed full virtualization (HVM) for Windows guests (3-6 times slower than on bare metal, while network I/O is less deteriorated). This makes critical impact on database performance in some cases (though parsing is done faster than on older bare metal systems), which makes the whole ERT system in average slower (e.g. closing the financial period in SAP takes 4 hours in batch instead of 2.5 hours as in production system). There are many workarounds and solutions for this problem - using paravirtualized I/O drivers if it was possible, iSCSI/NFS storage approach (not top performance, again), etc - but the best thing would be moving database to physical machine, using MCOD wherever possible. Database nodes wouldn't be easily reallocated, but one of the possible solutions in future would also involve adaptive infrastructure (SAP ACC, with Solution Manager - Adaptive Computing Controller) as part of the SAP strategy for system usage and consolidation: • all SAP systems can adapt to business needs very fast and in a flexible manner SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 10. • dynamic load balancing according to system load (physical and virtual) • easier SAP landscape maintenance • HP-UX prototype is being prepared (the goal is to make sample comparison with Xen) All these goals can be accomplished without SAP Solution Manager ACC plug-in, of course, as proposed in this paper with some additional manual procedures and way of system maintenance. Optimal solution would involve ACC because not only that it avoids better possible human errors, but it also follows vendor recommendations and standards better given with SAP Solution Manager as the base technical tool which is really useful in all system maintenance and management tasks according to GMP / ITIL / ITSM, and which is free of charge (unless additional Service Desk Extended component and functionalities are needed for non-SAP systems and Service Desk / Help Desk, and similar). Therefore it makes sense to make additional effort and improve this system and it's usage in our environment. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 11. Realization options These are the only existing usable IPF consolidation platforms and their perspectives: • acquisition of new physical servers (without hypervisor and virtualization) Pros: most efficient (but not optimal) use of existing knowledge and resources Cons: price of 1 Itanium server is between $5К and $20К in average, and more (without support agreements and other hidden maintenance costs, human labor) • HP-UX Integrity VM hypervisor: o HP Integrity VM (software virtualization) as part of VSE (Virtual Server Environment) offers also full isolation, as opposed to vPars (similar to OpenVZ and appropriate Parallels Virtuozzo solution) o HP nPars as part of VSE (cell based virtualization, similar to IBM LPARs, HP-UX as hypervisor is only controlling them), demanding specific hardware (rx7640 or stronger, like Superdome) Pros: a highly stable platform which represents on of the leading industry standards Cons: the licenses only for MC OE (DCOE, VSEOE now) are about $10000 per CPU (TCOE as a possible minumum is more than $2000 per CPU), performance factor • Xen with RHEL5 as hypervisor Good side: supported by SAP and HP, Open Source, quite stable and tested, in- house knowledge available. Cons: does not support all the features other options have (at least some of the features like live migration), performance factor All these solution are stable and robust, and some details about maintenance and support for RHEL / Xen infrastructure are given in the table bellow (prices are given in EUR, per host) – RHEL Support is excellent and reliable at least as any other vendor's support, with some advanced maintenance and provisioning features including standard ticketing and case management: Virtualized 1 year 3 year Type options Guests support support 1 x Red Hat Enterprise Linux 1. RHEL Advanced 1 x Red Hat Cluster Suite Platform, Standard 13x5 unlimited 1 x Red Hat Global File System 1199 3057 (unlimited sockets) 1 x Red Hat Network Update Module 1 x Red Hat Enterprise Linux 2. RHEL Advanced 1 x Red Hat Cluster Suite Platform, Premium 24x7 unlimited 1 x Red Hat Global File System 1999 5097 (unlimited sockets) 1 x Red Hat Network Update Module SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 12. Virtualized 1 year 3 year Type options Guests support support 3. RHEL Standard 1 x Red Hat Enterprise Linux Platform, Standard 13x5 4 1 x Red Hat Network Update 639 1629 (2 sockets) Module 4. RHEL Standard 1 x Red Hat Enterprise Linux Platform, Premium 24x7 4 1 x Red Hat Network Update 1039 2649 (2 sockets) Module * DVD prices for Red Hat Enterprise Linux 5 (for Intel Itanium) Media Kit: around 30 eur Table 2 For our testing purposes the 3. type of subscription would be sufficient, specially in first phases of realization (HA features are not needed yet, and so Advanced supported options for GFS / Cluster Suite, we can rely on FC SAN / EVA Storage, and we are not using RHEL PV guests at first), but as we advance and increase number of systems, and as we plan to test RHEL OS as guest platform, (on stronger servers) it makes sense to take the type 1. support subscription. Prices are estimated for one server – minimally 1-6 subscriptions can be considered (that is the number of servers available for HVM for Windows guests, not all should be covered from the start during testing, old servers support only RHEL PV guests). The smallest expense is 640 euros, some optimal expense is 1920 or 3840 a year at the moment (for HP-UX TCOE licenses themselves are 6-7 times more expensive, or up to 30 times for MCOE, without support – for more exact information can be obtained from the vendor or his partners). It always remains possible to change or move to a bigger number of subscriptions. RHEL or his partners might also provide us with experienced consultants in order to get help with some phases of implementation. There is a possibility to obtain RHEL support from HP as hardware vendor indirectly, but I would not recommend this (as it is not justified). I do not have prices for this option, but having our environment and level of support in the nearby call center (proactive support for SAP systems hardware) I recommend obtaining subscriptions directly from RHEL. We might get some additional suggestions from HP in the meantime, though. Realization phases In general, following implementation gradual phases and steps are proposed in a short overview: 1. Prototype preparation and testing (in process). 2. Virtualization of all IA64 sandbox (and test) systems and preparation of pool of servers (for system copies), Solution Manager (SOD), dialog instances of development systems. 2.1. In parallel, all 32-bit SAP servers on VMWare platform should be virtualized (old Solution Manager, IDES, CEN; even SAP routers, but not those for end users, for a start). 3. Complete virtualization of development systems. 4. Test drive virtualization for Q&A and production systems (they must be aligned, dialog instances first). 5. Complete virtualization of the whole SAP landscape. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 13. Taking Q&A servers together with productive servers is necessary because of the nature of their intended usage. The last phase is not yet planned (if it ever will be), at least not with a final date or specific requirements at the moment (higher reliability (disaster recovery), greater security and scalability). This might justify HP-UX as an option as it's implementation has references nearby, with full implementation and integration support from HP in a critical environment (with given dates). For the successful implementation of these phases, good preparation is crucial, specially about planning needed resources: • storage space • available network and optical ports and switches, settings on firewall, etc. • needed total number of servers in the landscape (categorized according to usage types and associated with possible hosts) • this provides information needed to estimate people-days needed for implementation of the pool of servers in each of the phases, and also physical resources needed for the requested service levels according to user requirements. Much of storage space is saved using HP BC Snapshot functionalities (as during upgrade), but these also have some level of impact on EVA performance and puts constraints on original VLUN, and makes storage maintenance more complex. Price of the EVA FC disks is about 15-20 eur/G which also has to be noted (snapshot grows up to about 10% of the original disk size in a normal usage period, which gives around 1.5-2 eur/G) – there are other storage options, but there also other parts of the TCO structure: • backup (about 7 eur/G in average, but varies from 1 to 15eur) • network equipment and security (antivirus, firewall, LAN, FC), licenses • guest OS licenses and support agreements • infrastructure in the data center room (space, air conditioning, power supply) • system maintenance (human resources, licenses for some monitoring solutions) • other hidden expenses ... Therefore, it is of the utmost importance to have business requirements carefully prepared and estimated, because all real technical requirements (like additional hardware or licenses, memory, servers, storage). For instance, if large enough number of servers is hosted (which grows very easily with virtualization), so grows the I/O bandwidth on SAN or the number of storage spindles. Validation practice and documentation is very important and the mandatory prerequisite in order to make everything work, among all the other standard operational procedures and working instructions – SOPs for system copies, virtual hosts and guests installation, pool of servers qualification above all, their maintenance, and all other standards that we follow give a good base for equal and smooth usage of virtual systems among the usage of all our systems and their maintenance. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 14. Added value and benefits for here described business environment The total cost of one server is certainly larger if seen through it's physical partitions (e.g. two BL860c are approximately physically same as one BL870c), not taking into account the support agreement price, and the return of investment grows with each added virtual guest system. For instance, if one BL870c server hosts 4 virtual guest systems, with about 1500 euros per year for support for RHEL and about 1500 euros for 16G RAM (giving 8G of RAM for each virtual system given the current state, which is enough for every standard SAP installations), we get 4 virtual guest systems with power similar to 4 physical BL860c servers with 2 CPUs given their level of usage (price of one such is in the order of 5000 euros, while the price of BL870c is almost three times as that). Then, the 5th server gives around 100% ROI (or, respectively, 3rd server if the price of host is already taken as what we have), under condition that the usage level of each guest does not go over the current usual usage level of BL870c (which is the case we have) and so the price of one virtual server equals with one physical partition in CPU power (about 90% of 1 BL860c). If we add to one such host additional 3 active guest systems with the investment in total 32G RAM (2000-3000 euros), ROI goes over 200%, and so on. In that manner we can go up to added or return value of at least 100000 euros (depending on the given ROI model) with existing 2-6 host servers and up to 20-28 virtual guest systems (with minimal total investment of 5000 euros, having not quite optimal but well balanced usage of physical resources). This goes, once again, only under condition that the total required active guests load is not greater than the available host resources (CPU can not be added into BL servers, but can be added to Integrity rx servers). If this solution is seen as part of the pool of servers, value can be far more bigger (not all systems are needed active) ! With HP-UX option, this ROI decreases drastically, or it almost diminishes. So, the number of all inactive servers is also very important having no significant limitation in their number (only about their storage space). Virtualization enables also dynamic resource allocation and balancing physical resources, but not more than that – host can get overloaded if this is not carefully planned, and sometimes only the physical consolidation is possible. This can be illustrated by following: two BL870c have the same number of SAPS (SAP's standard benchmarking system measurement, above about CPU power, but also about all other physical resources) as all 6 productive Integrity rx4640 servers have, but in such a physical consolidation it is not possible to have two MSCS clusters (not even with two additional BL860c), and there is no isolation between database and dioalog instances, and HA (High Availability) is then not justified – while these servers can work excellent as virtualization platform for the same purpose. What do we get, beside the obvious added value: • Flexible system maintenance and usage according to business needs (which is not possible otherwise, due to expenses, human resources or for other reasons). • SAP landscape remains platform homogeneous in the as much as possible extent (all guest systems and physical systems remain on the same IA64 platform, but some h/w differences remain – though, SAP and Oracle mostly rely on OS which is not that dependent) • Optimally used existing Itanium servers (both old ones and planned new ones, according to usage – no easy “throwing away”), better consolidation later on. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 15. Details about implementation Installation of current environment is divided in three groups: • installation of hosts (Dom0s) • installation of HVM (fully virtualized) Windows guests • installation of PV (paravirtualized) RHEL guests While the installation of hosts and PV guests has many similar steps (network setup – I am using trunk utility as local NTLM proxy for http/ftp, RHN registration, ntfs-3g uitilities installation, etc), there are things to be set only on hosts (xend service and boot parameters, HP support pack installation, etc). For guests I am generally proposing using copies of the template systems for many different reasons, one of which is using good features of EVA storage (snapshots and snapclones) or LVM on local disks. If making guest from the scratch (or new template), it is always best to use physical (phy:/) drives instead of files – they have better performance, enabling moving to another host (with shared storage), and always use multipathd. I am giving the following hints for people who have experience with Windows installations and SAP administration on Itanium (and who have some basic knowledge and awareness of things about Linux) – many things are similar, but there are also many misleading details. SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 16. HOST INSTALLATION - RHEL installation is quite simple and straightforward – no EFI boot driver is needed (Windows installation needs it), usual server preparation, firmware update if needed, boot device configuration in EBSU setup disk – boot from SAN is used, so HBA adapater should be configured there and a VLUN prepared, after booting from installation disk, installer should be started manually by entering “linux mpath vnc=1” (vnc is optional, GUI is then available later through vnc client on screen 1, or http://that-host:5901) - installation number can be entered after the installation (using rhn_register), Virtualization group should be chosen if available (beside Development, Web Server and others) - or later, packages to be installed are: xen, kernel-xen, xen-ia64-guest-firmware, virt-manager, libvirt and virt-viewer, or just “yum groupinstall Virtualization” (if doing it manually with rpm -ivh in VT folder of the installation disk: libvirt-python, python-virtinst, Virtualization-en-US, perl-Sys-Virt, xen-devel ../Server/bridge-utils ../Server/xen-libs ../Server/gnome-python2-gnomekeyring ../Server/iscsi-initiator-utils ../Server/gtk-vnc ../Server/gtk-vns-python ../Server/cyrus-sasl-md5) ... installation lasts up to 1 hour – all setup parameters (like network address, gateway, etc) should be prepared ... check if /boot/efi/efi/redhat/elilo.conf is set correctly before restarting ... /etc/resolv.conf should contain at least one line "nameserver DNS_IP_address" and one "domain group.company.com", enabling and disabling network interface is done with ifup eth0 / ifdown eth0 - on some bad terminal emulation/clients after restart, firstboot gets stuck (additional setup wizard), which shuts down automatically after about 10 minutes, and after logging in it can be disabled with “chkconfig firstboot off” - trunk (local NTLM proxy, http://ntlmaps.sourceforge.net) after unpacking with tar -xzf has to be configured (NT_DOMAIN:HEMONET, USER:proxyuser, PASSWORD:somepass, PARENT_PROXY:proxy_server) and started manually with “scripts/ntlmaps &” (or put into /etc/rc.local to start automatically after boot), then it is possible to start “rhn_register –proxy=http://127.0.0.1:5865” - also, for other utilities accessing internet, like yum (package installer), edit ~/.bash_profile (using gedit in GUI console, or vi editor): export http_proxy=http://127.0.0.1:5865 export HTTP_PROXY=http://127.0.0.1:5865 export ftp_proxy=http://127.0.0.1:5865 export FTP_PROXY=http://127.0.0.1:5865 ... update can be started with “yum update” after setting up software channels and other additional settings on RHN side (if needed) NOTE: LVM/multipath is not good is not good with FAT partitions, so before each kernel update/installation it is needed to mount manually /boot/efi (e.g. with “pvs” one can get sytsem disk device for system VG LogVol00, like /dev/sd..., for instance /dev/sdh2, and then EFI partition is mounted with: “mount -t vfat /dev/sdh1 /boot/efi”) - using Windows (cifs) shares: e.g. serverx$ can be mounted on /mnt/tmp with: mount -t cifs -o dom=HEMONET,username=someuser //server/x$ /mnt/tmp ... or for simple copying smbclient is useful (like ftp) SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 17. - ntfs-3g installation (http://www.ntfs-3g.org): cp /mnt/tmp/rhel/ntfs3g*.tar tar xvf ntfs3g.... cd ntfs3g... ./configure make make install yum install ntfsprogs - firewall should be configured (opening needed ports, systemconfigfirewall) and selinux (for a start and troubleshooting, “setenforce 0” and SELINUX=permissive in /etc/selinux/config) - Xen Dom0 configuration for several network adapters, create /etc/xen/scritps/network- multi-bridge: #!/bin/sh dir=$(dirname "$0") "$dir/networkbridge" "$@" vifnum=0 bridge=xenbr0 netdev=eth0 "$dir/networkbridge" "$@" vifnum=1 bridge=xenbr1 netdev=eth1 "$dir/networkbridge" "$@" vifnum=2 bridge=xenbr2 netdev=eth2 "$dir/networkbridge" "$@" vifnum=3 bridge=xenbr3 netdev=eth3 ... and then: “chmod +x /etc/xen/scritps/networkmultibridge” and edit /etc/xen/xend-config.sxp, putting “(networkscript networkmultibridge)” instead of “(networkscript networkbridge)” ... also, put there (dom0-min-mem 2048) ... and dom0_mem=2G into elilo.conf to reserve 2G for Dom0 and avoid memory ballooning (2G should be enough) – elilo.conf example: ... image=vmlinuz2.6.18164.10.1.el5xen vmm=xen.gz2.6.18164.10.1.el5 label=2.6.18164.10.1.el5xen initrd=initrd2.6.18164.10.1.el5xen.img readonly root=/dev/VolGroup00/LogVol00 append="dom0_mem=4224M palhalt rhgb quiet" ... - vncserver setup (/etc/sysconfig/vncservers), vncpasswd should be set for the running account, and initial vncserver start (“chkconfig vncserver on” for the service, too) - hp support pack installation: yum install netsnmp netsnmplibs ./install.sh (options: 1,2,3,8) yum install togpegasus service togpegasus start domainname=group.company.com is needed in /etc/sysconfig/network - in case of additional problems with hpsmhd start, in /etc/hosts should be put: host-IP-address host.group.company.com host ... initial configuration “/etc/init.d/hpima sample [agent]” (or better, reconfiguration, especially after each kernel update): SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 18. /etc/init.d/hpmgmtbase reconfigure ... (and additional restart of services: tog-pegasus, hpmgmtbase, hpima, hplip, hmsmhd) /etc/init.d/hpima reconfigure /etc/init.d/hpima start ... kernel parameters and related settings and recommendations (by Oracle, SAP): /etc/rc.local: ulimit u 16384 n 65536 modprobe hangchecktimer hangcheck_tick=30 hangcheck_margin=18 /etc/sysctl.conf: kernel.msgmni=1024 kernel.sem=1250 256000 100 1024 vm.max_map_count=300000 net.core.rmem_max=1048576 net.core.rmem_default=1048576 net.core.wmem_max=262144 net.core.wmem_default=262144 - old A/P storages should have excluesively Failover (not failback, not None) in the prefered path in the presentation options - multipath.conf example (starting the service and setting chkconfig also is needed): /etc/multipath.conf: blacklist { devnode "^(ram|raw|loop|fd|md|dm|sr|scd|st)[09]*" devnode "^(hd|xvd|vd)[az]*" wwid "*" } # Make sure our multipath devices are enabled. defaults { udev_dir /dev polling_interval 10 selector "roundrobin 0" path_grouping_policy group_by_prio getuid_callout "/sbin/scsi_id g u s /block/%n" prio_callout "/sbin/mpath_prio_alua /dev/%n" path_checker tur rr_min_io 100 rr_weight uniform failback immediate max_fds 65536 no_path_retry 12 user_friendly_name yes } blacklist_exceptions { wwid "3600508b4000129f70002900002030000" wwid "36005*" wwid "36001*" } multipaths { multipath { /* for newer A/A storages, from EVA VCS4.0 */ wwid 3600508b4000e302d0000a00001200000 SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 19. alias saptrnsys } multipath { /* for A/P storage, up to EVA VCS4.0 */ wwid 3600508b4000129f70002900000cb0000 path_grouping_policy multibus path_checker readsector0 prio_callout /bin/true no_path_retry queue rr_weight priorities alias sapbisemsys } ... } devices { device { /* EVA 3000/5000 with new firmware, EVA 4000/6000/8000, EVA 4400 */ vendor "(COMPAQ|HP)" product "HSV1[01][01]|HSV2[01]0|HSV300|HSV450" getuid_callout "/sbin/scsi_id g u s /block/%n" prio_callout "/sbin/mpath_prio_alua /dev/%n" features "0" hardware_handler "0" path_selector "roundrobin 0" path_grouping_policy group_by_prio failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 path_checker tur } /* MSA */ device { vendor "HP" product "MSA2[02]12fc|MSA2012i" getuid_callout "/sbin/scsi_id g u s /block/%n" features "0" hardware_handler "0" path_selector "roundrobin 0" path_grouping_policy multibus failback immediate rr_weight uniform no_path_retry 18 rr_min_io 100 path_checker tur } } - Data Protector installation (MUST be started from the PA-RISC disk, e.g. B6960-15008- 01.tar.gz or downloaded from ITRC SUM): yum install xinetd (if not installed) ... add port 5555 in iptables firewall (/etc/sysconfig/iptables "-A INPUT -m state --state NEW -m tcp -p tcp --dport 5555 -j ACCEPT") ./omnisetup source /root/hp/B69601500801/ server dprotector.group.company.com install da,ma,cc chkconfig list | grep omni ... if no service found, edit manually /etc/xinted.d/omni: SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 20. service omni { socket_type = stream protocol = tcp wait = no user = root server = /opt/omni/lbin/inet server_args = inet log /var/opt/omni/log/inet.log disable = no } ... INFO: linux rescue is a method of disaster recovery, by booting installation disk with “linux rescue” or better, “linux rescue mpath”, and it is also possilbe to do a reinstallation or upgrade over the existing OS file system (as a repair method) by choosing any of these options, guided by the installer dialigues !!!!! NOTE: always, always make backup of everything that is important ... SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 21. PV GUESTS - GUI installation is also available (vnc) as with Dom0, the only important difference is that NFS/FTP/HTTP can only be used as the installation source - installation procedure is as easy as with Dom0, Virtualization group should be select if given (installation number can be entered afterwards with rhn_register) and in /boot/efi/efi/redhat/elilo.conf must be set el5xen (xen kernel) option - Xen supports different kernel versions on host and guest, but SAP recommends they should be same - Oracle JDK/JRE home can be used for sapinst GUI, but it is recommended to use Sun JDK - naming conventions differ from Windows significantly; Oracle is using additional OS account ora<sid>, oracle home must is in format /oracle/<SID>/102_64 (this _64 is mandatory), even if you put something else, sapinst refuses to work normally later - after DBCA database creation (if needed): cd $ORACLE_HOME su <rootpassword> ./root.sh exit - starting / stopping SAP: - Oracle: (vi ~/.dbenv_<server_name>.csh and there can be changed SID, ORACLE_HOME, SCHEMA or whatever needed in oraerc's profile, where oraert is the template Oracle user): su oraert lsnrcstl start/stop sqlplus / as sysdba ... - SAP: su ertadm startsap r3 stopsap r3 (using all instead of r3 if Java stack is present, as it is with BI systems) - shutting down: shutdown h 0 (as with other guest or bare metal systems, there is always a risk with forced shutting down of damaging system disk or something more – linux rescue boot with prepared sap-rescue configuration with affected system disk would help, or restore from a backup ...) or init 0 !!!!! NOTE: always, always make backup of everything that is important ... SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 22. HVM GUESTS - It is enough to copy existing template configuration (e.g.. /etc/xen/sap-test) for a new guest, having at least changed name=new_machine (instead of name=sap-test), and also /usr/lib/xen/boot/nvram_sap-test into /usr/lib/xen/boot/nvram_new_machine (this is instead of nvrboot import from the backup on the EFI partition – HVM guests have their own EFI environment, just as any bare metal machine) - for new guest disks created either as snaphots / snapclones or CA replicas of the template disks, check the parted /dev/disk/by-id/... print before proceeding – if any I/O errors occur, try changing in EVA Command View the Presentation / Preferred path/mode on old EVA 5000 (A or B, when upgraded to new A/A VCS4.xx firmware this will no longer be a problem). It recommended to use for each VLUN it's wwid set accordingly in /etc/multipath.conf with appropriate /dev/mapper/alias ... With ntfsls utility one can even check the M$ partition without any mounting - for each new disk (which is not a copy/replication, not having gpt label) initialization must be done: parted /dev/disk/by-id/... mklabel gpt (it is also possible as usual in EFI shell) ... - after Windows boot and administrator login, first the IP address should be set in the GUI console (vnc) and remote desktop enabled (for a newly installed machine, it should be already set in the template) and then continue working with RDP (rdesktop) - the next step is changing the name to the machine and adding it to Winows domain: newsid /a novoime (restart should be done only from virt-manager or with xm create, anything other will fail to bring up network correctly) - machine preparation is the same as with any other Windows machine (Oracle and SAP installation, mind installing Montecito Oracle patches if needed) - SAP ERT 30 instance in the template with preinstalled ERT instance can be used only for the same machine, which comes down to same MAC address, having same database schema and licenses preserved with (otherwise, additional setup is needed): ... on the old system: exp file=lic.dmp userid='/ as sysdba' tables=(SAPERP.SAPLIKEY) ... on the newly copied system: imp file=lic.dmp userid='/ as sysdba' fromuser=SAPERP touser=SAPERP - SAPDATA_HOME and oraarch folders (or whatever set in sapinst for the template) are best to be kept on C: - HVM guests have that unfortunate constraint to have maximum four drives !!!!! NOTE: always, always make backup of everything that is important ... SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 23. KNOWN PROBLEMS I had several problems during during all these tests, and some of them are significant issues: • EFI environment can slow down vnc console and virt-manager refresh terribly for unknown reasons (xm commands are usable, but also a bit slower, while the guest domains and Dom0 work perfectly normal) – a RH service request is open on this one • ORA-07445 occurs intermittently on Dom0s and DomUs (but does not on normal kernel) – a metalink SR is open on this one (filed bug 8868468) - the only viable workarounds are not to use Xen for Oracle, or to somehow prepare database without Xen, and then use it under Xen (this is not causing serious issues, but it would be unacceptable in producion) SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 24. References Note 962334 - Linux: SAP on Xen virtual machine Note 1048303 - Red Hat Enterprise Linux 5.x: Installation and upgrade Note 958253 - SUSE LINUX Enterprise Server 10: Installation notes Note 941735 - SAP memory management for 64-bit Linux systems Note 964705 - Linux IPF: "floating-point assist fault" in Linux syslog Note 784391 - SAP support terms and 3rd-party Linux kernel drivers Note 527843 - Oracle RAC support in the SAP environment Note 592393 - FAQ: Oracle http://en.wikipedia.org/wiki/Xen http://www.linux-kvm.org/page/Small_look_inside http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Virtualization_Guide/index.html http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/task-virt-lab1.html http://www.redhat.com/legal/open_source_assurance_agreement.html http://kbase.redhat.com/faq/docs/DOC-10113 http://kbase.redhat.com/faq/docs/DOC-17364 http://kbase.redhat.com/faq/docs/DOC-23403 http://www.scribd.com/doc/3210098/xen-install-windows http://forums.citrix.com/thread.jspa?threadID=241177 https://www.redhat.com/apps/store/server/ http://www.redhat.com/rhel/virtualization/ presales 1-888-REDHAT1 ext 45031. http://kbase.redhat.com/faq/docs/DOC-23403 http://www.encyclopedia.com/doc/1G1-168820045.html http://www.eweek.com/c/a/IT-Infrastructure/Intel-Allies-Look-to-Linux-for-Itanium-Growth/ http://www.itaniumsolutions.org/resources/red_hat_enterprise_linux_virtualization_and_intel_itani um_linux_open_source_virtualization_with_enterpriseclass_features http://whitepapers.techrepublic.com.com/abstract.aspx?docid=301970 http://blog.xen.org/index.php/2009/11/18/fujitsu-and-va-linux-systems-japan-k-k-join-xen-advisory-board http://www.oracle.com/us/technologies/linux/025994.htm http://community.citrix.com/display/ocb/2008/12/18/The+Oracle+comes+to+Xen http://www.hp.com/go/lxintegritycert http://www.hp.com/go/integrityrhel http://h18000.www1.hp.com/products/servers/linux/redhat/howtobuy.html https://www.redhatrenewals.com/rhel/compare/ http://www.itjungle.com/tlb/tlb021908-story02.html http://www.centos.org/docs/5/html/5.2/Virtualization/chap-Virtualization-Configuring_ELILO.html http://docs.hp.com/en/5992-3193/apcs05.html Oracle VM http://download.oracle.com/docs/cd/E11081_01/doc/doc.21/e10898/createvm.htm http://ia64.koji.fedoraproject.org/fedora-ia64/rawhide-latest/development/ia64/os/Packages/ http://fedoraproject.org/wiki/Architectures/IA64 https://www.scientificlinux.org/news/archive/sl41.ia64 http://www.linux-kvm.com/content/using-windows-installer-paravirtual-network-drivers http://xenbits.xensource.com/ext/efi-vfirmware.hg SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 25. http://xenbits.xensource.com/ext/ia64, http://www.selenic.com/mercurial/wiki/ http://www.selenic.com/mercurial/wiki/index.cgi/UnixInstall http://www.kernel.org/pub/software/scm/git http://www.mail-archive.com/virtualization@lists.linux-foundation.org/msg05115.html http://logout.sh/computers/linux/netboot/ http://www.opensourcehowto.org/how-to/tftp/tftp--pxe--syslinux--fedora-install.html http://www.tpc.org/tpcc/results/tpcc_perf_results.asp http://www.tpc.org/tpcc/results/tpcc_price_perf_results.asp http://ts.fujitsu.com/ps2/press/read/news_details.aspx?id=2202 http://www.oracle.com/corporate/press/1524171.html http://www.nec.co.jp/press/en/0802/2101.html http://en.wikipedia.org/wiki/Rate_of_return http://en.wikipedia.org/wiki/Total_cost_of_ownership benchmarks: SAPS http://www.sap.com/solutions/benchmark/measuring/index.epx http://www.sap.com/solutions/benchmark/index.epx http://service.sap.com/benchmark http://www.sap.com/solutions/benchmark/sd3tier.epx?num=200 http://www.sap.com/solutions/benchmark/sd2tier.epx Solaris http://www.c0t0d0s0.org/archives/4962-SAP-Benchmarks-revisited.html http://www.sapdb.org/7.4/sap_db_benchmarks.htm http://www.redhat.com/f/pdf/Virt_SAP_SD_DB2_Reference_Architecture_V1_01-08-2009.pdf&ei=t3tbSrOSCsjI- QaQy5SBCQ&rct=j&q=SAP+benchmark+SAPS&usg=AFQjCNHLcq9bYTOPSXUPGzXDmtQIBwD6xA http://download.sap.com/download.epd? context=40E2D9D5E00EEF7C6D06E4404E078BAD5C01554ECD2027EFC759D35F83100EEE&ei=t3tbSrOSCsjI- QaQy5SBCQ&rct=j&q=SAP+benchmark+SAPS&usg=AFQjCNGNyIZqCbzYY5-HsKxeXCY8I9yK-A http://www.oracle.com/corporate/pricing/pricelists.html http://docs.hp.com/en/T2767-90141/index.html http://forums11.itrc.hp.com/service/forums/questionanswer.do? admit=109447626+1247514013348+28353475&threadId=1302823 http://h71028.www7.hp.com/enterprise/cache/258348-0-0-0-121.html adaptive infrastructure http://h71028.www7.hp.com/enterprise/cache/483409-0-0-0-121.html licensing http://h71028.www7.hp.com/enterprise/cache/262372-0-0-0-121.html tco http://h20338.www2.hp.com/hpux11i/cache/324574-0-0-0-121.html HPUX vs. linux http://h20338.www2.hp.com/hpux11i/cache/615059-0-0-0-121.html http://www.microsoft.com/windowsserver2008/en/us/hyperv-faq.aspx http://www.microsoft.com/presspass/press/2007/nov07/11-12HyperVPR.mspx http://www.microsoft.com/windowsserver2008/en/us/licensing-itanium.asp OpenVZ vs, Xen performance report: http://slashdot.org/article.pl?sid=07/05/15/019251 www.parallels.com/virtuozzo, hp osip/osms (open source), http://www.parallels.com/solutions/vdi/ http://www.parallels.com/partnernet/technology/company,11925 http://www.parallels.com/solutions/virtualization/itanium OpenVZ: http://wiki.openvz.org/Main_Page http://wiki.openvz.org/Download http://download.openvz.org/openvz.repo http://download.openvz.org/current/ http://service.sap.com/solutionmanager http://www.solutions2projects.com/pdfs/Good_Business_Practices_October%202006.pdf SAP, Linux, Virtualization and ... Itanium Popović Zoran
- 26. http://h10076.www1.hp.com/education/curr-itsm.htm http://technet.microsoft.com/en-us/library/cc305136(TechNet.10).aspx http://www.itil.org.uk/ http://www.itil-officialsite.com/home/home.asp http://www.itil.co.uk/ http://en.wikipedia.org/wiki/Information_Technology_Infrastructure_Library http://itsm.the-hamster.com/ SAP, Linux, Virtualization and ... Itanium Popović Zoran