Clustering the LCM and VMS in SAP BusinessObjects 4.0 SP4


Clustering the LCM and VMS in SAP BusinessObjects 4.0 SP4

With the release of SAP BusinessObjects Enterprise 4.0 (BOE 4) and subsequently with the release of Service Pack 4 (Including Feature Pack 3) administrators will find that both the Life Cycle Manager (Promotion Management or LCM) and the Version Management tools are included with the installation. In XI 3.1, when these tools were first released, a separate installation was required and it was considered an add-on to the platform. With BOE 4.0 they are included with the main server installation media. As of BOE 4.0 SP4 (with FP3) these tools are directly integrate into the Central Management Console (CMC). For single server (Full installation) deployments there is very little you need to know about the interworking of these tools. However, if you plan to deploy BOE 4.0 SP4 as a distributed or clustered environment there are a few tricks to successfully deploying LCM in a clustered environment. This document will serve as a means to explain the interworking of the LCM within a cluster and the steps to deploy LCM as an active / passive service.

Before we discuss the steps of procedures for clustering, let’s break the LCM down into its basic components.

The LCM Web Applications:

There are two main tools that encompass the LCM and VMS tools that are managed at the Java Web Application Server (JWAS) level. In SP4 you can access them via the CMC under “Promotion Management” and “Version Management”. There are also a few settings for these applications found within the “Applications” area of the CMC. The two main settings are for the SVN services (found under Version Management) and the overrides database (found under Promotion Management). Because these tools are a component of the CMC and run under the JWAS (Tomcat in most cases) they can be deployed on one or more JWAS (Tomcat for example) servers within a cluster. SAP does not provide a solution for clustering Web Servers out of the box, but you will like find that your network team already possesses a tool to help you cluster Web Servers. Typically organizations will utilize an IP Load Balancer (with sticks sessions) to cluster web servers but there are other options as well.

Subversion:

“Subversion” is a software versioning and revision control system distributed under an open source license. Developers use Subversion to maintain current and historical versions of files such as source code, web pages, and documentation. Its goal is to be a mostly-compatible successor to the widely used Concurrent Versions System (CVS).“ Reference . The LCM and VMS applications within BOE 4.0 will utilize this service to maintain snapshots or rollback instances of BOE content. Subversion is installed automatically with the main BOE 4.0 installation media as a 3rd party component. It will run on any node within your cluster and typically listens on port 3690. The service itself manages a file based repository. The default repository name is LCM_repository and its data will be stored on the local file system where Subversion is installed. When deploying a clustered environment it is best to only deploy subversion on one node. This will make it easier to migrate the repository when we start to cluster the services. The main executable for the SVN service is located in the subversion\bin directory found under the BOE 4.0 Windows folder “Program Files (x86)\ SAP BusinessObjects\”  or UNIX folder “/sap_bobj/enterprise_xi40/”. The LCM_repository folder will contain a file based database that tracks the version, location and other information about files that are being managed by the LCM and VMS tools.

The Check_out Directory:

The check_out directory is another file based repository or database that maintains an actual copy of each BOE 4.0 object that is either promoted or versioned using the LCM or VMS tools. If you browse the directory you will find subfolders named after each objects CUID. Within each of these subfolders you will find a small file based db and a series of .bair files that represent each object or object promotion. When establishing a cluster this directory must be moved to a central NAS share utilizing either SMB or NFS. Later we will configure BOE to look for these files under a shared UNC path or shared
NFS mount.

The LCM Overrides Directory:

The LCM Overrides directory is new in BOE 4.0. It is another apache derby file based database that is utilized to track the translation of connection, QaaWS and Crystal Reports connections from one system to another system. Again, when we cluster the LCM / VMS services we have to also migrate this database to a central shared file system (NFS or SMB).

The LCM Adaptive Processing Services:

The LCM Adaptive Processing Services consists of three components:

  1. Lifecycle Management ClearCase Service
  2. Lifecycle Management Service
  3. Visual Difference Service

In a properly configured environment you will undergo an exercise to split the default AdaptiveProcessingServer into multiple dedicated Java services. When doing so it is important to create a dedicated APS_LCM service that contains the above mentioned LCM sub-services. This will be important for LCM clustering as we will need to stop and start these services as a group on each node. The Lifecycle Management Service is very important to the management of the LCM / VMS processes. It is responsible for communicating directly to SVN repository, the Check_out and Overrides repositories.

Clustering the LCM (Overview):

Clustering the LCM is not a straight forward process so pay close attention to the steps and requirements. In short, because the LCM relies on file based databases or repositories there are restrictions to having multiple active services managing these files. File locking issues can be a real problem when two services are actively accessing the same file. As a result we have to run the LCM clustering in a true active / passive (manual passive) fashion. This means that both the subversion service and the LCM APS services can be running on one and only one node at any given time. It also means that they must be running together on the same node at any given time. The other main requirement is that we move the three main file based db repositories to a central shared file system. On Windows this can be an SMB UNC path or on Linux this can be a shared NFS mount.

Clustering the LCM (Steps):

Step 1: With the VMS and LCM working on Node 1, go tothe CMC, Version Management tool and add 1 or more reports to version management. This will help with subsequent testing as objects will be added to the repository.

Step 2: Stop both the LCM_APS (Adaptive Processing LCM Services) and the Subversion service on all nodes.

Step 3: Insure that the subversion services were installed on all nodes where it can potentially run. If the services were not installed you can add it again by re-running the BOE install media. For linux environments you can create two svnstartup.sh and svnshutdown.sh scripts to make the process of stop/starting easier (See SAP NOTE: 1755733 for Example). On Windows you will find the Subversion services listed under the standard Windows Services MMC. On Windows you can also create a .bat file to start / stop the services if you so desire.

Step 4: Copy the “Check_out”, “Overrides” and “LCM_repository” directories to a central shared file system. I recommend removing / deleting the original directories once you have everything working. I placed all the directories in a single path. For example:

Windows:\\servername\sharename\lcm

Linux: “\apps\bobj40\shared\lcm\”

Step 5: Update the SVN root path and then start the Subversion service on Node 1. Using information found on the http://subversion.tigris.org/ site, insure that the SVN services recognize that the LCM_repository now resides on the shared file path. In my environments, I simply update the SVN startup to include the shared file system as the new SVN root. If the repository directory was copied correctly and the SVN services are started with the correct root path, the LCM_repository should be recognized automatically.

Unix: “./svnserve –root=\apps\bobj40\shared\lcm\ –port “3690”

Note: Do not include the LCM_Repository folder in the root path. On UNIX the patch and repository name are different then on Windows.

Windows:

  1. Using Regedit (with Caution) Go to [HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\SVNSubversion]
  2. On the ImagePath String REG_EXPAND_SZ, update the path following the –r command.
  3. Change -r “<Install Path>\SAP BusinessObjects Enterprise XI 4.0\LCM_repository” to -r \\server\Share\lcm\LCM_repository . Note: Include the LCM_repository directory in the above path. On Windows the SVN repository is in a sub folder named svn_repository.
  4. Go To the CMC, Applications, Version Management, VMS Settings and make sure that the repository name field is set to  svn_repository

Step 6: On the remaining nodes (that will also potentially run the SVN services) update the subversion startup scripts or registry to make sure that they also will use the new SVN root on the shared drive when started. Stop and start the SVN services on each node (one at a time) and verify that the services are working and pointed to the correct SVN root directory. When the following command is executed on each node, you should see a list of folders matching the CUID of the objects that were checked in during step 1. You should be able run the following command on each server to verify.

Command:

  • (UNIX):    svn list svn://localhost/LCM_repository
  • (Windows) svn list svn://localhost/svn_repository

Note: The SVN can be found under the SAP BusinessObjects Installation directory. The paths below might be different in your environment:

Windows: <Install Directory>\SAP BusinessObjects Enterprise XI 4.0\subversion

Linux: <Install Directory>/sap_bobj/enterprise_xi40/subversion/bin

Step 7: Update the version management settings by going to the CMC, Application, Version Management and Version Management Setting Page. Set the options as follows:

  1. Server Name:    localhost <-default (must be localhost for this solution to work)
  2. Server Port:        3690 <-default
  3. User Name:        lcm <-default
  4. User Password: <do not change>
  5. Install Path: <Path to the “subversion\bin” directory> <-default
  6. Repository Name:           LCM_repository  <-default Unix   svn_repository <- default Windows
  7. Workspace Directory: <change to the new shared directory path of the “Check_out” directory.
  8. Protocol: SVN

Note: localhost is required. When objects are added the the check_out directory, the repository records the host name of the subversion service’s host. If these settings list an individual host, the fail over will fail because a different host name will be recognized.

Step 8: Update the “Overrides Directory Path on the APS_LCM services. Go to CMC, Applications, Promotion Management, LCM Override Settings. For each APS_LCM services configure the path for the overrides database to the shared directory location of the “overrides” folder.

Step 9: On one and only one node, start the Subversion and APS_LCM services. Insure that these services are stopped and disabled on all remaining nodes.

Testing the LCM failover:

At this stage you can either test the LCM failover or conduct an actual failover by stopping both the Subversion and LCM_APPS services on NODE1 and subsequently starting both the Subversion (first) and LCM_APS (Second) services on NODE2. The key is making sure that both of the services are running on the same NODE and the same time. You also need to make sure that these services are disabled and stopped on all remaining nodes.

Hopefully this will give you more insight into the process of clustering the LCM services and databases. Each environment will be different so it is better to first understand the processes and dependencies and then tailor this configuration to your environment.

27 comments

  1. I was working on LCM issues with version mgmt yesterday and it appears the ClearCase service is needed to support check-in / out process even when you are using Subversion. Crazy name for the APS. Thought I would mention that even though it seems like you could leave out the ClearCase service from the APS – LCM service you need it.

  2. Bob,

    Valid point and good catch. Many of the services do more than the documentation sometimes lists. I typically include the clear case service with the APS_LCM so I had not noticed.

    Thanks,

  3. Hi Jonathan! I hope all is well! I tried this on the linux environment, somehow unable to launch the Version Management in the CMC.. Please advise if possible?

  4. AJ,

    Can you send me the Java Stack error (if any)?
    Does the VMS page in the CMC fail to load or does it simply fail to work?
    Are you running both the LCM APS services and the subversion service on the same node?
    Did you stop the LCM APS and VMS services on the second node (Node not being utilized)?

    Thanks,

  5. Here is the error: when I launch the version management section in the CMC:

    com.crystaldecisions.sdk.exception.SDKException$OCAFramework: Unable to find servers in CMS —-:6400 and cluster @—- with kind pjs and service LCMServiceOCA. All such servers could be down or disabled by the administrator. (FWM 01014) cause:com.crystaldecisions.enterprise.ocaframework.OCAFrameworkException$AllServicesDown: Unable to find servers in CMS —-:6400 and cluster @—- with kind pjs and service LCMServiceOCA. All such servers could be down or disabled by the administrator. (FWM 01014) detail:Unable to find servers in CMS —-:6400 and cluster @—- with kind pjs and service LCMServiceOCA. All such servers could be down or disabled by the administrator. (FWM 01014)

    Yes, the svn and LCM services are running on the same note.

    Yes, I did shutdown the svn and LCM services on the second node for now, to make sure it works first in place.. also I did bounce the services couple of times after creating the LCM_reposiroty but no luck.. 😦

  6. Based on the error it appears that no LCM services are started and/or enabled in the cluster. I typically create a dedicated APS service to manage the LCM processes on each node. At a minimum you need the two LifeCycle Management services within this APS container but I generally include the Visual Difference Services as it pertains to the same or similar tasks. If you have them setup properly and it is still not working, you can check the error logs for the APS service to find any possible Java exceptions with the APS container. ../sap_bobj/logging/ . Keep in mind that an APS services can be listed as started and enabled even if a .jar file for one it’s sub-processes failed to load. It’s also important that each APS must be configured with a distinct Java work directory (on each node). That is set on the service’s command line within the CMC servers page (at the end of the cmd line). I typical add the service name right after the /work path in the command line to make it unique. BOBJ will do this as well but it uses the name AdaptiveProcessingServer1 (or 1+). This name might not be unique if you have added and subsequently removed services multiple times.

    Typical deployment of the APS LCM services.

    Service Name:
    APS_LCM

    Contains sub-services:
    Lifecycle Management ClearCase Service
    Lifecycle Management Service
    TraceLog Service
    Visual Difference Service

  7. Jonathan,
    Have you heard of cases with the BI LaunchPad 4.0 “Search functionality” breaking or becoming completely un-usable? We are able to schedule index “crawls” but the Search box throws errors. Any ideas?

    Thanks,
    Phil

  8. Phil,

    I see this all the time. There are multiple reasons this could occur ranging from a corrupt index, lack of Java Heap memory on the APS SearchService, to locked records in the CMS database. If you search the SAP notes or call support they can generally walk you through the processes to repair the service.

  9. Thanks, Johnathan,
    With our BI 4.0 SP4 Patch 1 environment, SAP has informed us that this is a current ADAPT 0151015234 issue. Sounds like they want us to wait until 4.1. Error code is: “An error occurred while searching: Index is not ready. Please try again later (PSS 01004). (PSS 01004)”. What version have you seen the issues with?

    Thanks,
    Phil

  10. 4.0 SP2 Patch 18+ and 4.0 SP4 Patch 1 through 4.

    4.1 will not be out for some time but I’m not surprised support gave up on you so quickly. Their job is to close all open tickets because of some ridiculous open case duration KPI. It would be nice if support were judged on customer satisfaction and not speed of service.

    Did they not instruct you to limit the content that you index and to disable crawl? I have always been able to fix this issue using at least one of the many different recommendations from the SAP notes.

  11. Yes we are limiting to Metadata search only using a non-continuous crawl. They wanted us to try out this SAP Note 1731644, which is wiping out the search data directories on the drive and rebuilding from scratch.

  12. Yes, we split out each APS equally among 3 BOE clusters and beefed up the memory XMX to 3 Gigs. We named them APS_SEARCH.

  13. I am assuming that you updated the CMC / Applications / Platform Search Application Index Location Paths so that all three servers will store their index data on the same UNC Path (Windows Share) or NFS mount (Linux Share) ?

  14. Hi Jonathan. The scheduled Search indexing ran for 12 hours and threw an error when using the (Windows Share) folder directory instead of the “out of the box” configuration: ({bobj.enterprise.home}/data/PlatformSearchData). My setting is \\server\d$\businessobjects\…\data\.

    Is this the same syntax you have used in the past?

  15. For one, I normally use a non administrative share. In order to use any share the SIA service account must have full access to the path specified. By default the SIA runs under the local service account which would not have access to an admin share (d$) or other NTFS secured shares. Is you SIA running under a service account with both share and file level access to the UNC path specified?

  16. Yes the SIA is running using a service acct with both share and file level access to this UNC from each cluster.

  17. Hi, Jonathan,
    We are also facing a similar issue related to Platform Search in out 2 node BO 4.0 SP5 cluster.Master Index Location is pointing to a common share path with both share & file level access. The Scheduler run fine but search through the error message the that search is not ready.
    Appreciate if you could help.

    Regards,
    Khan

  18. Hi Jonathon, thanks for the post. I’ve followed the steps successfully in one environment but I’m having difficulties in another which I hope you can help me with. The problem environment is clustered SP4 Patch 4 using Tomcat and Windows (as was the successful one). I’ve made all the changes but when I try to test versioning SVN errors pop up in the Version Management console that I am currently working through. The errors are difficult to record as they only appear for a second or so but the one that appears to be the root cause mentions that two UUIDs are not in sync as expected. Subsequent to this I get errors saying that objects have already been versioned but this is not reflected in the management console. In the background it looks like the versioning process gets to a certain point and then stops – folders with CUIDs are created in the checkout folder but they do not appear in the list returned by svn list svn://localhost/svn_repository (nothing gets returned). Based on messages in SVN forums around UUIDs I assume that I made a mistake when I copied the repository to the shared location and somehow ended up with a repository with an incorrect UUID but I am at a bit of a loss as the best way to rectify the situation – if you have any advice it would be appreciated!

    Thanks,
    Colin

  19. All sorted – I deleted all the folders under the CheckOut folder (the working copy I think) and everything was fine 🙂

  20. Can we have an acces to the LCM(subversion) outside BO4 In case we use subversion in our it for different application (not BO) could we have on global subversion?

  21. Hi,

    I can move other objects i.e., webi and Crystal, except Excel. I searched SAP support portal and unable to find anything related. It occurred a few times already. The workaround is to clean up WACS tempfiles and restart it. Error appeared is “Failure: Application not available”.

    I already have SPLIT APS with LCM Services. Also I have dedicated LCM Server as a best practice. I am using BO BI 4.0 SP6 Patch 3 on Windows Server 2008 R2. No other Web Application Server is availabe besides WACS.

    Your help would be appreciated.

    Thanks,
    Micky

  22. Hi Jonathan,

    We have SAP BOB XI 4.0, our problem is Root Folder “Folders” are missing in CMC, Can you help us on this?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s