Please note that as of version 4.1 the System Configuration Wizard in the CMC now automates the process of splitting the adaptive processing servers. The following assumes you are manually splitting the APS services in SAP BusinessObjects 4.0. You can also use this knowledge to manually split the APS services in 4.1 but 4.1 specifics are not mentioned below.
With the release of SAP BusinessObjects 4.0 many of the new and even existing services or applications within the platform were grouped into a single process called the “Adaptive Processing Server” (APS). From the perspective of the operating system, the APS runs as a single PID (Process ID). From the perspective of SAP BusinessObjects, this single service is a container for 20+ different processes (Sub-Services) within the BusinessObjects 4.0 platform. The APS is a pure Java based process and it is initialized with 1 GB of Java Heap (-Xmx1g). For most environments this is not a sufficient amount of RAM. In addition it was not wise for SAP to have placed such a high number of sub-processes in a single PID. Because the default SAP BusinessObjects 4.0 deployment essentially places all the eggs in a single basket with very little Java memory bandwidth, a considerable amount of re-configuration has to be implemented, post installation, to properly size and deploy these services. There have been multiple blog postings and SAP notes created in relation to this topic. Most of which I will reference in this blog posting. The main goal of this article is to simplify the reasons that splitting these services is needed,to aggregate all the information that is currently available online and to provide a few thoughts on how you can approach the splitting of the services.
Why Split?
Splitting the Adaptive Processing Server is needed for the following reasons:
- There are multiple disparate sub-processes running in the default APS container. In some cases a single sub-process can exceed the default Java memory parameters and crash the entire stack of services. In short, it is very risky (from a high availability perspective) to run everything in a single PID. If one sub-service fails, they all fail. To prevent a single task from crashing the entire stack, the sub-processes should be split into different APS containers in a way that each container has its own OS level PID (Process ID).
- Each sub-service will have its own unique Java memory requirements depending on the amount of data and the kind of data source. The only way to give each sub-service a unique or custom Java memory parameter is to run it as is own service or PID.
- Depending on the hardware specifications, it might be necessary to run some of the sub-services on their own node. They will have to be split out in order for them to be horizontally deployed.
- If there is a need to stop a single sub-service (without stopping the entire stack) the sub-service must exist in its own PID. There are also other deployment scenarios where some sub-services must be stopped and disabled on one node and enabled and started on another. If they are not split out, this is not possible.
How to allocate?
How you choose to split and group sub-services together into individual containers or groups of containers is almost subjective. The APS sizing guides from SAP will give you a general initial RAM amount for each service and recommendations for deployment. If each services is split into its own container and run on the same node, that node would need over 26 GB of RAM + several more GB assuming other BOBJ services are running on the node as well. This might be a bit extreme if 26+ GB of RAM is hard to obtain for a single node. However, as the document states, you don’t always have to deploy everything individually. When I go to a customer’s site, I general ask them which products they plan to use. Based on that list I try to anticipate the size and complexity of the reports to determine how the APS needs to be split to manage the system. For some clients I end up with 8 distinct containers with others I might have just 10. It all depends on whether they plan to utilize individual tools associated with any given APS sub-service. In general, I group the processes together according to the expected utilization of the platform and similarities of the services. I have established a standard base line that I deploy at each customer site and then I make a few changes based on the expected utilization. I will also run each grouping on an appropriate SIA node if the environment is setup as a distributed cluster. The following link will give you an example of my standard base line Link to Document. After reviewing the linked document above, the following will list the most likely candidates for change based on my experience.
- APS_DATAFEDERATOR: If the customer plan to utilize multi-source UNX universes, the Java Heap (-Xmx) will need to be anywhere from 4GB to 10GB+ depending on the size of the data processed by DF and the complexity of cross database joins.
- APS_MDAS: If Analysis for OLAP is utilized and the datasets are large, this might need to go up to the 4GB+ range as well. In addition, if I expect heavy utilization of the MDAS and the use of STS, I will generally move STS into its own container. It’s not that I expect STS to use a lot of RAM, but if MDAS crashes, I don’t want the STS to go down with it.
- APS_PLATFORMSEARCH: If you are running a continuous crawl and have a lot of reports and universes, you might need to increase this services heap as high as 4GB.
- APS_WEBI: The most likely candidate for change in APS_WEBI would be the isolation of the DSL Bridge Service and the Visualization Service. The DSL bridge services can take a big hit with Direct Binding to BEx reports. The visualization services can take a big hit if the Web Intelligence charts contain a lot of data points. The max heap setting for the DSL Bridge service might need to go as high as 32GB+. The memory allocation all depends on the amount of data that is extracted from BW via a BEx (BICS) query. I have also seen a lot of bug induced crashes for both the DSL Bridge and Visualization services. This is just yet another reason to isolate these services.
The memory settings and spiting up of the remaining services can be approached as needed. In general I try not to over engineer the break out. Doing so requires a lot of RAM on the server that might never get utilized. As I group more services together, I lower the overall RAM requirements for the system. In addition, just because the HEAP is set to a high number of GBs for each service does not mean that the service will utilize all of the allocation. You can always take a chance and run 20 GB of services (Max Heap) on a system with only 16 GB of RAM. To be safe you should always have sufficient RAM to support the sum total of all running Java processes Heap, but risk takers might take a more participle approach. In some cases I might just leave a few sub-services in the original APS container with a 1GB Java Heap because I know they will never get utilized.
Should I clone or create new?
I generally create each server from scratch. I have noticed that cloning a server will carry over command line arguments from sub-processes that are unrelated to the actual sub-services that will eventually be running in the container. This does not necessarily create any issues, but I prefer the command line to be as clean as possible. After I create the new service from scratch, I update the following command line arguments:
- The Java Heap Max Size (-Xmx1g): This is the total allocation of RAM that the JVM is allowed to reserve from the OS. If the Java process needs more than the amount specified, it will likely crash. I generally copy the APS service command line into a text editor, find the argument, make the changes, copy the text and paste it back into the command line. Be careful. Any mistakes will prevent the service from starting. Use the guides linked in the document to determine the setting size.
- The Java work directory (-work ): This is the path to where the work directory will exists. It is very important that the directory is not utilized by another APS services. I generally add the service name to the end of the path. For example: I will add “APS_PUBLISH” to the end of the work directory path. -workdir …/java/pjs/container/work/APS_PUBLISH . This insures that no other APS service uses the same path. BOBJ will try to mitigate this in the event that you forget. However the BOBJ method is to simply add the word “AdaptiveProcessingServer1” to the end of the work path. For each additional service it adds an additional number. This does not always work if you are adding and removing APS services frequently to the node. In addition, it is much easier to find the work directory, for a given process, when the path contains the name of the server.
Links to other blogs
Dallas Marks started the conversation on the APS several months ago. I added a few comments to his blog when BOBJ 4.0 was first released. This is a really good starting point.
http://www.dallasmarks.org/blog/2011/11/sizing-the-adaptive-processing-server-in-bi-4-0/
Raphael Branger mentions an issue with the Visualization Service
http://rbranger.wordpress.com/2012/08/29/webi-4-0-charts-and-adaptive-processing-server/
Links to SAP document:
Official SAP APS Sizing Guide: sdn.sap.com
APS Sizing SAP Note: https://service.sap.com/sap/support/notes/1694041
Great post. Saving this for an upcoming BO4 implementation, thank you very much.
Thanks for sharing this Jonathan.. Very useful. I have one question.. In the clustered servers i am assuming it is good to split the APS on each individual node, if you distribute the services among the two nodes then failure of 1 node might impact the system availabilty right ?
I have 2 nodes with 16 GB each so trying to come up with best possible approach.
Raj,
Most of the APS services can run Active / Active. In short, you can deploy them on 2 or more nodes to provide both load balancing and fault tolerance. However, there are a few that cannot run Active / Active or Active / Passive. The APS_LCM and the APS_MONITORING services can only run on a single node at any given time due to the derby db they use to store data or subversion. For these services there are options to use manual fail-over or external active / passive clustering services to manage fail-over for these services. Also, the monitoring data can be moved to a RDMS but SAP has not released a note to indicate if using a RDMS for the monitoring data then provides Active / Active clustering capabilities.
Thanks,
Hi Jonathan, quick question.. I see in SAP note and also in your document about “TraceLog Service” but i cannot find that service in the list of services under APS.. We are on BO 4.0 SP5 patch4.. Where can i find the tracelog service..
Raj,
The “TraceLog” service (as of SP4) is now automatically added to each APS container. While you can not add or remove it, it is still there.
Thanks,
Thanks Jonathan.. with single APS once the concurrent users reach 85 to 100 users.. APS is crashing, now i am splitting the APS and will do another load test..
YouTube Video: Splitting BI 4.0 Adaptive Processing Servers
Dear SAP Colleagues, Partners, and Customers:
I posted an instructional video to SAP’s YouTube channel. My video explains how to split the Adaptive Processing Server (APS) to improve scalability and performance of SAP BI Platform 4.0—especially if you’re reporting on data in BW or HANA:
Tasks demonstrated in the video:
1. Intelligently splitting the APS into 7 instances
2. Setting an appropriate Java heap setting for each APS instance
The target audience is BI 4.0 administrators and consultants. Please feel free to share this email to help promote important tuning practices throughout our BI ecosystem.
Related knowledge resources:
1. APS Best Practices Whitepaper. With over 9,000 views, the APS best practices article is a must-read for customers and partners implementing BI 4.0. (My video covers a simplified version of the techniques discussed in the whitepaper.)
http://scn.sap.com/docs/DOC-31711
2. SAP BI 4.0 Ecosystem how-to videos.
http://scn.sap.com/docs/DOC-34251
3. More knowledge sharing. In our SCN upgrade page (http://scn.sap.com/docs/DOC-25474) and related webcasts (https://twitter.com/tpowlas/status/307549667170869248) we recommend customers to update their implementation skills for BI4. ASUG webinar here (login required: http://www.asug.com/events/detail/SAP-BusinessObjects-BI4-Platform-Upgrade-To-BI4)
Hi Jonathan, Just wanted to check with you about the performance in BI 4.0, we are on 4.0 SP5 patch 4 and i see that CMS is taking lot of CPU and the servers are running close to 100%.. I did split the APS and we have 2 nodes. Did you notice this behavior ? I migrated 3.1 content last weekend and after that i see this behavior.. we have 2 nodes with 4 CPU each and 24 GB memory..
I’m running SP5 P4 at a few sites. No issues like yours. Is it running on a Virtual Machine? Also how many InfoObjects do you have in your repo. Any heavily nested groups or users with 100 + Group memberships.
Yes, all our servers are VM’s except Database Servers, its a company policy to use VM’s, tried to get physical machines but dint work. We have around 300K info objects and we use SAP authentication and we have different roles/groups and around 63,000 users in the system.. less than 1000 active users, less than 150 concurrent users.
Its hard to assess the situation over the internet but I would suggest that you have your VMWare ESX admin make sure your not on an overly subscribed guest. In addition, it sounds like you might be a bit under-sized. I generally suggest a minimum of 8 CPUs per node (With some exceptions). 300K infoobjects is quite large and you might need more CMS instances and a larger CMS cache to manage the environment with that size. Again, I can’t make a honest assessment over the internet but I thought I would give you a few ideas.
Note: I generally advise against using VM’s in production unless you properly spread the services across multiple nodes and give the guests the proper sizes. At a minimum I recommend 3 nodes on VMWARE (8vCPU X 20+ GB RAM). At a minimum 6 nodes if you wan’t clustering.
Hi Jonathan,
Very good information from you posts. I have a question regarding your splitting the APS server post, does it matter which web application you are using when you are doing the splitting (Tomcat v/s Websphere). Most of the articles talk about tomcat, I was wondering if all the same logic and sequence applies when you are using Websphere.
Will appreciate your comments.
Thanks,
KM
In short, the APS services have no dependences with the Tomcat instance running the BOBJ Web Application. They are different services all together. You can run any supported Java Application Server to host the BOBJ web apps, regardless of how you size or split the APS.
sounds great!! i followed your steps. it’s working fine. Thank you Jonathan.
Hi Jonathan, Just a comment on your update “The following only applies to SAP BusinessObjects 4.0”. The System Configuration Wizard in BI 4.1 does perform APS splitting but in a very formulaic way and without the context of specific customer environments. I think it’s still important for architects and admins to understand your article even for 4.1, adjust anything the wizard spits out and not necessarily rely on it’s raw output as the final word. Cheers, Ray.
I don’t disagree. My note was only a warning that this article was written in the context of 4.0
I am a big fan of your Jon.
Recently i came across a customer who uses Crystal (80%) of their total number of objects. The reports are direct on top of DB. What kind of optimal configuration i should suggest for them? Why Crystal APS not available? Is it because it’s .Net?
They are completely out of SAP ECC/BW world, should i still create a DSLBridge APS for them?
Please advise.