How To Change Java Heap Size In R12 To Avoid Java.Lang.OutOfMemoryError: Java Heap Space Error?

Recently we had encountered a situation, wherein end users where not able to get the R12 login page and
it just hangs, but all the opmn processess were up and running – adopmnctl.sh status gives status as
alive for all the components viz., HTTP, oafm, oacore & forms. The same environment was available and
end users were able to access without any isuses 15 minutes before. This error happens only when
multiple people login to the same page and perform simillar activity like employee self service form
etc. So where and what exactly could be the problem.

This is how we approached and resolved the issue.

First we checked apache error log. The following error was reported in the error log.

==============================================================
[warn] [client IP Address ] oc4j_socket_recvfull timed out
[error] [client IP Address ] mod_oc4j: There is no oc4j process (for destination: application://oacore) available to service request.
[error] [client  IP Address] mod_oc4j: request to OC4J [servername:OC4J AJP Port Range for Oacore]
failed: Connect failed
==============================================================

Above error message does give us a hint that the problem is with oacore, but as i said earlier the oacore is alive according to opmn.

Next, we verified the oacore log file.

($INST_TOP/logs/ora/10.1.3/opmn/oacore_default_group_1/oacorestd.err)

Error Message in the file

=======================================================
Exception in thread “OC4JMonitorThread” java.lang.OutOfMemoryError: Java heap space
=======================================================

While checking the error in file, parallely we opened another window to see the CPU and memory usage and we could see one java process was taking more than 100% CPU. This process was spawned by the opmn -d process and the process id didnt match with the oacore process id. Hint: adopmnctl.sh status will give the status as well the process id. It looked like a end client java process.

At this stage we had 3 options.

1. Kill the java process which is consuming high cpu.
2. Bounce oacore component
3. Adjust java (jvm) memory parameters

Each of the above options has its own implications and advantages. To minimize the downtime we decided  to kill the java processes, and the moment we killed the java process, all the browsers which were hanging reported instantaneously Internal Server Error. This proved to be a bad decision.

So we moved to the next option, bouncing the oacore service which we are sure will resolve the issue  (temporarily) and it did as expected. Basically when you bounce the services all the existing
connections and processess will be released which results in more available memory when you re-start the services.

Ok, now we tackled the problem and provided a temporary solution but we need to find a long term solution. This is option 3, adjusting java memory size.

Steps to change the heap size.

First, you need to identify how much is the maximum heap size that you can set. Click here.

Once you had identified the maximum heap size, we need to change the configuration files to make it
permanent.

Step 1: Edit opmn.xml file.

Location :$INST_TOP/ora/10.1.3/opmn/conf/

Open opmn.xml

Search for string Xms or Xmx or module-id=”OC4J”

This search should lead you to below location

‘<’process-type id=”oacore” module-id=”OC4J” status=”enabled” working-dir=”$ORACLE_HOME/j2ee/home”‘>’
‘<’category id=”start-parameters”‘>’
‘<’data id=”java-options” value=”-server -verbose:gc -Xmx512M -Xms128M ……]

The default value for Maximum (-Xmx) and Minimum (-Xms) heap sizes are 512M and 128M respectively.

Again here you have options, you can set both Xms and Xmx has the same value as Xmx if you feel all your sessoins require higher memory or set a lower value for Xms and the maximum value for Xmx. Dont forget to change the values under ‘<’category id=”stop-parameters”‘>’

opmn.xml also contains jvm configurations for other components – oafm & forms.

Step 2: Edit oc4j.properties file.

Location :($INST_TOP/ora/10.1.3/j2ee/oacore/config)

This step is optional since we had already made changes in opmn.xml but there is no harm in making the change here. This step will come handy for troubleshooting specific components of Oracle viz., configurator, iSupplier or any other option which heavily utilizes/consumes CPU/memory.

Search for string Xms or Xmx or wrapper.

Option 1: If you find any of the above parameters change the values corresponding to the value you had  mentioned in opmn.xml or even more than that, as long as you dont exceed the maximum heap size limit.

Option 2: If you DO NOT find any of the above parameters, then make an entry like this, under the heading “# Java Object Cache Configuration Parameters”

wrapper.bin.parameters=-Xms[Value]M -Xmx[Value]M -XX:NewSize=256M -XX:MaxNewSize=256M

Step 3: Edit Applications Context file

vi $CONTEXT_FILE

Location:$INST_TOP/appl/admin/SID_hostname.xml

search for string s_oacore_jvm_start_options

Change Xms and Xmx value. Repeat the same step for parameter s_oacore_jvm_stop_options.

Changes made in Step 3 will take effect the next time you run autoconfig, whereas Step 1 & 2 changes will take effect the next time you bounce opmn services, but the values are not permanent in the sense these will be wiped off next time you run autoconfig. Yes you can preserve the changes by placing it inbetween Begin and End customizations.

You can also increase the Garbage Collection threads parameter (-XX:ParallelGCThreads) to a higher value if you have JDK 1.5 or more than 2 cpus or more memory. For more information on this you can refer to Metalink Note: 362851 – Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12.

http://oracamp.com/how-change-java-heap-size-r12-avoid-javalangoutofmemoryerror-java-heap-space-error

libXtst.so.6 cannot open shared object file error during Oracle Installation

The "libXtst.so.6 cannot open shared object file" is quite common if you install any oracle product on 64bit system and misses something in the system 
prerequisites. A quick resolution is installing the  libXtst package for both 32 bit and 64 bit. But, last time while installing OID on 64bit Linux, I found that though libXtst package is already there, still this error is coming. See the error below and its solution.


The following error is coming during starting of installer.

$ ./runInstaller -ignoreSysPrereqs

-------------------------------------------------------------------------------------------------------------------
The OUI Screen may take around 5 to 30 seconds to come up depending upon system performance. Please Wait .......
-------------------------------------------------------------------------------------------------------------------

Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be redhat-Red Hat Enterprise Linux AS release 4, redhat-2.1, redhat-3, SuSE-9 or UnitedLinux-1.0
                                      Failed <<<<


>>> Ignoring required pre-requisite failures. Continuing...

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-02-11_01-46-40PM. Please wait ...[xxxxxx@Disk1]$ Oracle Universal Installer, Version 10.1.0.5.0 Production
Copyright (C) 1999, 2006, Oracle. All rights reserved.

Exception java.lang.UnsatisfiedLinkError: /tmp/OraInstall2013-02-11_01-46-40PM/jre/1.4.2/lib/i386/libawt.so: libXtst.so.6: cannot open shared object file: No such file or directory occurred..
java.lang.UnsatisfiedLinkError: /tmp/OraInstall2013-02-11_01-46-40PM/jre/1.4.2/lib/i386/libawt.so: libXtst.so.6: cannot open shared object file: No such file or directory
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1560)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1477)
        at java.lang.Runtime.loadLibrary0(Runtime.java:788)
        at java.lang.System.loadLibrary(System.java:834)
        at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:38)

Solution

Although the following command is showing that libXtst package is installed for both 32bit and 64bit.
#  rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})
libXt-1.0.2-3.2.el5 (x86_64)
libXt-1.0.2-3.2.el5 (i386)

But oracle installer is trying to get file libXtst.so.6 from /usr/lib/libXtst.so.6 where libXtst.so.6 file doesn't exist. The 32bit version of the file is in /usr/X11R6/lib/libXtst.so.6 location.

So, I have created a softlink using following command and restarted the installation.
ln -s /usr/X11R6/lib/libXtst.so.6 /usr/lib/libXtst.so.6

cached from: http://oracle-latest-technology.blogspot.com/2013/02/libxtstso6-cannot-open-shared-object.html

Creating a Local Yum Repository Using an ISO Image

Note

The system must have sufficient storage space to host a full Oracle Linux Media Pack DVD image (approximately 3.5 GB for Oracle Linux Release 6 Update 3).

To create a local yum repository (for example, if a system does not have Internet access):

1. On a system with Internet access, download a full Oracle Linux DVD image from the Oracle Software Delivery Cloud at http://edelivery.oracle.com/linux onto removable storage (such as a USB memory stick). For example,V33411-01.iso contains the Oracle Linux Release 6 Update 3 Media Pack for x86 (64 bit).
2. Transfer the removable storage to the system on which you want to create a local yum repository, and copy the DVD image to a directory in a local file system.

   # cp /media/USB_stick/V33411-01.iso /ISOs

3. Create a suitable mount point, for example /var/OSimage/OL6.3_x86_64, and mount the DVD image on it.

   # mkdir -p /var/OSimage/OL6.3_x86_64
   # mount -o loop /ISOs/V33411-01.iso /var/OSimage/OL6.3_x86_64

4. Create an entry in /etc/fstab so that the system always mounts the DVD image after a reboot.

   /ISOs/V33411-01.iso /var/OSimage/OL6.3_x86_64 loop defaults 0 0

5. In the /etc/yum.repos.d directory, edit the existing repository files, such as public-yum-ol6.repo or ULN-base.repo, and disable all entries by setting enabled=0.
6. Create the following entries in a new repository file (for example, /etc/yum.repos.d/OL63.repo).

   [OL63]
   name=Oracle Linux 6.3 x86_64
   baseurl=file:///var/OSimage/
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
   gpgcheck=1 
   enabled=1 

7. Clean up the yum cache.

   # yum clean all

8. Test that you can use yum to access the repository.

   # yum repolist
   Loaded plugins: refresh-packagekit, security
   ...
   repo id                    repo name                                status
   OL63                       Oracle Linux 6.3 x86_64                  25,459
   repolist: 25,459

Ref: http://docs.oracle.com/cd/E37670_01/E37355/html/ol_create_repo.html

Writing udev rules

Built-in persistent naming schemes

udev provides persistent naming for some device types out of the box. This is a very useful feature, and in many circumstances means that your journey ends here: you do not have to write any rules.
udev provides out-of-the-box persistent naming for storage devices in the /dev/disk directory. To view the persistent names which have been created for your storage hardware, you can use the following command:

# ls -lR /dev/disk

This works for all storage types. As an example, udev has created /dev/disk/by-id/scsi-SATA_ST3120827AS_4MS1NDXZ-part3 which is a persistent-named symbolic link to my root partition. udev creates /dev/disk/by-id/usb-Prolific_Technology_Inc._USB_Mass_Storage_Device-part1 when I plug my USB flash disk in, which is also a persistent name.

Rule writing

Rule files and semantics

When deciding how to name a device and which additional actions to perform, udev reads a series of rules files. These files are kept in the /etc/udev/rules.d directory, and they all must have the .rules suffix.
Default udev rules are stored in /etc/udev/rules.d/50-udev.rules. You may find it interesting to look over this file - it includes a few examples, and then some default rules proving a devfs-style /dev layout. However, you should not write rules into this file directly.
Files in /etc/udev/rules.d/ are parsed in lexical order, and in some circumstances, the order in which rules are parsed is important. In general, you want your own rules to be parsed before the defaults, so I suggest you create a file at /etc/udev/rules.d/10-local.rules and write all your rules into this file.
In a rules file, lines starting with "#" are treated as comments. Every other non-blank line is a rule. Rules cannot span multiple lines.
One device can be matched by more than one rule. This has it's practical advantages, for example, we can write two rules which match the same device, where each one provides its own alternate name for the device. Both alternate names will be created, even if the rules are in separate files. It is important to understand that udev will not stop processing when it finds a matching rule, it will continue searching and attempt to apply every rule that it knows about.

Rule syntax

Each rule is constructed from a series of key-value pairs, which are separated by commas. match keys are conditions used to identify the device which the rule is acting upon. When all match keys in a rule correspond to the device being handled, then the rule is applied and the actions of the assignment keys are invoked. Every rule should consist of at least one match key and at least one assignment key.
Here is an example rule to illustrate the above:

KERNEL=="hdb", NAME="my_spare_disk"

The above rule includes one match key (KERNEL) and one assignment key (NAME). The semantics of these keys and their properties will be detailed later. It is important to note that the match key is related to its value through the equality operator (==), whereas the assignment key is related to its value through the assignment operator (=).
Be aware that udev does not support any form of line continuation. Do not insert any line breaks in your rules, as this will cause udev to see your one rule as multiple rules and will not work as expected.

Basic Rules

udev provides several different match keys which can be used to write rules which match devices very precisely. Some of the most common keys are introduced below, others will be introduced later in this document. For a complete list, see the udev man page.

• KERNEL - match against the kernel name for the device
• SUBSYSTEM - match against the subsystem of the device
• DRIVER - match against the name of the driver backing the device

After you have used a series of match keys to precisely match a device, udev gives you fine control over what happens next, through a range of assignment keys. For a complete list of possible assignment keys, see the udev man page. The most basic assignment keys are introduced below. Others will be introduced later in this document.

• NAME - the name that shall be used for the device node
• SYMLINK - a list of symbolic links which act as alternative names for the device node

As hinted above, udev only creates one true device node for one device. If you wish to provide alternate names for this device node, you use the symbolic link functionality. With the SYMLINK assignment, you are actually maintaining a list of symbolic links, all of which will be pointed at the real device node. To manipulate these links, we introduce a new operator for appending to lists: +=. You can append multiple symlinks to the list from any one rule by separating each one with a space.

KERNEL=="hdb", NAME="my_spare_disk"

The above rule says: match a device which was named by the kernel as hdb, and instead of calling it hdb, name the device node as my_spare_disk. The device node appears at /dev/my_spare_disk.

KERNEL=="hdb", DRIVER=="ide-disk", SYMLINK+="sparedisk"

The above rule says: match a device which was named by the kernel as hdb AND where the driver is ide-disk. Name the device node with the default name and create a symbolic link to it named sparedisk. Note that we did not specify a device node name, so udev uses the default. In order to preserve the standard /dev layout, your own rules will typically leave the NAME alone but create some SYMLINKs and/or perform other assignments.

KERNEL=="hdc", SYMLINK+="cdrom cdrom0"

The above rule is probably more typical of the types of rules you might be writing. It creates two symbolic links at /dev/cdrom and /dev/cdrom0, both of which point at /dev/hdc. Again, no NAME assignment was specified, so the default kernel name (hdc) is used.

Matching sysfs attributes

The match keys introduced so far only provide limited matching capabilities. Realistically we require much finer control: we want to identify devices based on advanced properties such as vendor codes, exact product numbers, serial numbers, storage capacities, number of partitions, etc.
Many drivers export information like this into sysfs, and udev allows us to incorporate sysfs-matching into our rules, using the ATTR key with a slightly different syntax.
Here is an example rule which matches a single attribute from sysfs. Further detail will be provided later in this document which will aid you in writing rules based on sysfs attributes.

SUBSYSTEM=="block", ATTR{size}=="234441648", SYMLINK+="my_disk"

Device hierarchy

The Linux kernel actually represents devices in a tree-like structure, and this information is exposed through sysfs and useful when writing rules. For example, the device representation of my hard disk device is a child of the SCSI disk device, which is in turn a child of the Serial ATA controller device, which is in turn a child of the PCI bus device. It is likely that you will find yourself needing to refer to information from a parent of the device in question, for example the serial number of my hard disk device is not exposed at the device level, it is exposed by its direct parent at the SCSI disk level.
The four main match keys introduced so far (KERNEL/SUBSYSTEM/DRIVER/ATTR) only match against values corresponding to the device in question, and do not match values from parent devices. udev provides variants of the match keys that will search upwards through the tree:

• KERNELS - match against the kernel name for the device, or the kernel name for any of the parent devices
• SUBSYSTEMS - match against the subsystem of the device, or the subsystem of any of the parent devices
• DRIVERS - match against the name of the driver backing the device, or the name of the driver backing any of the parent devices
• ATTRS - match a sysfs attribute of the device, or a sysfs attribute of any of the parent devices

With hierarchy considerations in mind, you may feel that rule writing is becoming a little complicated. Rest assured that there are tools that help out here, which will be introduced later.

Heterogenous connection from Oracle database to Microsoft SQL server

Using Oracle's heterogenous services also known as HS, we can make a connection to non-Oracle databases and access the data remotely. I used MS SQL server as the target non-Oracle database on Windows and used HS to access from Oracle database residing on Linux. Here, we'll see step by step instructions on how to set up a connection from Oracle database to Microsoft SQL server.

In this example, I've used the following:
Source Oracle database on Linux:test-oralin
Target SQL server on Windows:test-mssqlwin
DSN name:MSORA

Our objective is to make a connection from test-oralin to test-mssqlwin and access the SQL server database from Oracle. Here are the steps:

1. On test-mssqlwin, install Oracle Home. Example: E:\OraHome
Note: Only Oracle binaries are needed here so you need not create a database.

2. On test-mssqlwin, install ODBC drivers so that Oracle can access SQL server

3. On test-mssqlwin, configure ODBC by setting up a system DSN(Data Source Name)
Typically you can find this at Start->Settings->Control Panel->Administrative Tools->Data Sources(ODBC) however depending on the Windows version, this may differ slightly.
On System DSN tab, click Add button and then choose the driver you want to use to connect to SQL server. Continue with the ODBC configuration by entering the details of the SQL server you wish to connect to along with necessary credentials. For this test, I've chosen "MSORA" as my DSN name.

4. On test-mssqlwin, setup the heterogenous services init file.
Look for inithsodbc.ora in $ORACLE_HOME/hs/admin/ which in our case would be E:\OraHome\hs\admin. Make a copy of it and rename it to initMSORA.ora. Remember we had chosen "MSORA" as our DSN name. Open the file and enter the DSN name that you just created recently i.e "MSORA". This is how E:\OraHome\hs\admin\initMSORA.ora should look like:
# This is a sample agent init file that contains the HS parameters that are needed for an ODBC Agent.
#
# HS init parameters
#
HS_FDS_CONNECT_INFO = MSORA
HS_FDS_TRACE_LEVEL = OFF

5. On test-mssqlwin, configure listener.ora. Take a sample copy from E:\OraHome\hs\admin\listener.ora.sample and copy it to E:\OraHome\network\admin\listener.ora
This is how your E:\OraHome\network\admin\listener.ora should look like:

LISTENER_MSORA =
 (ADDRESS_LIST=
      (ADDRESS=(PROTOCOL=tcp)(HOST=test-mssqlwin)(PORT=1521))
      (ADDRESS=(PROTOCOL=ipc)(KEY=PNPKEY)))

SID_LIST_LISTENER_MSORA=
  (SID_LIST=
      (SID_DESC=
         (SID_NAME=MSORA)
         (ORACLE_HOME = E:\OraHome)
         (PROGRAM=hsodbc)
      )
  )

Note: SID_NAME should be your DSN name. PROGRAM=hsodbc tells Oracle to use heterogenous services. Restart your listener to make sure the settings are in effect.

We have completed our configuration on the target MS SQL server i.e test-mssqlwin. Now let us move to our source Oracle database i.e test-oralin from where we wish to access test-mssqlwin

6. On test-oralin, configure your $ORACLE_HOME/network/admin/tnsnames.ora
MSORA =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = test-mssqlwin)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SID = MSORA)
    )
   (HS=OK)
  )
Note:SID should be the DSN name created on test-mssqlwin. PORT should be same as what was configured for the listener on Oracle Home that was installed on test-mssqlwin. HS=OK means Oracle will use heterogenous services.

7. Verify the connection from test-oralin to test-mssqlwin by doing a tnsping MSORA from test-oralin.

8. We are all set now so let us access the SQL server database now from Oracle.
On test-oralin, connect to the Oracle database from where you want to access SQL server database located on test-mssqlwin
Make sure that global_names parameter is set to false
SQL> sho parameter global_names

NAME               TYPE             VALUE
------------------ ---------------- -------
global_names       boolean          FALSE

Create a database link:
CREATE DATABASE LINK <> CONNECT TO <> IDENTIFIED BY <> USING '<>';
Example:
SQL> CREATE DATABASE LINK MSORA CONNECT TO REPL IDENTIFIED BY REPL USING 'MSORA';

Database link created.

Note: REPL is a user on SQL server database i.e test-mssqlwin with REPL as it's password. 'MSORA' was given as a connection string in tnsnames.ora. Here, we have chosen MSORA as dblinkname, you could have any name.

We have completed our configuration. You may start accessing the data now.
Example:
SQL> desc emp@msora
 Name          Null?      Type
 ------------- --------   --------------
 EMPID         NOT NULL   NUMBER(1)
 EMPNAME                  VARCHAR2(60)
 CREATED                  DATE

reference: http://www.ritzyblogs.com/OraTalk/PostID/107/Heterogenous-connection-from-Oracle-database-to-Microsoft-SQL-server

http://www.dba-oracle.com/t_heterogeneous_database_connections_sql_server.htm