12.1.2 to 12.1.3 upgrade on Solaris

We have recently upgrade our instances to 12.1.3. Here is what we did.

1) 10.1.3.5.0 patch 8626084
2) From Apps tier Source $INST_TOP/ora/10.1.3/sid_hostname.env file to set $ORACLE_HOME as 10.1.3 home.
3) cd /8626084/Disk1
4) ./runInstaller -invPtrLoc $ORACLE_HOME/oraInst.loc
5) On fourth screen it will ask for oc4jadmin password. Kindly put "oafm".
6) Stop opmn processes, keeping same 10.1.3 env.
$ORACLE_HOME/opmn/bin/opmnctl stopall
7) Unzip patch p6880880_101000_SOLARIS64.zip in 10.1.3 and 10.1.2 OH to have latest opatch version.
8) Run Autoconfig on both Apps nodes.
9) apply patch p8999551_101350_Generic.zip in 10.1.3 home via opatch :
opatch apply
10) Confirm patchset version:
Run the command 'opatch lsinventory -detail'
Look for 'Oracle Application Server PatchSet' in the 'Product' column. 'Version' should show 10.1.3.5.0.
11) Now apply apps patches, enable Maintenance Mode via adamin
12) Apply patch p9239089_R12.AD.B_R12_SOLARIS64.zip
Create $ORACLE_HOME/appsutil/admin on the database server.
Copy adgrants.sql (UNIX) from this patch directory to
$ORACLE_HOME/appsutil/admin
Set the environment to point to ORACLE_HOME on the database server
Login as sysdba and run
SQL> @$ORACLE_HOME/appsutil/admin/adgrants.sql APPS
13) Apply patch 9239090 12.1.3 patch. (Take 5-6 hrs)
14) Apply online help patch p9239095_R12_GENERIC.zip
15) Apply post patches p9817770_R12.ATG_PF.B_R12_SOLARIS64.zip and p9966055_R12.FND.B_R12_GENERIC.zip
16) Run autoconfig on both Apps tier
17) Run the admkappsutil.pl utility to create the file appsutil.zip in the
/admin/out directory
perl $AD_TOP/bin/admkappsutil.pl
18) On Database tier copy this file
19) And unzip in OH cd
unzip -o appsutil.zip
20) Run autoconfig on DB Tier
21) Run adpreclone on both DB Tier and Apps Tier
perl adpreclone.pl dbTier
perl adpreclone.pl appsTier
22) Disable Maintenance Mode via adamin
23) Needs to deploy forms before starting services on apps tier
24) Stop all the Application Services
cd $ADMIN_SCRIPTS_HOME
adstpall.sh
25) Back up the original system-jazn-data.xml file.
cp $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml.orig
26) Add the following :
vi $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml
========= file snippet original =================

oc4jadmin
OC4J Administrator
oc4jadmin
OC4J Administrator
OC4J Administrator
{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO

================================================
=========== with change ============================

oc4jadmin
OC4J Administrator
oc4jadmin
OC4J Administrator
OC4J Administrator
!welcome

===================================================
:wq!
27) Back up the original opmn.xml file
cp $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml.orig
28) Add the following line:
ipaddr remote="realhost.domain.com" request="virtualhost.domain.com"
as
notification-server interface="ipv4"
ipaddr remote="realhost.domain.com" request="virtualhost.domain.com"
port local="#local_port" remote="#remote_port" request="#request_port"
29) Run the deployment script
$FND_TOP/bin/txkrun.pl -script=CfgOC4JAppEnter
Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome
Run Autoconfig ? No
30) Restore both system-jazn-data.xml and opmn.xml from the backup taken under step 25 and 27 respectively
31) Run autoconfig
$ADMIN_SCRIPTS_HOME/adautocfg.sh
32) Start the services.
$ADMIN_SCRIPTS_HOME/adstrtal.sh apps/appspwd

Once done, you will be able to access forms now on 12.1.3.
Happy Troubleshooting !!!

Finding XML Publisher Version in R12

Its being long I have posted something. Got Stuck in major milestone in our project and 12.1.3 upgrade.

1) The following SQL statement will show which patches and thus with version of Oracle XML Publisher is installed on an Oracle E-Business Suite Release 12.0 and 12.1 environment:

SELECT DECODE (bug_number, '4440000', '5.6.3') PATCH, bug_number
FROM ad_bugs WHERE bug_number IN('4440000');
PATCH BUG_NUMBER
----- ------------------------------
5.6.3 4440000

2) The version of a specific Java Class has a one to one relationship with the currently installed version of Oracle XML Publisher. This method is suitable for both Oracle E-Business Suite 11 and Release 12.

$ adident Header $OA_JAVA/oracle/apps/xdo/common/MetaInfo.class:
$Header MetaInfo.java 120.6 2006/08/17 01:25:30 bgkim noship $


Metainfo.class XML Publisher Patch Reference Included in ... R12
120.6 5.6.3 4440000 Note 422508.1

Happy Troubleshooting !!!

Unable to start OACORE OC4J in R12

We were doing JRE upgrade today and had issue in one of the instance not coming up. Basically OACORE OC4J service was unable to get start. Script adstrtal.sh was failing to start this service and erroing out:
==================================================
Executing service control script:/Path/admin/scripts/adoacorectl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance ...
****************************************************
===================================================

After looking at log file :
===================================================
06/18/10-01:32:04 :: adoacorectl.sh: Starting OPMN managed OACORE OC4J instance
opmnctl: starting opmn managed processes...
================================================================================opmn id=tsgsd2107:6204
0 of 1 processes started.
ias-instance id=XXXXXXXXXXXXX.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/oacore/default_group/
Error
--> Process (index=1,uid=237255567,pid=24554)
failed to start a managed process after the maximum retry limit
Log:
/Path/logs/ora/10.1.3/opmn/default_group~oacore~default_group~1.log
06/18/10-01:36:43 :: adoacorectl.sh version 120.13
06/18/10-01:36:43 :: adoacorectl.sh: Stopping OPMN managed OACORE OC4J instance
==============================================
OPMN log says this:

==============================================
10/06/18 01:35:32 [libopmnoc4j] Host and Port information for port type jms not sent by the OC4J process
10/06/18 01:35:32 [libopmnoc4j] WARNING: OC4J did not send the protocol value for port id: jms
10/06/18 01:35:32 [libopmnoc4j] Port information in the ONS notification is incorrect for proc: 237255567.
Some of the reasons for the failure are:
1. ajp, rmi and jms ports could not be bound by the process.
2. If a port value for a certain port ID is handed by OPMN to the OC4J process and this value is not returned back in the ONS notifications.
For example, this can happen for certain bad JGroups related configurations.
10/06/18 01:35:32 [libopmnoc4j] Process Start Error: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:34 [pm-process] Stopping Process: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:34 [libopmnoc4j] Process Stop Error: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:34 [libopmnoc4j] Failed to construct stop command for proc: 237255567
10/06/18 01:35:34 [libopmnoc4j] Forcefully Terminating Process: default_group~oacore~default_group~1 (237255567:24554)
10/06/18 01:35:36 [pm-requests] Request 7 Completed. Command: /start?process-type=oacore
=======================================

Now big thing is solution. Here you go:
Go to $INST_TOP/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1 on application tier and look for jms.state and *.lock files. If you find them in this location once all you services are stop. Delete or rename them. Once you are done try again. Now OACORE OC4J service will come up.

Happy Troubleshooting !!!

11gR2 upgrade with 12.1.2

Steps for 11gR2 Upgrade
Pre-Upgrade Task
Remove DB from archive log mode
Complete Backup of the Database & Application Binaries
Apps Patches:
9062910 -- 11g Release 2 interoperability patch for Release 12.1
7651166 -- 12.1 TXK Delta 2 patch
Adconfig Patches
8966480 -- FWDPORT 8942827 - 11.2DB:APPS11I:NEED CHANGES TO TEMPLATE FILES TO SUPPORT 11GR2
9366993 -- ADBLDXML AND AUTOCONFIG COMPLETES WITH NJNI11:JAVA.LANG.UNSATISFIEDLINKERROR

Set the path of oraInventory in /var/opt/oracle/oraInst.loc file.
Make entry for the oracle home that needs to be upgrade in /var/opt/oracle/oratab file.
Take a backup of the existing RDBMS oraInventory folder.

Software Installation Task
Set the path for ORACLE_SID and ORACLE_BASE env variable
Create a 11.2.0 directory under the $ORACLE_BASE.
Execute the runInstaller to start the 11.2.0 Software only installation.
Click the Ignore all checkbox on Perform Prerequisite Checks when the checks fail for OS Patch - 124861-15 as this is an optional patch.
Run root.sh through root user when prompted.
Install the 11g Examples CD

"After the installation, make sure that: "
The ORACLE_SID environment variable must be set accordingly.
The ORACLE_BASE environment variable must be set accordingly.
The ORACLE_HOME environment variable points to the new 11.2.0 Oracle home.
The PATH environment variable includes $ORACLE_HOME/bin and the directory where the new perl executable is located (usually $ORACLE_HOME/perl/bin).
The LD_LIBRARY_PATH environment variable includes $ORACLE_HOME/lib.
The PERL5LIB environment variable points to the directories where the new perl libraries are located (usually $ORACLE_HOME/perl/lib/[perl version] and $ORACLE_HOME/perl/lib/site_perl/[perl version])

Example:
ORACLE_SID=test11g; export ORACLE_SID
ORACLE_BASE=/test11g/oracle; export ORACLE_BASE
ORACLE_HOME=/test11g/oracle/11.2.0; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/perl/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib:/usr/dt/lib:/usr/openwin/lib; export LD_LIBRARY_PATH
PERL5LIB=$ORACLE_HOME/perl/lib/5.10.0:$ORACLE_HOME/perl/lib/site_perl/5.10.0; export PERL5LIB
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN

Create nls/data/9idata directory
From 11.2.0 Oracle Home run the following:
perl $ORACLE_HOME/nls/data/old/cr9idata.pl
Check the directory created --- $ORACLE_HOME/nls/data/9idata
set the env variable ORA_NLS10=$ORACLE_HOME/nls/data/9idata

Additional RDBMS Patch
Apply RDBMS Patch - 9218789
Note: Do not run any of the post install instructions as those will be done after the upgrade
Stop the Application Processes and database listener.
From old ORACLE_HOME --Connect DB as sysdba and Drop SYS.ENABLED$INDEXES
SQL> drop table sys.enabled$indexes;

Prepare for DB Upgrade
Copy utlu112i.sql from 11.2.0 ORACLE_HOME/rdbms/admin to a temporary folder.
"From Old ORACLE_HOME, connect DB as SYSDBA"
SQL> SPOOL upgrade_info.log
SQL> @utlu112i.sql
SQL> SPOOL off
Check for the output of the script.
Add space to SYSAUX Tablespace.
SQL> alter database datafile '//dbdata/data1/sysaux01.dbf' resize 600M;

add the following paraneter to init.ora file:
java_pool_size=128M
comment out the following parameters from init.ora file:
#plsql_native_library_dir = //oracle/11.1.0/plsql/nativelib
#plsql_native_library_subdir_count = 149

Restart the Database
Connect database as SYSDBA and execute the following queries:
SQL> SELECT * FROM v$recover_file;
SQL> SELECT * FROM v$backup WHERE status != 'NOT ACTIVE';
SQL> SELECT * FROM dba_2pc_pending;
SQL> PURGE DBA_RECYCLEBIN;
Re-execute utlu112i.sql

DB Upgrade Task
Run netca from old 11.1.0 Oracle Home and delete the listener.
Shutdown the database and comment out the following parameter in 11.1.0 Oracle Home's init.ora # local_listener='_LOCAL'

Make sure the following envs' are set properly:
$ export ORACLE_SID=
$ export ORACLE_HOME=
$ export PATH=$ORACLE_HOME/bin:$PATH
$ export ORACLE_BASE=
$ export TNS_ADMIN=$ORACLE_HOME/network/admin
Run netca from new 11.2.0 Oracle Home and create the new listener.
Run database upgrade assistant as follows:
"$ dbua -initParam ""_disable_fast_validate=TRUE"""

Modify initialization parameters
Use the init.ora parameter setting for the 11.2.0 as per NoteID:396009.1
"Ensure the parameter ""_disable_fast_validate=TRUE"" is set.."

Restart the Database.
Start the Database listener.
Connect with SYS and run only catmgdidcode.sql and utlrp.sql as part of the post install instructions in patch 9218789. The other scripts were already run as part of the dbua upgrade.
SQL> conn / as sysdba
SQL> @?/md/admin/catmgdidcode.sql
SQL> @?/rdbms/admin/utlrp.sql

Post DB Upgrade Tasks
Copy $APPL_TOP/admin/adgrants.sql from admin node to temporary folder.
Run as follows:
"$ sqlplus ""/ as sysdba"" @adgrants.sql apps"

Copy $AD_TOP/patch/115/sql/adctxprv.sql from admin node to temporary folder.
Run as follows:
$ sqlplus apps/[APPS password] @adctxprv.sql \
[SYSTEM password] CTXSYS
"$ sqlplus ""/ as sysdba"""
"SQL> exec ctxsys.ctx_adm.set_parameter('file_access_role', 'public');"

Login to Admin Node and execute the following script connected with APPS schema.
$ sqlplus apps/[APPS password] @$FND_TOP/patch/115/sql/wfaqupfix.sql APPLSYS APPS

Implement and run AutoConfig
Make sure all the above pre-req Patches are applied mentioned in lines 8-13
Create appsutil.zip
Login to Application Tier and Source the env
perl $AD_TOP/bin/admkappsutil.pl
the scripts will create appsutil.zip in $INST_TOP/admin/out
Copy AutoConfig to the RDBMS ORACLE_HOME
Copy or FTP the appsutil.zip file to the
cd
unzip -o appsutil.zip

Install JRE on the Database tier
Download Latest Update of JRE 6.0
http://java.sun.com/products/archive/
Ftp the file to $ORACLE_HOME/appsutil
$ cd $ORACLE_HOME/appsutil
$ sh jre1.6.0._latest_update#.sh
$ mv jre1.6.0._latest_update# jre

Generate the Database Context File
$ cd $ORACLE_HOME/appsutil/bin
$ perl adbldxml.pl
"Note: After creating the XML context file, do the following changes:"
1. Backup the existing context_file.
2. rename the existing context_file (_.xml) to _.xml [E.g SID_hostname.xml -> sid_vhost.xml]
3. Open the _.xml and change all the occurance of to vi sid_vhost.xml
4. ensure variable s_jretop points to a proper JRE directory i.e. $ORACLE_HOME/jdk/jre.
5. ensure variable s_db_tnsadmin points to $ORACLE_HOME/network/admin/$CONTEXT_NAME

Run AutoConfig on the Database tier
$ cd $ORACLE_HOME/appsutil/bin
$ ./adconfig.sh contextfile=

Gather statistics for SYS schema
Copy $APPL_TOP/admin/adstats.sql from admin node to temporary folder.
Note: adstats.sql has to be run in restricted mode.
Make sure that you have at least 1.5 GB of free default temporary tablespace.
"$ sqlplus ""/ as sysdba"""
SQL> alter system enable restricted session;
SQL> @adstats.sql
"$ sqlplus ""/ as sysdba"""
SQL> alter system disable restricted session;
SQL> exit;

Re-create custom database links (conditional)
Re-create grants and synonyms
"Run ""Recreate grants and synonyms for APPS schema"" through ADADMIN."

Restart Applications server processes
Synchronize Workflow views
"Log on to Oracle E-Business Suite with the ""System Administrator"" responsibility"
Run the request with the following parameteres:
Request Name = Workflow Directory Services User/Role Validation
p_BatchSize = 10000
p_Check_Dangling = Yes
Add missing user/role assignments = Yes
Update WHO columns in WF tables = No

Refernce Metalink Notes:
ID 1058763.1 Interoperability Notes EBS R12 with Database 11gR2.
ID 387859.1 Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12.
ID 396009.1 Database Initialization Parameters for Oracle Applications Release 12.
ID 418664.1 Overview of Using Java with Oracle E-Business Suite Release 12
ID 455492.1 Using Latest Update of Java 6.0 with Oracle E-Business Suite Release 12

------------------------------------------
Happy Troubleshooting !!!

12.1.2 CU patchset

We have recently upgraded our 12.1.1 env to 12.1.2, it has only one AD patch prereq nothing else. It went smoothly with no errors.

We are planning to move our 11gR1 database to 11gR2 soon once minimum patch requirement for OS level met. Will keep you posted.

Happy Troubleshooting !!!