ADOP usage in EBS R12.2

Hai, in this blog we learn some useful tips and tricks while using ADOP:

List of Topics (Linux, Database, RAC, EBS)

################################################################################################################

################################################################################################################

ADOP Session error and LOG files findings:

Find log command for fsclone in sequential procedure:

ls -ltrh $ADOP_LOG_HOME

find $ADOP_LOG_HOME/28 -iname ADOPValidations*.log

find $ADOP_LOG_HOME/28 -iname txkADOPValidations.log

find $ADOP_LOG_HOME/28 -iname txkCloneAcquirePort.log   --depend till validation completes

find $INST_TOP/admin/log/clone/ -iname FSCloneStageAppsTier*.log

find $INST_TOP -iname FSCloneStageAppsTier*.log    --gets created after start of txkADOPPPS

find $ADOP_LOG_HOME/28 -iname FSCloneStageAppsTier*.log   --gets created and provide result after INST_TOP StageAppTier file gets finished with success/error

find $ADOP_LOG_HOME/28 -iname txkADOPPreparePhaseSynchronize.log   --create after acquire port complete

find $COMMON_TOP/adop* -iname FMW_Home.jar

watch -n 1 'ls -ltrh'

watch -n 1 'ls |wc -l $PATCH_BASE/EBSapps/appl/*'

watch -n 1 'ls |wc -l $PATCH_BASE/EBSapps/comn/*'

watch -n 1 'ls |wc -l $PATCH_BASE/EBSapps/10.1.2/*'

watch -n 1 'ls |wc -l $PATCH_BASE/FMW_Home/*'

   

find $INST_TOP/admin/log/clone/ -iname FSCloneApplyAppstier*.log --imp. depend till above folders deletion completes

find $INST_TOP/admin/log/clone/ -iname CLONE*.log

find $INST_TOP/admin/log/clone/fmwT2PApply/ -iname CLONE*.log

find $INST_TOP/admin/log/clone/wlsT2PApply/ -iname CLONE*.log

find $INST_TOP/admin/log/clone/ohsT2PApply/ -iname CLONE*.log

ls -ltrh $INST_TOP/admin/log/clone/fmwT2PApply/CLONE*.log

find $ADOP_LOG_HOME/28 -iname FSCloneApplyAppstier*.log    --gets created and provide result after INST_TOP ApplyAppTier file gets finished with success/error

---------------- additional information logs in fsclone.

find $INST_TOP -iname CLONE*.error

find $INST_TOP -iname *.error

find $ADOP_LOG_HOME/28 -iname *.error

find $ADOP_LOG_HOME/28 -iname CLONE*.log

find $ADOP_LOG_HOME/28 -iname CLONE*.error

ls -ltrh $ADOP_LOG_HOME

find $ADOP_LOG_HOME/28 -iname ADOPValidations*.log

find $ADOP_LOG_HOME/28 -iname txkADOPValidations.log

find $ADOP_LOG_HOME/28 -iname txkCloneAcquirePort.log

----------------------------------------------------------

----------------------------------------------------------

Find log command for prepare phase:

find $ADOP_LOG_HOME/28 -iname txkADOPValidations.log

find $ADOP_LOG_HOME/28 -iname txkADOPPreparePhaseSanityCheck.log

find $ADOP_LOG_HOME/28 -iname txkADOPPreparePhaseSynchronize.log

find $ADOP_LOG_HOME/28 -iname ADOPValidations*.log

find $INST_TOP/admin/log/clone -iname FSCloneApplyAppsTier_*.log

Find log command for apply phase udriver log:

find $ADOP_LOG_HOME/28 -iname u*log

find . -type f \( -iname \u*.log -o -iname \u*.lgi \)

Find log command for finalize cutover cleanup phase:

find $ADOP_LOG_HOME/28 -iname actualize_all.log

find $ADOP_LOG_HOME/28 -type f \( -iname \actualize_all.log -o -iname \actualize_all.lgi \)

find $ADOP_LOG_HOME/28 -type f \( -iname \cutover.log -o -iname \cutover.lgi \)

find $ADOP_LOG_HOME/28 -type f \( -iname \cleanup.log -o -iname \cleanup.lgi \)

find $ADOP_LOG_HOME/28 -iname adzdallddls.out generates when ran with actualize & cleanup full mode

Find log command for abort phase:

find $ADOP_LOG_HOME/28 -iname adzdshowlog.out

find $ADOP_LOG_HOME/28 -iname abort.log

Find log command for worker log in apply phase:

find $ADOP_LOG_HOME/28 -iname adwork*.log

find $NE_BASE/EBSapps/log/adadmin/log -iname adwork*.log

ps -ef | grep adworker

Find out which worker failed if autoskip flag parameter added

grep -i 'AD Worker error:\|failed' $ADOP_LOG_HOME/28/*/apply/$HOSTNAME/*/log/adwork*.log (Brings the worker file which failed, from that find aborted DB object details in mentioned file as belwo)

grep abort -i $ADOP_LOG_HOME/28/*/apply/uatapp01/28840844_AR/log/l9875758.req

ADOP parameter file:

Troubleshooting ADOP failures:

grep -A 6 MISSING_TRANSLATED_VERSION $APPL_TOP/admin/${TWO_TASK}/adalldefaults.txt

################################################################################################################

################################################################################################################

Backup Procudure for PROD instances for full consistent backup:

- stop application and make tar bkp, So no schedule requests will be running to update data in database

- Make RMAN full Backup after stop applciation

- Make DB_HOME tar if appliang any db patches

- Make list of all invalid object

To avoid ADOP network dropout set parameters before adop and run adop session in screen or tmux:

[root@rac1 ~]# tmux -f ~/.tmux.conf (force load all customizing in tmux session)

[root@rac1 ~]# tmux ls ----> tmux attach -t 0 ----> copy/paste shift + selection to paste Shift + R.clk paste

ADOP Phases and different mode options: 

current state of adop session and prepare state:

and try to run autoconfig before starting adop cycle, So it will apply current context variables changes to Run FS, which will copied to Patch FS

SQL> select * from all_objects where status like 'INVALID' and owner='APPS';

SQL> show parameter cpu

[applmgr@ebsprodapp3 ~]$ 

[applmgr@ebsprodapp3 ~]$ grep s_ohs_installed $CONTEXT_FILE

         <config_option type="techstack" oa_var="s_ohs_installed">generateEBSOHSConfigFiles</config_option>

[applmgr@ebsprodapp3 ~]$

[applmgr@ebsprodapp3 ~]$ adop -status

[applmgr@ebsprodapp3 ~]$ adop -validate

adop exiting with status = 0 (Success)

[applmgr@ebsprodapp3 ~]$ free -h -w

[applmgr@ebsprodapp3 ~]$ time adop phase=prepare  (approx. 25 mins)

ps -ef | grep adworker

ps -a | grep adworker

Determining Number of Workers Oracle EBS Maintenance Guide Recommendation (page 285):   

The AD utilities provide a default number of workers of twice the number of CPUs on the database server. Oracle recommends you choose a number of workers between 2-4 times the number of CPUs. For example, if there are four CPUs on the database server, you should choose something in the range of 8-16 workers.

Big Patches better to apply in downtime mode (if more than 2000+ jobs):

if applying a big apps patch like Product Family(hr_pf14) on PROD and if PRODDB is on RAC then make cluster=false and bring db to single node is mandatory. moreover for big apps patch add extra workers=16/24 to speed up (default is 8 workers)

SQL> show parameter cpu (8x4=32, 8x2=16) best is 16/24

SQL> shut immeidate # on all RAC-NODES except 1

SQL> alter system set cluster_database=FALSE scope=spfile sid='*'; #MP

SQL> show parameter cluster

SQL> select count(*) from v$session; --gv$session;

select name,open_mode,LOG_MODE,FLASHBACK_ON,FORCE_LOGGING from v$database;

select * from applsys.TXK_TCC_RESULTS;

SQL> select to_char(sysdate,'DD-MM-YYYY HH24:MI:SS') from dual;

SQL> alter system switch logfile;

SQL> alter database flashback on;

SQL> create restore point before_patch guarantee flashback database;

select * from v$restore_point;

[applmgr@ebsprodapp3 ~]$ tnsping PRODDB

Attempting to contact (DESCRIPTION_LIST= (LOAD_BALANCE=YES) (FAILOVER=YES) (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=RAC-DB where EBS DB is up and running R-node should be 1st in list of tns connection description, if (LB & FO) are set to "NO" then no DB connection after cluster is false)

[applmgr@ebsprodapp3 ~]$ sqlplus apps/apps

Running ADOP Fails With Error "ETCC Not Run In The Database Node" (Doc ID 2191678.1)

SQL> select * from apps.TXK_TCC_RESULTS;--etcc checkdb, checkmt ran or not 

[applmgr@ebsprodapp3 ~]$ ls -ltrh $PATCH_TOP

[applmgr@ebsprodapp3 ~]$ 

[applmgr@ebsprodapp3 ~]$ free -h -w

echo 1 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a && printf '\n%s\n' ' Ram-cache and the swap get cleared'

(Number of Parallel Workers (Page 34) --> "N" CPU X 2 to 4 times )

Adop apply fails with " Do you wish to apply this patch now [No] ? No *" (Doc ID 2355751.1)

grep -A 6 MISSING_TRANSLATED_VERSION  $APPL_TOP/admin/${TWO_TASK}/adalldefaults.txt

[applmgr@ebsprodapp3 ~]$

time adop phase=apply patches=XX,XX_AR:uXX.drv apply_mode=downtime workers=24 patchtop=/u01/EBS_PATCH

[applmgr@ebsprodapp3 ~]$ find $ADOP_LOG_HOME/28 -iname u*.log

[applmgr@ebsprodapp3 ~]$ find $ADOP_LOG_HOME/28 -iname u*.lgi

[applmgr@ebsprodapp3 ~]$ ls -ltrh $APPL_TOP/admin/${TWO_TASK}/out/adzdshowlog.out

[applmgr@ebsprodapp3 ~]$ 

[applmgr@ebsprodapp3 ~]$ time adop phase=cleanup workers=24  (approx. time 42 Sec)

if apply phase failed with error then re-try with with option abandon=yes to restart apply from very beginning same or different patch.

[applmgr@ebsprodapp3 ~]$

adop phase=apply patches=XX,XX_AR:uXX.drv apply_mode=downtime workers=24 patchtop=/u01/EBS_PATCH abandon=yes

after completing patching restore DB to RAC and start db on all RAC Nodes:

SQL> alter system set cluster_database=TRUE scope=spfile sid='*';

SQL> startup # all RAC-NODES

SQL> select * from v$restore_point;

Services up time Patching direct on RFS with custom patchtop::

[applmgr@ebsprodapp3 ~]$ adop phase=apply patches=XXXXX hotpatch=yes patchtop=/u01/EBS_PATCH

Downtime mode Patching Command with custom patchtop: 

[applmgr@ebsprodapp3 ~]$ adop phase=apply patches=XXXXX apply_mode=downtime patchtop=/u01/EBS_PATCH 

Downtime mode with reapply Patching Command:

[applmgr@ebsprodapp3 ~]$ adop phase=apply patches=XXXXX apply_mode=downtime options=forceapply 

Language Patching Command:

[applmgr@ebsprodapp3 ~]$ adop phase=apply apply_mode=downtime patches=30467183,30467183_AR:u30467183.drv 

[applmgr@ebsprodapp3 ~]$

[applmgr@ebsprodapp3 ~]$ adop phase=finalize,cutover,cleanup  (approx. 25 mins)

[applmgr@ebsprodapp3 ~]$ 

after completing patching cycle restore DB to RAC and start db on all RAC Nodes:

SQL> alter system set cluster_database=TRUE scope=spfile sid='*';

SQL> select * from v$restore_point;

if all set Run Gather Schema:

=============================================================

NOTE: before Running FS_CLONE take tar backup of Patch File System and set below jvm parameters:

(below jvm and timeout parameters are used only for adcfg dualfs & fs_clone)

free -h -w

export TIMEDPROCESS_TIMEOUT=-1

export CONFIG_JVM_ARG="-Xms1024m -Xmx2048m"

export CONFIG_JVM_ARG="-Xms2048m -Xmx4096m"

echo $CONFIG_JVM_ARG

ps -fux

free -h

export USER_MEM_ARGS="-Xms10240m -Xmx10240m -XX:MaxPermSize=2048m"

export CONFIG_JVM_ARGS="-Xmx4096m -Xms2048m -XX:MaxPermSize=512m -XX:-UseGCOverheadLimit"

export TIMEDPROCESS_TIMEOUT=-1

du -sh $COMMON_TOP/util/jdk32_old

du -sh $COMMON_TOP/util/jdk64_old

du -sh $FMW_HOME/webtier/jdk_old

du -sh $ORACLE_HOME/jdk_old

du -sh $FMW_HOME/utils/bsu/cache_dir

pwd

[applmgr@ebsprodapp3 ~]$ time adop phase=fs_clone  (approx. 2hrs 30 mins)

If fsclone fails with any reason alway start fsclone again with option force=yes

[applmgr@ebsprodapp3 ~]$ time adop phase=fs_clone force=yes (approx. 2hrs 30 mins)

[applmgr@ebsprodapp3 ~]$ 

ADOP abort phases Rarely used:

Abort cycle depends on applied patch size and ddl/dml operation executed on PFS

it is better to let it execute and monitor in adopmon (some time 3+ hrs.)

[applmgr@ebsprodapp3 ~]$

[applmgr@ebsprodapp3 ~]$ time adop phase=abort  (wait until @ADZDSHOWLOG.sql execute normally.)

[applmgr@ebsprodapp3 ~]$ adop phase=abort force=yes (N/A for abort, force option for FSCLONE only )

[applmgr@ebsprodapp3 ~]$

Always check list of invalid objects after abort phase:

SQL> select * from all_objects where status like 'INVALID' and owner='APPS';

[applmgr@ebsprodapp3 ~]$ 

[applmgr@ebsprodapp3 ~]$ adop phase=cleanup cleanup_mode=full

[applmgr@ebsprodapp3 ~]$

=============================================================

=============================================================

Beginning application tier FSCloneStage - Fri Jan 21 13:42:32 2022

. . . . .

Completed FSCloneStage...

Fri Jan 21 13:50:28 2022

*******FATAL ERROR*******

PROGRAM : (/u01/dev/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

TIME    : Fri Jan 21 13:51:24 2022

FUNCTION: main::removeDirectory [ Level 1 ]

ERRORMSG: Failed to delete the directory /u01/dev/fs2/EBSapps/10.1.2.

ERRORMSG: Failed to delete the directory /u01/dev/fs2/FMW_Home

If fsclone fails with function to removeDirectory then simply rename the given directory.

Usually will be only above two. Then re-run fsclone with option force=yes

mv /u01/dev/fs2/FMW_Home /u01/dev/fs2/del_FMW_Home

mv /u01/dev/fs2/EBSapps/10.1.2. /u01/dev/fs2/EBSapps/del_10.1.2.

=============================================================

=============================================================

source Dropping Old Editions With the Actualize_all Phase

In EBS R12.2, as each online patching cycle is completed, the database will accumulate an additional old database edition. An additional column ZD_EDITION_NAME is populated in the seed tables.If the number of these grows too large, system performance will start to be affected. When the number of old database editions reaches 25 or more, we should consider dropping all old database editions by running the adop actualize_all phase and then performing a full cleanup. I normally consider to do it, once the count reaches 15 editions in my Production environment.

Important: This procedure will take a large amount of time (significantly longer than a normal patching cycle), and should only be performed when there is no immediate need to start a new patching cycle.

Before starting, you should ensure that the system has the recommended database patches and latest AD-TXK code level installed.

When no patches need to be applied in Online Patching

To proceed, run the following commands in the order shown:

sqlplus apps @$AD_TOP/sql/ADZDSHOWOBJS.sql

sqlplus apps @$AD_TOP/sql/ADZDSHOWED.sql

SELECT * FROM dba_editions;

SELECT * FROM ad_adop_sessions;

select * from apps.ad_zd_logs order by log_sequence desc; -- to check current actives of adop in table

To proceed, run the following commands in the order shown:

$ ps -eo pid,lstart,cmd|grep -i 'sqlplus\|adop\|tar.gz\|zip' (to check adop session start time)

$ time adop phase=prepare

$ time adop phase=actualize_all   ( to capture adopmn output use Linux script command or adopmn > a.txt (or else) adopmon |& tee output.txt)

$ time adop phase=finalize finalize_mode=full

$ time adop phase=cutover

$ time adop phase=cleanup cleanup_mode=full

=============================================================

=============================================================

Multi Node Application:

if fs_clone fails due to any error then in next retry use option force=yes so that fs_clone starts from the beging insted of continue from fail part

[applmgr@ebsprodapp3 ~]$ adop phase=fs_clone allnodes=no action=db force=yes

[applmgr@ebsprodapp4 ~]$ adop phase=fs_clone allnodes=no action=nodb (approx. 10 Mins on node 2, dmz)

[applmgr@ebsprodpubapp1 ~]$ adop phase=fs_clone allnodes=no action=nodb

[applmgr@ebsprodapp3 ~]$ adop -status

Major difference between prepare phase and fsclone is:

Prepare phase synchronize ADOP patches only, it does not sync patches of{oracle home, oracle_common, Middle ware home(Webtier)/Weblogic patches, JDK/JRE upgrade Patches}

Where as fsclone makes a complete Replica of RFS.

Conclusion: when applying ADOP patch no need to run FS_CLONE, but when applied other than ADOP patches it is mandatory to sync RFS with PFS before starting next prepare phase.

################################################################################################################

################################################################################################################

ADOP Fsclone Expected errors and troubleshoot:

Reported Error 1 for FS_CLONE:

12.2 E-Business Suite Applications DBA Online ADOP Utility Fs_clone Fails With "ERROR: Number of servers are not in sync between Run and Patch Context" [1582473.1]

[applmgr@ebsprodapp3 ~]$ find $ADOP_LOG_HOME/30 -iname FSCloneApplyAppstier*.log

+++++++++++++++++++++++++

configProperty id = Server8

Count for NodeIterator nextNode = 3

ERROR: Managed Server's are not in sync between file system context and DB context

ERROR: Update Moveplan Fail

START: Inside exitClone....

Updating status INCOMPLETE for ApplywlsTechStack

START: Updating status INCOMPLETE for action ApplywlsTechStack

END: Updated status INCOMPLETE for action ApplywlsTechStack

Solution:

Verify Managed Servers List in WebLogic console as per application Context file and delete unwanted/additional MS/Cluster/deployments (SSO) configured in WL Console which not attached to any machines (where machine name is empty)

grep _cluster $CONTEXT_FILE

grep oacore_server $CONTEXT_FILE

grep oa_managed_server $CONTEXT_FILE

grep _server_ports $CONTEXT_FILE

=============================================================

=============================================================

Reported Error 2 for FS_CLONE:

[applmgr@ebsprodapp3 ~]$ find $ADOP_LOG_HOME/30 -iname FSCloneApplyAppstier*.log

Undeploying Application accessgate from targets

=================================================================

Undeploying application accessgate ...

java.lang.IllegalArgumentException: [J2EE Deployment SPI:260134]Unable to identify application for undeploy operation.Failed to undeploy the application with status failed

Current Status of your Deployment:

Deployment command type: undeploy

Deployment State       : failed

Deployment Message     : java.lang.IllegalArgumentException: [J2EE Deployment SPI:260134]Unable to identify application for undeploy operation.

No stack trace available.

Error while deploying application accessgate

Exiting WebLogic Scripting Tool.

<Jul 29, 2022 7:48:57 PM AST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>

.end std out.

.end err out.

Script Executed in 188415 milliseconds, returning status 1

ERROR: Failed to execute /u01/TEST/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adPostDomainCloneCfg.pl

Solution:

Delete accessgate deployment from Weblogic console which was created for PROD SSO

=============================================================

=============================================================

Reported Error 3 for prepare phase:

Adop Prepare Phase Fails While Calling ad_zd.prepare" With Error "ORA-04031: unable to allocate 256 bytes of shared memory" (Doc ID 2043199.1)

SQL> 

show parameter sga

show parameter pga

show parameter recovery

alter system set sga_max_size=16G scope=spfile sid='*';

alter system set sga_target=12G scope=spfile sid='*';

alter system set pga_aggregate_target=6G scope =spfile sid='*';

alter system set db_recovery_file_dest_size=1500G scope =spfile sid='*';

Solutionfor adop prepare:

1.Set sga_target = 6g and pga_aggregate_target = 6g for adop.

2.Restart the database.

3.run adop phase=prepare again.

=============================================================

=============================================================

Reported Error 4 Restoring PFS after flashback when FS has changed fs2->fs1:: Tested and verified, old PFS xml should be restored form fs1_del

Note: in this scenario current RFS is fs1 and before flashback RFS was fs2 and we had fs2-tar-bkp before flashbk/cutover

[applmgr@uatapp01 ~]$ . EBSapps.env run

  E-Business Suite Environment Information

  ----------------------------------------

  RUN File System           : /u01/UAT/fs1/EBSapps/appl

  PATCH File System         : /u01/UAT/fs2/EBSapps/appl

  Non-Editioned File System : /u01/UAT/fs_ne

  DB Host: uatdb01.psau.edu.sa  Service/SID: UAT

  Sourcing the RUN File System ...

[applmgr@uatapp01 ~]$

---# this can be achived by 2 methods, 

1. need to clone PFS from RFS

2. is successful fsclone

---# after flashback restore of DB now untar fs2 also and run autoconfig on DB and apps

[applmgr@uatapp01 ~]$ mv fs2 fs2_del

[applmgr@uatapp01 ~]$ mv fs1 fs1_del

[applmgr@uatapp01 ~]$ tar -xvzf /bkp/fs2_15jan.tar.gz -C /u01/UAT

[applmgr@uatapp01 ~]$ . EBSapps.env run

  E-Business Suite Environment Information

  ----------------------------------------

  File System Type          : SINGLE

  RUN File System           : /u01/UAT/fs2/EBSapps/appl

  PATCH File System         : NOT APPLICABLE

  Non-Editioned File System : /u01/UAT/fs_ne

  DB Host: uatdb01.psau.edu.sa  Service/SID: UAT

  Sourcing the RUN File System ...

[applmgr@uatapp01 ~]$

[applmgr@uatapp01 ~]$ grep s_file_edition $CONTEXT_FILE

         <file_edition_type oa_var="s_file_edition_type">run</file_edition_type>

         <file_edition_name oa_var="s_file_edition_name">fs2</file_edition_name>

[applmgr@uatapp01 ~]$

exec fnd_conc_clone.setup_clean;

[oracle@uatdb01 ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh

---# fnd_oam_context_files table updated with RFS and PFS xml's run adautocfg.sh on Apps Tier

select distinct(PATH),EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') as FILE_SYSTEM from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') in ('run','patch') -- check RFS/PFS context files uploaded to db or not

select distinct(PATH) from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') in ('run','patch');-- check RFS/PFS context files uploaded to db or not

[applmgr@uatapp01 ~]$ adautocfg.sh

AutoConfig completed successfully.

[applmgr@uatapp01 ~]$ 

---# if auto config failed on apps tier then use below and retry autoconfig

$AD_TOP/bin/adconfig.sh contextfile=$CONTEXT_FILE -syncctx

---# then check apps.fnd_oam_context_files run adautocfg.sh on Apps Tier

if PFS xml is missing then upload with API by sourcing RFS and path of xmlfile (i.e db will updated and and PFS xml gets created)

Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone (Doc ID 2090393.1)

Copy fs1 xml file which was run edition before restore and update below value to patch

[applmgr@uatapp01 ~]$ cp /u01/UAT/fs1_del/inst/apps/UAT_uatapp01/appl/admin/UAT_uatapp01.xml /u01/UAT/fs1/inst/apps/UAT_uatapp01/appl/admin/

[applmgr@uatapp01 ~]$ grep s_file_edition /u01/UAT/fs1/inst/apps/UAT_uatapp01/appl/admin/UAT_uatapp01.xml

         <file_edition_type oa_var="s_file_edition_type">patch</file_edition_type>

         <file_edition_name oa_var="s_file_edition_name">fs1</file_edition_name>

[applmgr@uatapp01 ~]$

s[applmgr@uatapp01 ~]$ grep fs2 /u01/UAT/fs1/inst/apps/UAT_uatapp01/appl/admin/UAT_uatapp01.xml

         <OTHER_BASE oa_var="s_other_base">/u01/UAT/fs2</OTHER_BASE>

         <clonestage oa_var="s_clonestage">/u01/UAT/fs2/EBSapps/comn/adopclone_uatapp01</clonestage>

[applmgr@uatapp01 ~]$

[applmgr@uatapp01 ~]$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<Enter Your Full Patch Context XML File Location> logfile=~/patchctxupload.log

[applmgr@uatapp01 ~]$ grep s_file_edition $CONTEXT_FILE RFS and PFS

Source env again and now PFS should be normal

[applmgr@uatapp01 ~]$ . EBSapps.env run

  E-Business Suite Environment Information

  ----------------------------------------

  RUN File System           : /u01/UAT/fs2/EBSapps/appl

  PATCH File System         : /u01/UAT/fs1/EBSapps/appl

  Non-Editioned File System : /u01/UAT/fs_ne

  DB Host: uatdb01.psau.edu.sa  Service/SID: UAT

  Sourcing the RUN File System ...

[applmgr@uatapp01 ~]$

[applmgr@uatapp01 ~]$ time adop phase=fs_clone force=yes

Adop Phase fs_clone is Failing on TxkADOPPreparePhaseSynchronize.pl RC-50204: Error: - ONS Remote Port in Use (Doc ID 2045627.1)

Stop any running BGP of RFS, PFS and kill all user sesion 

[applmgr@uatapp01 ~]$ adstpall.sh apps/apps

[applmgr@uatapp01 ~]$ pkill -u applmgr

---# If fsclone got hang due to any reason then fail corresponding ADOP session ID and retry with option force=yes

select * from ad_adop_session_patches --where ADOP_SESSION_ID='30';--fs_clone hang in RUNNING state 

--update ad_adop_session_patches set status='F' where ADOP_SESSION_ID='30';

=============================================================

=============================================================

Reported Error 5 for CLEANUP in FULL Mode when ran with actualize all:

Oracle SR Engineer advised to delete type 10 and type 88 objects

type#10 are  non existing objects, 

type#88 are orphaned stubs

Note: actualize all and full cleanup are running first time on instance for 19c upgrade. default ORA$BASE Edition also exists in EBS DB

When adop cleanup in full mode is failed with ORA-600 

ORA-00600: internal error code, arguments: [16500], [kqdobu], [D], [2], [2270], [EMPLOYEESUNIONVIEW], [], [], [], [], [], []

-- Type#10=non exixitng objects, type#88=orphaned stubs 

--The ORA-600 [kqlhdlod-bad-base-objn] suggests that a stub row in obj$ has a base obj# that does not correspond to an object. The following query executed as the SYS as SYSDBA can be used to identify orphaned stubs:

-- Identify orphaned stubs: and non exists

select o.obj# from obj$ o where type# = 88 and not exists (select 1 from obj$ oi where oi.obj# = o.dataobj#);

Set flashback point before delete then start db in restrict mode and delete with SYS user

Make sure you delete after cleanup is failed else it will mess if deleted in finalize or other phases

-- check object types (10, 88) and delete queries

select obj#,dataobj#,owner#,name,status,flags,type# from obj$ where name in ('EMPLOYEESUNIONVIEW')-- sys schema 

select count(*) from obj$ where type#=10;

select count(*) from obj$ where type#=88;

--- remove objects (type 10) (Note DB should be started in restrict mode before deleting type 10 and 88 objects)

startup restrict

delete from obj$ where type#=10;

-- Identify orphaned stubs: (type 88)

select o.obj# from obj$ o where type# = 88 and not exists (select 1 from obj$ oi where oi.obj# = o.dataobj#);

-- Remove the orphaned stubs: (approx 15-20Mins)

startup restrict

delete from obj$ o where o.type# = 88 and not exists (select 1 from obj$ oi where oi.obj# = o.dataobj#);

commit;

alter system flush shared_pool;

commit;

shut abort

startup normal

################################################################################################################

################################################################################################################

Other utilities in while running ADOP Phases:

[applmgr@ebsprodapp3 ~]$ adopmon |& tee ~/adopmon_XXX.txt

[applmgr@ebsprodapp3 ~]$ adopmon  ( to capture output use Linux script command or adopmn > a.txt)

[applmgr@ebsprodapp3 ~]$ adctrl

[applmgr@ebsprodapp3 ~]$ adopreports

[applmgr@ebsprodapp3 ~]$

################################################################################################################

################################################################################################################

=============================================================

admsi.pl run Instruction :  (if instructed to run admsi as per patch readme)

R12: AP: How do you run admsi.pl for the Patch Application Assistant? (Doc ID 2591487.1)

SOLUTION: The admsi.pl is used to generate customized installation instructions for a patch. To run it, you can do the following:

perl $AD_TOP/bin/admsi.pl -patch_top=patch path -appspass=apps

[applmgr@ebsprodapp3 ~]$ perl $AD_TOP/bin/admsi.pl -patch_top=/home/applmgr/33777036 -appspass=prod1

Gathering Information..

Logfile for this session is located at admsi.log

Generating installation instructions for patch 33777036..

Updating database....

install_33777036.html generated successfully

install_33777036.txt generated successfully

[applmgr@ebsprodapp3 ~]$ 

[applmgr@ebsprodapp3 ~]$ vi admsi.log

[applmgr@ebsprodapp3 ~]$ vi install_33777036.txt   (for special instructions after admsi for patching)

After Reviewing generated Install_PATCHNUMBER.txt follow -- Apply the patch -- Finishing tasks -- Additional information)

[applmgr@ebsprodapp3 ~]$ 

[applmgr@ebsprodapp3 ~]$ perl $AD_TOP/bin/admsi.pl --done

Gathering Information..

Enter the patch_top : =/home/applmgr/33777036

Enter the APPS user password:

Logfile for this session is located at admsi.log

Error:

Invalid patch_top or this patch may not be applicable for running admsi.pl

See admsi.log for more details.

[applmgr@ebsprodapp3 ~]$

[applmgr@ebsprodapp3 ~]$ perl $AD_TOP/bin/admsi.pl --done -patch_top=/home/applmgr/33777036 -appspass=prod1

Gathering Information..

Logfile for this session is located at admsi.log

Updating database for completion..

Done.

[applmgr@ebsprodapp3 ~]$

################################################################################################################

################################################################################################################

=============================================================

Latest legislation data update (Hrglobal installation):  (need to run as per readme/doc)

by default if we run hrglobal in downtime mode it will set 32 workers by it self, so no need to specify workers

R12.2: How to Install HRMS Legislative Data Using Data Installer and hrglobal.drv (Doc ID 2006776.1)

DataInstall and HRGLOBAL Application: 12.2 Specifics (Doc ID 1469456.1)

Latest Oracle HRMS Legislative Data Patch Available (HR Global / hrglobal) (Doc ID 145837.1)

The command line for the Data Install program is as follows:

java oracle.apps.per.DataInstall apps (password) thin (server):(db port):(SID)

  or

java oracle.apps.per.DataInstall apps (password) thin (server):(db port)/(Service Name)

where:

(password) refers to the APPS schema password

(server) is the database server name or ip address

(db port) is the sql net or database port number default is 1521

(SID) is the database system identifier like PROD

(Service Name) is the service_name in tnsnames.ora connection string

for example:

java oracle.apps.per.DataInstall apps apps thin dbsvr1:1521:proddb

java oracle.apps.per.DataInstall apps apps thin dbsvr1:1521/prod

[apploci@ebsprodapp3 ~]$ adop -status   (Take PFS bkp and then run fsclone)

[apploci@ebsprodapp3 ~]$ tnsping PROD

[apploci@ebsprodapp3 ~]$ java oracle.apps.per.DataInstall apps apps_pswd thin ebsdbhost:1521:PROD     (approx. Time immediate)

  or

[apploci@ebsprodapp3 ~]$ java oracle.apps.per.DataInstall apps apps_pswd thin ebsdbhost:1521/PROD     (approx. Time immediate)

DataInstall for hrglobal

Patch Edition: null

NO PATCH EDITION EXISTS - WILL TRY RUN EDN

Run Edition: V_20220121_1047

+-------------------------------------------------+

|             DataInstall Main Menu               |

+-------------------------------------------------+

1.    Select legislative data to install/upgrade

2.    Select college data to install/upgrade

3.    Select JIT to install/upgrade

4.    Select International HRMS Setup to install

5.    Exit to confirmation menu

Enter your choice : 1

 # Localisation         Product(s)               Leg. Data? Action

-- -------------------  ------------------------ ---------- -------------

 1 Global               Human Resources          Installed

 2 Australia            Human Resources

 3 Australia            Payroll

 4 Belgium              Human Resources

 5 Canada               Human Resources

 6 Canada               Payroll

 7 China                Human Resources

 8 China                Payroll

 9 Denmark              Human Resources

10 Finland              Human Resources

11 France               Human Resources

12 Germany              Human Resources

13 Hong Kong            Human Resources

14 Hong Kong            Payroll

15 Hungary              Human Resources

16 India                Human Resources

17 India                Payroll

18 Ireland              Human Resources

19 Ireland              Payroll

20 Italy                Human Resources

21 Japan                Human Resources

22 Japan                Payroll

23 Korea, Republic of   Human Resources

24 Korea, Republic of   Payroll

25 Kuwait               Human Resources

26 Kuwait               Payroll

27 Mexico               Human Resources

28 Mexico               Payroll

29 Netherlands          Human Resources

30 Netherlands          Payroll

31 New Zealand          Human Resources

32 New Zealand          Payroll

33 Norway               Human Resources

34 Poland               Human Resources

35 Romania              Human Resources

36 Russian Federation   Human Resources

37 Saudi Arabia          Human Resources           Installed

38 Saudi Arabia          Payroll                   Installed

39 Singapore            Human Resources

40 Singapore            Payroll

41 South Africa         Human Resources

42 South Africa         Payroll

43 Spain                Human Resources

44 Sweden               Human Resources

45 United Arab Emirates Human Resources

46 United Arab Emirates Payroll

47 United Kingdom       Human Resources          Installed

48 United Kingdom       Payroll                  Installed

49 United States        Human Resources

50 United States        Payroll

51 United States        US Federal Human Resources

 <Product #><Action> - Change Action

 where <Action> is [I : Install, C : Clear]

 [Return]     - To return to main menu.

Enter your choice (for example 2I) : 37I

Enter your choice (for example 2I) : 38I

37 Saudi Arabia         Human Resources          Installed  Install

38 Saudi Arabia         Payroll                  Installed  Install

 <Product #><Action> - Change Action

 where <Action> is [I : Install, C : Clear]

 [Return]     - To return to main menu.

Enter your choice (for example 2I) : Press enter to go back to main menu

+-------------------------------------------------+

|             DataInstall Main Menu               |

+-------------------------------------------------+

1.    Select legislative data to install/upgrade

2.    Select college data to install/upgrade

3.    Select JIT to install/upgrade

4.    Select International HRMS Setup to install

5.    Exit to confirmation menu

Enter your choice : 5

          DataInstall - Actions confirmation

Do you really wish to exit and save your changes?

      [Y]      - Yes, save then exit

      [N]      - No, don't save but exit

      [Return] - To return to the DataInstall Main Menu

Enter your choice (for example Y) : Y

          DataInstall - Actions summary

          -----------------------------

The following actions will be performed:

Localisation         Product(s)               Leg. Data? Action

-------------------- ------------------------ ---------- -------------

Saudi Arabia         Human Resources          Installed  Install

Saudi Arabia         Payroll                  Installed  Install

Localisation   College Data? Action

-------------- ------------- -------------

United Kingdom

United States

Option                         Data?         Action

------------------------------ ------------- -------------

JIT/Geocode

Legislation                            Action

------------------------------         -------------

ALL  All Legislations

[apploci@ebsprodapp3 appl]$

Optional Instruction for applying hrglobal.drv in Downtime mode:(Number of Parallel Workers (Page 34) --> "N" CPU X 2 to 4 times )

lscpu

SQL> show parameter cpu

adstpall.sh apps/apps

[applmgr@ebsprodapp3 ~]$ adop phase=apply apply_mode=downtime patchtop=$PER_TOP/patch/115 patches=driver:hrglobal.drv options=nocopyportion,nogenerateportion,forceapply   (approx. 20 Mins)

[applmgr@ebsprodapp3 ~]$

[applmgr@ebsprodapp3 ~]$ adop phase=cleanup workers=24

[applmgr@ebsprodapp3 ~]$

Apply the required patches and run fsclone to syncronize:

if all set Run Gather Schema when ran on PROD:

select * from hr_legislation_installations where status='I';-- verify hrglobal run

select * from pay_action_parameters;-- to check threads 

=============================================================

Configure multi-threads enablement for EBS Payroll run:

How Do I Improve Payroll Performance? (Doc ID 219306.1)

Tips for Improving Payroll Performance (Doc ID 1079475.6)

How to Determine the Best Setting for the THREADS Parameter in the Pay_Action_Parameters Table (Doc ID 359354.1)

Quick List of PAY_ACTION_PARAMETERS Values (Doc ID 181304.1)

Verify current/default parameters from payroll run request log file

conn apps/apps_pswd

col PARAMETER_VALUE for a20;

select * from pay_action_parameters;

insert into pay_action_parameters values ('BAL_BUFFER_SIZE','100');

insert into pay_action_parameters values ('EE_BUFFER_SIZE','100');

insert into pay_action_parameters values ('RR_BUFFER_SIZE','100');

insert into pay_action_parameters values ('RRV_BUFFER_SIZE','100');

insert into pay_action_parameters values ('CHUNK SIZE','100');

insert into pay_action_parameters values ('THREADS','24');

commit;

NOTE: 

- as per above ref. docs. threads cal (1.5/2 X no. of process on appstier)

- defined threads are depend on number of standard managers enabled, else slave payroll process will be in pending phase only while payroll run is going on.

################################################################################################################

################################################################################################################

ADOP Tables for node information configured with Single/Multi Apps Node:

 1.1) select * from fnd_nodes; --(only 3:AUTHENTICATION, DBNODE, APPSNODE)

 1.2) select * from ADOP_VALID_NODES; -- (prod apps entries;(Doc ID 2064223.1)

 1.3) select * from ad_nodes_config_status; -- (prod apps entries

 1.4) select * from fnd_oam_context_files;

ADOP Session Status from Database Query:

Query 1:

select ADOP_SESSION_ID,PREPARE_STATUS,APPLY_STATUS,FINALIZE_STATUS,CUTOVER_STATUS,CLEANUP_STATUS,ABORT_STATUS,STATUS,ABANDON_FLAG,NODE_NAME

from AD_ADOP_SESSIONS where ADOP_SESSION_ID='8';

Query 2:

select * from ad_adop_sessions --where ADOP_SESSION_ID='30';

--update ad_adop_sessions set status='C' where ADOP_SESSION_ID='30';

Query 3:

select * from ad_adop_session_patches --where ADOP_SESSION_ID='30';--fs_clone hang in RUNNING state 

--update ad_adop_session_patches set status='F' where ADOP_SESSION_ID='30';

Query 4:

select * from adop_valid_nodes

select * from fnd_oam_context_files; 

Query 5:

SQL> select ad_zd.get_edition_type from dual;

GET_EDITION_TYPE

-----------------------------------------------------------------------------

PATCH

SQL> 

Query 6: uploading missing RFS/PFS Context file entries in fnd_oam_context_files table

SQL> select distinct(PATH),EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') as FILE_SYSTEM from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') in ('run','patch'); -- check RFS/PFS context files uploaded to db or not

. ~/EBSapps.env Patch

$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=$CONTEXT_FILE logfile=~/patchctxupload.log

Query 7:

SQL> select ad_patch.is_patch_applied('R12',-1,20034256) from dual; -- (Doc ID 1963046.1) EXPLICIT=applied, NOT APPLIED = not applied / aborted

Query 8:

SQL> select * from ad_bugs where bug_number in('31745969','31206584','30980446')

################################################################################################################

################################################################################################################

grep s_shared_file_system $CONTEXT_FILE

Thanks for Reading

Regards,

Mohammed Areefuddin.

Suggested Topics :

Linux	DATABASE	RMAN	RAC	EBS
5.7 Install	DATABASE Install		11201 Install	R1226 Install
Open Filer Instl	Active DataGaurd	Rman DataGuard	12102 RAC INSTAL	R1229 M7 Clone
DNS in Linux 7	WIN-LINUX MIGRATE	RMAN DB CLONE	RAC DataGuard	Pluggable DB Clone
lpadmin cups	ASMCMD ASM Migrt	RMAN ASM Migrt	RAC DB Clone	appsutil for DB
	11204 M-Upgrade		RAC - RAC Clone	JDK JRE upgrade
	12102 M-Upgrade		12C-19C RAC UPGRADE	Add EBS Node
	Flashback DB			ADOP in R12.2.9
				12102 EBS UPGRADE
Linux Basics	DATABASE Basics	RMAN Basics	RAC Basic	EBS Basics