my-orcl@Areef DBA, my-orcl@Arif DBA: EBS R12.2.9 Full Clone With RMAN Duplicate m7

Hi, in this topic we learn how to create Full Clone of EBS Database & Application release EBS R12.2.9 on Solaris Sparc Server.

Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1)

Database Initialization Parameters for Oracle E-Business Suite Release 12 (Doc ID 396009.1)

List of Topics (Linux, Database, RAC, EBS)

for cloning EBS with a pluggable databse click here

################################################################################################################

################################################################################################################

OS Check before Prepare:

root@ebsprodapp3:~# cat /etc/release

                            Oracle Solaris 11.4 SPARC

  Copyright (c) 1983, 2020, Oracle and/or its affiliates.  All rights reserved.

                             Assembled 26 March 2020

root@ebsprodapp3:~#

root@ebsprodapp3:~# uname -a
SunOS ebsprodapp3 5.11 11.4.20.4.0 sun4v sparc sun4v

root@ebsprodapp3:~#

root@ebsprodapp3:~# prtconf

System Configuration: Oracle Corporation sun4v

Memory size: 391680 Megabytes

System Peripherals (Software Nodes):

prtconf: devinfo facility not available

root@ebsprodapp3:~#

root@ebsprodapp3:~# pkg info entire

Name: entire

Summary: entire incorporation including Support Repository Update

(Oracle Solaris 11.4.20.4.0).

Description: This package constrains system package versions to the same

build. WARNING: Proper system update and correct package

selection depend on the presence of this incorporation.

Removing this package will result in an unsupported system.

For more information see:

https://support.oracle.com/rs?type=doc&id=2433412.1

Category: Meta Packages/Incorporations

State: Installed

Publisher: solaris

Version: 11.4 (Oracle Solaris 11.4.20.4.0)

Branch: 11.4.20.0.1.4.0

Packaging Date: Thu Mar 26 17:29:32 2020

Last Install Time: Wed Oct 14 20:51:30 2020

Size: 2.52 kB

FMRI: pkg://solaris/entire@11.4-11.4.20.0.1.4.0:20200326T172932Z

root@ebsprodapp3:~#

################################################################################################################

New Clone Procedure to clone from RAC – ASM EBS DB Instance to Non-RAC ASM clone Server:

Source EBS Instance – PROD 

EBS Applications Size -> 63 GB

EBS Database (ORACLE_HOME & DATAFILES) -> 23+750 GB

-	du -sh $ORACLE_HOME/rdbms/audit Before preparing source machine delete audit file --- if required mv & recreate same folder name
-	RMAN old backup delete for controlfile in home
-	Run gather schema on prod before a day of clone
-	cat $ORACLE_HOME/oraInst.loc (create same as in oraInst.loc)
-	create pfile from spfile; --- AHCTEST  sp/pfile Backup or copy spfile
-	Before deleting Target Instance (TestM7 & UPGTEST) take backup of init, spfile, orapwSID, (listener, tns, sqlnet files) --- if required
-	lsnrctl status SIDNAME –-- cat $TNS_ADMIN/listener.ora to check clone port number to recreate
-	create pfile='/export/home/oramgr/initAHCTEST12042021.ora' from spfile;
-	cwallet, ewallet –Backup before delete Instance if ssl was configured
-                                                                          	
-	inform corresponding team/company(AJ) to backup their scripts before drop
 

Target servers existing DB port if need to re use::

grep PORT $TNS_ADMIN/listener.ora --current db port configured for Clone/Target DB server

grep login $CONTEXT_FILE --current apps port configured for Clone/Target DB server

Regular logs files to delete from PROD Run FS -- not for cloning
. If files not deleted, then java heap size error during fsclone
. Do not delete or rename the current file or logs folder directly

GRID Listener (alert & trace)logs:
du -sh /u01/app/grid/diag/tnslsnr/$HOSTNAME/*

RDBMS Generated Logs::
du -sh /u01/app/oracle/diag/rdbms/prod/$ORACLE_SID
du -sh $ORACLE_HOME/dbs  #verified,tested it is safe to remove core_* folders
du -sh $ORACLE_HOME/rdbms/audit

Application Generated Logs
du -sh $PATCH_TOP $PATCH_TOP/*
du -sh $EBS_DOMAIN_HOME/servers/*   
du -sh $IAS_ORACLE_HOME/instances/* (or)
du -sh $IAS_ORACLE_HOME/instances/EBS_web_OHS*/diagnostics/logs/OHS/EBS_web/

if log files in EBS domain server folder (Admin, forms & oacore_server) have reached more than 10GB the delete all acess0, log0 & out0 files: (alternately carefully delete first command by *000* for most of all logs at once)

cd $EBS_DOMAIN_HOME/servers/AdminServer/logs/
rm access.log0* AdminServer.log0* EBS_domain.log0*

cd $EBS_DOMAIN_HOME/servers/forms_server1/logs/
rm access.log0* forms_server1.log0* forms_server1.out0*

cd $EBS_DOMAIN_HOME/servers/oacore_server1/logs
rm access.log0* oacore_server1.out0* oacore_server1.log0*

cd $EBS_DOMAIN_HOME/servers/oacore_server2/logs
rm access.log0* oacore_server2.out0* oacore_server2.log0*

cd $EBS_DOMAIN_HOME/servers/oacore_server3/logs
rm access.log0* oacore_server3.out0* oacore_server3.log0*

 Monthly once delete only conc request logs folder but not from output folder:
du -sh $APPLCSF/log
find $APPLCSF/log -type f | wc -l
find $APPLCSF/log | wc -l
cd $APPLCSF/log --file name ---find1rm.sh---
ls |wc -l
ls -ltr $APPLCSF/log/find1rm.sh
find . -type f -name "*.req" -mtime +3 -exec rm {} \;
after this run fsclone to synchronize with Patch FS


Monthly once delete OS ExaWatcher Archives from root partition on Super Cluster Machine:
only on ALL DB Serevers:
du -sh /opt/oracle.ExaWatcher/archive/*
cd /opt/oracle.ExaWatcher/archive/RDSinfo.ExaWatcher

find . -type f -name "*.bz2" -mtime +1 |wc -l
ls |wc -l
find . -type f -name "*.bz2" -mtime +1 -exec rm {} \;
rm -rf *.bz2
ls -ltrh

Worked below db drop option, instead physically deleting files from ASM/OS
this deletes DB's C-R-D files plus DB's SPFILE & ARCHIVES logs except current i.e. 2 for th#1 & 3 for th#2))
select NAME from v$controlfile;
select file_name from dba_data_files;
select name from V$TEMPFILE;
select member from v$logfile;
archive log list;

select group#,thread#,members,status from v$log;
select THREAD#, STATUS, ENABLED from v$thread;
select group# from V$log where THREAD#=2;

col DEST_NAME for a25;
col DESTINATION for a50;
set linesize 200 pagesize 400;
select dest_name,status,destination from v$ARCHIVE_DEST;


Method-1 (this drops database plus archivelog files from RECO and spfile also)
-bash-4.1$ df -h
RMAN> select name,OPEN_MODE from v$database;
RMAN> STARTUP FORCE MOUNT
RMAN> ALTER SYSTEM ENABLE RESTRICTED SESSION;
RMAN> select INST_ID,value from gv$diag_info where name='Diag Trace';
RMAN> delete noprompt archivelog alX; (no need if executing below command)
RMAN> DROP DATABASE INCLUDING BACKUPS NOPROMPT; ( worked fine except current i.e. 2 for th#1 & 3 for th#2))
-bash-4.1$ df -h

srvctl config database
srvctl remove database -d TEST  (before duplicate remove any registered database)


Method-2 (this drops only controlfile and data files from DATA disk)
SQL> shut immediate
SQL> startup mount exclusive restrict
SQL> drop database;


Other useful OS Commands:
netstat -tulpn
netstat -tulpn |grep LISTEN
netstat -tulpn |grep LISTEN |grep /java
netstat -tulpn |grep LISTEN |grep /httpd.worker
netstat -tulpn |grep LISTEN |grep /tnslsnr

[root@rac2 ~]# history | grep key_word_to_find
[root@rac2 ~]# screen -ls
[root@rac2 ~]#
There is a screen on:
        29206.pts-0.rac2    (Attached)
1 Socket in /var/run/screen/S-root. 
[root@rac2 ~]# screen -d -r 29206.pts-0.rac2


Enable tmux:
[root@rac2 ~]# tmux -f ~/.tmux.conf (force load all customizing in tmux session)
[root@rac2 ~]# tmux ls
0: 1 windows (created Tue Dec 14 06:27:17 2021) [200x57] (attached)
[root@rac2 ~]# tmux at -t 0
[root@rac2 ~]#

[root@rac2 ~]# unzip '*.zip'
[root@rac2 ~]# unzip abc.zip -d /u02

OS Tar Commands:
root@ebsprodapp4:~# cat /apptarbkp
#/usr/bin/tar Ecf - /u01/AHC | gzip -3 - > /mnt1/BKP_DIR/APPS/AHC.tar.gz
/usr/bin/tar Ecf - /u01/AHC | gzip -3 - > /u01/BKP-DAILY/AHC.tar.gz
root@ebsprodapp4:~#

tar Ecf - /u01/AHC/fs2/EBSapps <--do not execute tar cmd like this. As it will create all folders as in path /u01/AHC/fs2 inside when you untar at Targeted Clone Server. Go to the path where EBSapps folder in Run File and use below cmd only.

root@ebsprodapp4:/u01/AHC/fs2# tar Ecf - EBSapps | gzip -3 - > /u01/BKP-DAILY/EBSapps_15022021.tar.gz (approx. 1Hr 40+ Mins & wknd 1hr)

after SCP (approx. …5 Mins in M7 100mb/sec) for untar application:
root@ebsprodapp4:~# gunzip EBSapps_20.tar.gz (approx. 11 Mins M7 wknd)
root@ebsprodapp4:~# time tar -xvf EBSapps_20.tar EBSapps (approx. 7 Mins)

Regular Command tar gunzip:
tar -C /u01/AHC/ -cvzf /u01nfshare/backup_prod/10082017/EBSapps.tar.gz EBSapps (to tar in different loc. & Exclude path while uncompress)

root@ebsprodapp4:/u01/AHC/fs2# tar -cvzf /u01/AHC/tar_bkp_EBSapps_fs2/EBSapps_fs2_10042021.tar.gz EBSapps
ps -eo pid,lstart,cmd|grep -i 'sqlplus\|adop\|tar.gz\|zip' (to check tar session start time)

Compress dumpfile with tar (E) if file size is more than 10GB in Solaris:
root@ebstestdb:/u01/app/oramgr/expdp# tar -cvzf AJ_LIVE_26062021.dmp.tar.gz AJ_LIVE_26062021.dmp
Compressing 'AJ_LIVE_26062021.dmp.tar.gz' with '/usr/bin/gzip'...
tar: AJ_LIVE_26062021.dmp too large to archive.  Use E function modifier.
root@ebstestdb:/u01/app/oramgr/expdp# tar -Ecvzf AJ_LIVE_26062021.dmp.tar.gz AJ_LIVE_26062021.dmp



scp EBSapps_fs2_10042021.tar.gz root@192.168.4.15:/u01/OCI:

root@ebstestapp:/u01/OCI# time tar -xvzf EBSapps_fs2_10042021.tar.gz -C /u01/OCI/fs2

Start the Application and database Pre clone and tar files creation:
 
Solaris Commands:
Application tar backup with gunzip in Solaris 11:
Start Application Tar Backup for EBSapps directory:

Source Database Size including temp and Redo logs:
select 'DATA_n_INDEX: ' || sum(bytes)/1024/1024/1024 || ' GBytes' DATABASE_SIZE from dba_data_files union
select 'TEMP: ' || sum(bytes)/1024/1024/1024 || ' GBytes' from dba_temp_files union
select 'REDO LOGS: ' || sum(bytes)/1024/1024/1024 || ' GBytes' from v$log union
select 'CONTROLFILE: ' || sum(FILE_SIZE_BLKS*BLOCK_SIZE)/1024/1024 || ' MBytes' from v$controlfile;
-- select CPU_CORE_COUNT_CURRENT,CPU_COUNT_CURRENT from V$LICENSE; -- top -> 1


applahc@ebsprodapp4:~$ cd $ADMIN_SCRIPTS_HOME
applahc@ebsprodapp4:~$ ls -ltrh $COMMON_TOP/*/FMW/FMW_Home.jar
applahc@ebsprodapp4:~$ ls -ltrh $INST_TOP/logs/appl/admin/log/adst*.log (verify when last restarted apps)
applahc@ebsprodapp4:~$ ls -ltrh $INST_TOP/admin/log/clone/StageAppsTier_*.log (verify when last ran preclone)
applahc@ebsprodapp4:scripts$ time perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier  (approx. 35mins)
select to_char(RESETLOGS_TIME, 'DD-MON-YYYY, hh24:mi:ss') from v$database;--(when database was cloned)

oramgr@ebsproddb1:~$ cd $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME
oramgr@ebsproddb1:~$
oramgr@ebsproddb1:~$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adpreclone.pl dbTier (approx. 2mins)
root@ebsproddb1:/u01/app/oramgr/product/12.1.0# tar -cvzf 12cdb1_10042021.tar.gz dbhome_1   (approx. 30mins)

Compress dumpfile with tar (E) if file size is more than 10GB in Solaris:
#/$ tar -C /home/oracle/expdp/ -cvzf /home/oracle/expdp/AJ_LIVE_27092021.dmp.tar.gz AJ_LIVE_27092021.dmp (to tar in different loc. & Exclude path /home/oracle/expdp while uncompress)

Untar to different location with in server:

tar -C /u01/AHC/fs2 -cvzf /u02/CLONE_FILES/EBSapps_fs2_10042021.tar.gz EBSapps (to tar in different loc. & Exclude path while uncompress)

scp 12cdb1.tar 192.0.100.63:/oradata/AHC    -- copy oracle home tar to target.
root@ebstestapp:/u02/CLONE_FILES# time tar -xvzf EBSapps_fs2_10042021.tar.gz -C /u01/AHCTEST/fs2     (approx. 3hrs)(approx. 35mins)

While Cloning on a newly Installed Linux Server:

Section-1 User Creation in Solaris 11.4
For a new ASM Instance create new user with ASM groups
root@ebstestdb:~# cat /etc/passwd
root@ebstestdb:~# 
root@ebstestdb:~# mkdir /export/home/oraar
root@ebstestdb:~# mkdir -p /u01/app/oraar/product/12.1.0.2
root@ebstestdb:~# touch /export/home/oraar/.profile  <-- cp /export/home/oramgr/.profile /export/home/oraar

useradd -u 1002 -g oinstall -G dba,asmdba,asmadmin -s /usr/bin/sh oraar <-- This will not execute 

useradd -u 1002 -g oinstall -G dba,asmdba,asmadmin oraar <-- for a New User
useradd -g dba applar <-- for Application User

usermod -s /usr/bin/sh oraar <--orardate bash of user
usermod -g oinstall -G dba,asmdba,asmoper,racoper oraar <-- for update groups of an existing user

passwd oraar  <-- update password oraar12

root@ebstestdb:~# 
chown -R oraar:oinstall /export/home/oraar/
chown oraar:oinstall /export/home/oraar/.profile
chown -R oraar:oinstall /u01/app/oraar
 
################################################################################################################
################################################################################################################

Section-2 Initial Setup for New Installed Linux Server
While configuring clone on new Server make sure all server checks set Acc. run all required prechecks:

After Linux Admin had hand over the newly created server, verify on yourself ALL basic server checks like 
1. firewall/Selinux
2. Network Setup/Internet/DNS
3. Global Inventory Setup
4. RDA Reports Status
5. Other Basic Server Checks if required
6. Yum Installation for Required Preinstallations:
-. Finally re-run again RDA Report to verify missing settings

Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)
Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)

1. Firewall disable in Linux 7,6: (RC-40201: Unable to connect to Database for client connections)
firewall-cmd --list-all
systemctl status firewalld  <- in Linux 7,8
systemctl stop firewalld
systemctl disable firewalld

systemctl status avahi-daemon
systemctl stop avahi-daemon
systemctl disable avahi-daemon

ss -tulpn 
netstat -tulpn |grep -i listen
netstat -tulpn |grep -i vnc
nc -zv uatapp01 8015
nc -zv uatdb01 1521
systemctl start firewalld
firewall-cmd --permanent --add-port={8010/tcp,7011/tcp,7021/tcp}
firewall-cmd --permanent --remove-port={8010/tcp,7011/tcp,7021/tcp}
firewall-cmd --add-port=5901/tcp --perm  (default vnc port 5900+N)
firewall-cmd --reload
firewall-cmd --list-all


service iptables status  <- in Linux 6
service iptables stop
chkconfig --list iptables
chkconfig iptables off

[root@rac1 ~]# sestatus
[root@rac1 ~]# grep SELINUX /etc/selinux/config
[root@rac1 ~]# grep SELINUX /etc/sysconfig/selinux
SELINUX=disabled

2. Network Setup/Internet/DNS:
# cat /etc/hosts
# ping with IP and hostname b/w DB,APPS, Source,Target
# ping www.google.com 
# ping yum.oracle.com
# nslookup newhost (check if Linux Admin added new server to DNS Server)
# tnsping newserver (DB port test from other host to new server if available)


3. Create Global Inventory Folders:
[root@rac1 ~]# cat /etc/oraInst.loc
#define global inventory
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[root@rac1 ~]# mkdir -p /u01/app/oraInventory
[root@rac1 ~]# chown -R oracle:oinstall /u01/app/oraInventory
[root@rac1 ~]# chmod -R 775 /u01/app/oraInventory

4. RDA Reports Status (Option 25 for EBS R12.2)
Remote Diagnostic Agent (RDA) - Getting Started (Doc ID 314422.1)
How To Generate HCVE Report - Step By Step Instructions (Doc ID 2324096.1)
[oracle/applmgr@rac1 ~]$ unzip p21769913_204201020_Linux-x86-64.zip
[oracle/applmgr@rac1 ~]$ cd rda
[oracle/applmgr@rac1 rda]$ ./rda.sh -T hcve

5. Other Basic Server Checks if required before/after yum (Swap, Storage)
# top   #option-1 for core check
# free -hw    #memory check
# grep Huge /proc/meminfo
# vi /etc/security/limits.d/oracle-ebs-server-R12-preinstall.conf
# sysctl -p
# grep kernel.sem /etc/sysctl.conf (or) ipcs -ls

# grep SwapTotal /proc/meminfo
SwapTotal:      30716272 kB

# grep module /usr/share/Modules/init/bash   (moudule error in new server) 
# df -h
# cat /etc/fstab

# dbcli  list-dbhomes  -for list of RDBMS database home on server


6. Yum Installation for Required Preinstallations:
Oracle Linux: Yum Mirror Configuration Error: 'Repository "repo_name" is listed more than once in the configuration' When Setting Up A Client (Doc ID 2164184.1)

Backup files before yum updates:
# cp /etc/sysctl.conf /etc/sysctl.conf_bkp
# cp /etc/sysconfig/network /etc/sysconfig/network_bkp
# cp /etc/resolv.conf /etc/resolv.conf_bkp
# cp /etc/security/limits.conf /etc/security/limits.conf_bkp


No package oracle-ebs-server-R12-preinstall available.
# ping yum.oracle.com
# cd /etc/yum.repos.d/
# grep latest /etc/yum.repos.d/*
# grep addon /etc/yum.repos.d/*
# ls -ltrh /etc/yum.repos.d/*.repo  (if there is more than 1 *.repo file the rename to bkp and try again)
# mv /etc/yum.repos.d/oracle-linux-ol7.repo /etc/yum.repos.d/oracle-linux-ol7.repo_bkp
# mv /etc/yum.repos.d/public-yum-ol7.repo /etc/yum.repos.d/public-yum-ol7.repo_bkp

# wget https://yum.oracle.com/public-yum-ol7.repo
- enable as shown in EBS Installation doc (Doc ID 1330701.1)
ol6_latest
ol6_addons
ol6_UEK_latest (not available in ol7.repo)


# yum search oracle-ebs oracle-rdbms oracle-database  (for EBS-R12, 11g,12c,18c,19c,21c) 
# yum search openmotif21 xorg-x11-libs-compat

# yum -y install oracle-ebs-server-R12-preinstall
# yum -y install openmotif21
# yum -y install xorg-x11-libs-compat


# rpm -qa oracle-ebs*
oracle-ebs-server-R12-preinstall-1.0-5.el7.x86_64
# rpm -qa openmotif21
openmotif21-2.1.30-11.el7.i686
# rpm -qa xorg-x11-libs-compat
xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386

-. Finally re-run again RDA Report to verify missing settings
 

Untar oracle binaries:

Untar to different location with in server:
tar -C /u01/AHC/ -cvzf /u01nfshare/backup_prod/10082017/apps.tar.gz apps (to tar in different loc. & Exclude path while uncompress)

tar -xvzf apps.tar.gz -C /u01/CLONE (to untar in different loc.)

root@ebstestdb:~# mv /u01/app/oratest/product/12.1.0.2/dbhome_1 /u01/app/oratest/product/12.1.0.2/dbhome_1_bkp
root@ebstestdb:~# tar -xvzf 12cdb1.tar -C /u01/app/oratest/product/12.1.0.2/
root@ebstestdb:~# chown -R oratest:dba dbhome_1
 

set user profile

VI Editor Command:
###--- a  append text, after the cursor position
###--- o  open new line below cursor, enter insert mode
###--- yyp -- to copy current line and paste just after.
###--- j and k move down and up one line, so 10j and 10k move down and up ten lines. You can repeat any motion by putting a number before it.
###--- dd is to delete one line, so 10dd deletes up to 10 line below from cursor.

root@ebstestdb:~# su - /oramgr/
root@ebstestdb:~# vi .profile
### user oramgr settings with $ORACLE_HOME
#export ORACLE_UNQNAME=AHCTEST
export ORACLE_SID=AHCTEST
export ORACLE_BASE=/u01/app/oramgr
export ORACLE_HOME=$ORACLE_BASE/product/12.1.0/dbhome_1
export PATH=/usr/local/bin:$ORACLE_HOME/bin:$ORACLE_HOME/perl/bin:$ORACLE_HOME/jdk/jre/bin:/usr/ccs/bin:/usr/sbin:/usr/bin:$ORACLE_HOME/OPatch:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/dt/lib:/usr/openwin/lib:$ORACLE_HOME/ctx/lib:$ORACLE_HOME/lib32
export LIBPATH=$ORACLE_HOME/lib:/usr/dt/lib:/usr/openwin/lib:$ORACLE_HOME/ctx/lib:$ORACLE_HOME/lib32
export PERL5LIB=$ORACLE_HOME/perl/lib:$ORACLE_HOME/perl/lib/site_perl:$ORACLE_HOME/appsutil/perl
export ADPERLPRG=$ORACLE_HOME/perl/bin/perl
export ORA_NLS10=$ORACLE_HOME/nls/data/9idata
#. /u01/app/oramgr/product/12.1.0.2/dbhome_1/AHCTEST_ebstestdb.env
echo 'Sourcing-------------->>>>> NPROD1_hq01-nebsdb01.env'


Save the profile options and relogin with new parameters:
su – oramgr
cat .profile

Relink RDBMS Binaries:

	
cd $ORACLE_HOME/bin
pwd
relink all

cd $ORACLE_HOME/rdbms/lib
pwd
make -f ins_rdbms.mk rac_off

cd $ORACLE_HOME/rdbms/lib
pwd
make -f ins_rdbms.mk ioracle
pwd

Step - 1  relink all	
 
Step - 2  make rac_off

Step -  3  make ioracle

if ORACLE_HOME is not
generated from Source, then check nls & perl)

echo $ORA_NLS10

/u01/app/oramgr/product/12.1.0.2/dbhome_1/nls/data/9idata

cd /u01/app/oramgr/product/12.1.0.2/dbhome_1/nls/data/9idata

Interoperability Notes EBS Release 12.2 with Database 12cR1 (12.1.0.1) (Doc ID 1959604.1)

(to
create 9idata check any interoperability notes (Doc ID 1612017.1))

perl $ORACLE_HOME/nls/data/old/cr9idata.pl

which perl

echo $PERL5LIB

perl -v

Perl version should be same as version in Source. 5.14 (when
EBS_DB.env is sourced)

check perl –v before starting dbTechStack….the version should be 5.14 and which perl should point to perl in oracle_home….

Running Rapid Clone with command:

tar -cvzf $ORACLE_HOME/perl_source.tar.gz $ORACLE_HOME/perl (Worked)

Running:

Beginning rdbms home Apply - Thu Sep 10 13:30:56 2020

/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/bin/../jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=true -Doracle.installer.oui_loc=/u01/app/oramgr/product/12.1.0.2/dbhome_1/oui -classpath /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/xmlparserv2.jar:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/ojdbc6.jar:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/java:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/oui/OraInstaller.jar:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/oui/ewt3.jar:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/oui/share.jar:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/oui/srvm.jar:/u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/jlib/ojmisc.jar   oracle.apps.ad.clone.ApplyDBTechStack -e /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/AHCTEST_ebstestdb.xml -stage /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone   -showProgress

APPS Password : Log file located at /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/log/AHCTEST_ebstestdb/ApplyDBTechStack_09101330.log

  |      0% completed       RC-00110: Fatal: Error occurred while
relinking of ApplyDBTechStack

ERROR while running Apply...

Thu Sep 10 13:31:43 2020

 ERROR: Failed to execute /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/clone/bin/adclone.pl

 Please check logfile.

oramgr@ebstestdb: ls -ltr $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/*  (ohclone.log)

oramgr@ebstestdb:

Error Reported 1: with RC-00110
To avoid Above Error do below OR Test by doing same as APPS Inventory:
Add the below oraInst.loc file in dbhome and Create Directory “oraInventory” in ORACLE_HOME

vi $ORACLE_HOME/oraInst.loc
inventory_loc=/u01/app/oramgr/product/12.1.0.2/dbhome_1/oraInventory
inst_group=oinstall

cd $ORACLE_HOME
ls -ltr $ORACLE_HOME/ora*

mkdir -p $ORACLE_HOME/oraInventory
chmod -R 777 $ORACLE_HOME/oraInventory/
ls -ltr $ORACLE_HOME/ora*
pwd

Error Reported 2: with RC-00110 (jdk and jre versions mis match)
[oracle@uatdb01 ~]$ $ORACLE_HOME/appsutil/jre/bin/java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
[oracle@uatdb01 ~]$ $ORACLE_HOME/jdk/jre/bin/java -version
java version "1.6.0_75"
Java(TM) SE Runtime Environment (build 1.6.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.75-b01, mixed mode)
[oracle@uatdb01 ~]$

[oracle@uatdb01 ~]$ du -sh $ORACLE_HOME/appsutil/jre
132M    jre
[oracle@uatdb01 ~]$ du -sh $ORACLE_HOME/jdk/jre
95M     jre_old/
[oracle@uatdb01 ~]$
[oracle@uatdb01 ~]$ mv $ORACLE_HOME/jdk/jre $ORACLE_HOME/jdk/jre_old
[oracle@uatdb01 ~]$ cp -r $ORACLE_HOME/appsutil/jre/ $ORACLE_HOME/jdk/
[oracle@uatdb01 ~]$


WARNING: RC-50013: Fatal: Instantiate driver did not complete successfully.

tailf $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ohclone.log
SEVERE:Abnormal program termination. An internal error has occured. Please provide the following files to Oracle Support :

"/u01/app/oracle/product/12.1.0/dbhome_1/oraInventory_old/logs/oraInstall2022-11-20_11-06-56AM.err"
"/u01/app/oracle/product/12.1.0/dbhome_1/oraInventory_old/logs/oraInstall2022-11-20_11-06-56AM.out"
"/u01/app/oracle/product/12.1.0/dbhome_1/oraInventory_old/logs/oraInstall2022-11-20_11-06-56AM.log"

ERRORCODE = 255 ERRORCODE_END

[root@uatdb01 ~]# cat /u01/app/oracle/product/12.1.0/dbhome_1/oraInventory_old/logs/oraInstall2022-11-20_11-06-56AM.err
Exception java.lang.UnsupportedClassVersionError: oracle/sysman/ccr/collector/install/InvalidValueException : Unsupported major.minor version 51.0 occurred..
java.lang.UnsupportedClassVersionError: oracle/sysman/ccr/collector/install/InvalidValueException : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at oracle.sysman.oii.oiif.oiifw.OiifwOCMInfoWCDE.<init>(OiifwOCMInfoWCDE.java:103)
        at oracle.sysman.oii.oiic.OiicCloneSession.getIterElements(OiicCloneSession.java:1133)
        at oracle.sysman.oii.oiic.OiicPullSession.initialize(OiicPullSession.java:1423)
        at oracle.sysman.oii.oiic.OiicSessionWrapper.initialize(OiicSessionWrapper.java:784)
        at oracle.sysman.oii.oiic.OiicSessionWrapper.initialize(OiicSessionWrapper.java:768)
        at oracle.sysman.oii.oiic.OiicInstaller.run(OiicInstaller.java:568)
        at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:969)
        at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:906)
[root@uatdb01 ~]# 
[root@uatdb01 ~]#


===========================================================
 
For the below need to try wirh Global Inventory remove home if already exist,
Try next time with attach and detach command, anyways above will also work:
[root@rac1 ~]$
[root@rac1 ~]$ chown -R oraoci:oinstall /u01/app/oraInventory  ß if required
[oraoci@rac1 ~]$ cat /var/opt/oracle/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[oraoci@rac1 ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2021, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.2.0.7.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraDB19Home1" LOC="/u01/app/oraoci/product/19.3.0.0/dbhome_1" TYPE="O" IDX="1"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
oraoci@rac1:~$
[oraoci@rac1 bin]$
[oraoci@rac1 bin]$ $ORACLE_HOME/oui/bin/runInstaller -silent -attachHome -invPtrLoc /var/opt/oracle/oraInst.loc ORACLE_HOME="/u01/app/oraoci/product/19.3.0.0/dbhome_1" ORACLE_HOME_NAME="OraGI19Home1"
[oraoci@rac1 ~]$
[oraoci@rac1 bin]$ $ORACLE_HOME/oui/bin/runInstaller -detachHome /u01/app/oraoci/product/19.3.0.0/dbhome_1 -invPtrLoc /var/opt/oracle/oraInst.loc
[oraoci@rac1 bin]$
[oraoci@rac1 ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
[oraoci@rac1 ~]$

opatch lsinventory 
ß(OPatch failed with error code 73)

Run Perl Script for
Database Tier Configuration:

Correct bash setting to display username click here 
[oratest@ebstestdb ~]$ ps -ef | grep -i /oramgr/
[oratest@ebstestdb ~]$ ps -ef | grep tns
[oratest@ebstestdb ~]$ cd $ORACLE_HOME/appsutil/clone/bin

[oratest@ebstestdb bin]$ set  <- crosscheck no parameter should set
[oratest@ebstestdb bin]$ 

Make sure you ran 3 relink commands if source home is from RAC DB:

NOTE: If intended to use grid port 1521 then Verify 1521 port is free (grid listener)
(if you got any error 1521 port already in use then check grid listener port and stop grid listener and retry dbTechStack)
[oratest@ebstestdb bin]$ perl adcfgclone.pl dbTechStack (approx. 3Mins)

   Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle Applications Rapid Clone
                                 Version 12.0.0
                      adcfgclone Version 120.31.12010000.12

Enter the APPS password :

Running:
/oradata/AHC/dbhome_1/appsutil/clone/bin/../jre/bin/java -Xmx600M -cp /oradata/AHC/dbhome_1/appsutil/clone/jlib/java:/or
adata/AHC/dbhome_1/appsutil/clone/jlib/xmlparserv2.jar:/oradata/AHC/dbhome_1/appsutil/clone/jlib/ojdbc6.jar oracle.apps.
ad.context.CloneContext -e /oradata/AHC/dbhome_1/appsutil/clone/bin/../context/db/CTXORIG.xml -validate -pairsfile /tmp/
adpairsfile_17409.lst -stage /oradata/AHC/dbhome_1/appsutil/clone -dbTechStack 2> /tmp/adcfgclone_17409.err; echo $? > /
tmp/adcfgclone_17409.res

Log file located at /oradata/AHC/dbhome_1/appsutil/clone/bin/CloneContext_0222110046.log

Provide the values required for creation of the new Database Context file.

Target System Hostname (virtual or normal) [testdb] :

Target Instance is RAC (y/n) [y] : n

Target System Database SID : AHCTEST

Target System Base Directory : /u01/app/oramgr

Role separation is supported y/n [n] ? :
Target System utl_file_dir Directory List : /usr/tmp

Number of DATA_TOP's on the Target System [3] : 1

Target System DATA_TOP Directory 1 : +DATA (init parameter db_file_name_convert)

Target System RDBMS ORACLE_HOME Directory [/oradata/AHC/db/tech_st/12.1.0] : /u01/app/oramgr/product/12.1.0.2/dbhome_1

Do you want to preserve the Display [0.0] (y/n)  : n

Target System Display [testdb:0.0] :

Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 1

Checking the port pool 1
done: Port Pool 1 is free
Report file located at /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/temp/portpool.lst
The new database context file has been created :
. . . . .
. . . . .
APPS Password : Log file located at /u01/app/oramgr/product/12.1.0.2/dbhome_1/appsutil/log/AHCTEST_ebstestdb/ApplyDBTechStack_09280945.log
  /      0% completed

Completed Apply...
Mon Sep 28 09:47:36 2020

Starting database listener for AHCTEST:
Running: 
. . . . . . 
. . . . . . 
. . . . . . 
Running ETCC to check status of DB technology patches...


 +===============================================================+
 |    Copyright (c) 2005, 2020 Oracle and/or its affiliates.     |
 |                     All rights reserved.                      |
 |             Oracle E-Business Suite Release 12.2              |
 |          Database EBS Technology Codelevel Checker            |
 +===============================================================+

Validating context file: /u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/AHCOCI11_ebstestdb.xml

Using context file from command line argument:
/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/AHCOCI11_ebstestdb.xml


Starting Database EBS Technology Codelevel Checker, Version 120.83
Sat Apr 10 12:21:44 +03 2021
Log file for this session : /u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/clone/bin/log/checkDBpatch_8779.log

Identifying database release.
Database release set to 12.1.0.2.

Connecting to database.

+-----------------------------------------------------------------------------+
[WARNING] DB-ETCC: Could not connect to database, so unable to check:
  - Whether database is in READ-WRITE mode.
  - Existence of table needed to store DB-ETCC results.
  - Enablement of database In-Memory option.
    If this feature is enabled, additional fixes need to be verified.

Resolve the database connectivity issue, and then rerun DB-ETCC.
+-----------------------------------------------------------------------------+

Running on an Engineered System.

Bugfix file /u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/etcc/db/onprem/txk_R1220_DB_base_bugs.xml : 120.0.12020000.58
This file will be used for identifying missing bugfixes.

Mapping file /u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/etcc/db/onprem/txk_R1220_DB_mappings.xml : 120.0.12020000.33
This file will be used for mapping bugfixes to patches.

Checking Bugfix XML file for 12.1.0.2_BP

Obtained list of bugfixes to be applied and the list to be rolled back.
Now checking Database ORACLE_HOME.

The opatch utility is version 12.2.0.1.24.
DB-ETCC is compatible with this opatch version.

Found patch records in the inventory.

Checking Mapping XML file for 12.1.0.2.210119ProactiveBP

All the required one-off bugfixes are present in Database ORACLE_HOME.

[WARNING] Database not available. DB-ETCC results cannot be stored.

Finished checking fixes for Oracle Database: Sat Apr 10 12:22:10 +03 2021

Log file for this session: /u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/clone/bin/log/checkDBpatch_8779.log

============================================================================
[oratest@ebstestdb ~]$ ps -ef | grep tns
[oratest@ebstestdb ~]$ ps -ef |grep -i oratest|wc -l
[oratest@ebstestdb ~]$ lsnrctl status $ORACLE_SID   <- set Profile with new env and recheck

Check the log file……it
should show …ApplyDBTechStack Completed Successfully

Once this is done……immediately edit the .profile of /oramgr/ user……

Comment all the values which we had set before starting dbtechstack…..and add in the profile the .env file generated above after dbtechstack completed successfully…like below……

vi ~/.profile

. /u01/app/oratest/product/12.1.0.2/dbhome_1/AHCTEST_ebstestdb.env

su – oratest

lsnrctl status $ORACLE_SID  <- Now listener Status should be UP and
running.

Verify OraInventory:

[oratest@ebstestdb ~]$ $ORACLE_HOME/OPatch/opatch version
[oratest@ebstestdb ~]$ $ORACLE_HOME/OPatch/opatch lsinventory |more
[oratest@ebstestdb ~]$ $ORACLE_HOME/OPatch/opatch lsinventory

which opatch  ß if not, update PATH in env created add $ORACLE_HOME/OPatch

opatch lsinventory

Check if sqlplus / as
sysdba is connecting to an idle instance or not…….

Modify the init file for
the target clone server

Delete old AHCTEST CRD File
from ASM, so AHCTEST folder also gets Deleted

Copy backup of spfile/pfile/password
file from oracle home

cd $ORACLE_HOME/dbs
cp ~/spfileAHCTEST.ora $ORACLE_HOME/dbs
cp ~/initAHCTEST.ora $ORACLE_HOME/dbs
cp ~/orapwAHCTEST $ORACLE_HOME/dbs
 
orapwd file=$ORACLE_HOME/dbs/orapwAHCTEST password=sys4nwr entries=30 force=y    --- the password of sys must be same as the sys user of prod db

OR recreate new spfile by modifying created pfile as below:

[oratest@ebstestdb ~]$ cd $ORACLE_HOME/dbs

[oratest@ebstestdb dbs]$ cp $ORACLE_HOME/dbs/init${ORACLE_SID}.ora $ORACLE_HOME/dbs/init${ORACLE_SID}.ora_bkp_cfg

[oratest@ebstestdb dbs]$ cd $ORACLE_HOME/dbs

[oratest@ebstestdb dbs]$ sqlplus / as sysdba

create spfile from pfile;

create pfile from spfile;

exit

ls -ltr

[oratest@ebstestdb dbs]$ mv $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora_cfg

[oratest@ebstestdb dbs] vi init${ORACLE_SID}.ora ß ctl, diag, pga, sga, plsql code & add addtnl Duplicate parameters.

*.control_files='+DATA/cntrl01a.dbf','+DATA/cntrl02b.dbf','+DATA/cntrl03c.dbf' #Verify file name dose not exists

*.diagnostic_dest='/u01/app//oramgr/' #$ORACLE_BASE for Alertlog as PROD

*.log_archive_dest_1='LOCATION=+RECO' #Verify loc with crosscheck. +DATA/AHCTEST/ARCHIVELOG/2022_11_17/thread_1_seq_2.439 & Directory in ASM

ORA-15173: entry 'archive' does not exist in directory '/'

*.max_dump_file_size=UNLIMITED# trace file size # default 2048 (Doc ID 1153040.1)

*.pga_aggregate_target=16G  # default 1G

*.plsql_code_type='NATIVE'

*.sga_target=16G   # default 2G

*.shared_pool_size=800M # default 400M.

sga_max_size=16G

db_recovery_file_dest_size=1500G

db_recovery_file_dest='+RECO' # $ORACLE_BASE/flash_recovery_area

db_create_file_dest='+DATA' # $ORACLE_BASE -> /oradata/AHC/testdata/

# if restoring from rman bkp pcs add below parameters also

#cluster_database=FALSE # Mandatory if restoring from RMAN bkp pcs.

#db_file_name_convert='+DATA/PROD/DATAFILE','+DATA','+DATA/PROD/TEMPFILE','+DATA'

#db_file_name_convert='+DATA/PROD/DATAFILE','/u01/TEST/oradata','+DATA/PROD/TEMPFILE','/u01/TEST/oradata' # Set according if restoring from rman bkp pcs for non ASM

#log_file_name_convert='+DATA/PROD/ONLINELOG','+DATA' # Set according if restoring from rman bkp pcs

[oratest@ebstestdb dbs]$ 

[oratest@ebstestdb dbs]$ cp init${ORACLE_SID}.ora init${ORACLE_SID}.ora_rman

[oratest@ebstestdb dbs]$ mv spfile${ORACLE_SID}.ora spfile${ORACLE_SID}.ora_cfg

SPFILE will created Automatically by RMAN  we can ignore this step:

sqlplus / as sysdba

create spfile from pfile;

exit;

cd $ORACLE_HOME/dbs

ls -ltr

orapwd file=$ORACLE_HOME/dbs/orapwAHCTEST password=sys4nwr entries=30 force=y --- the password of sys must be same as the sys user of prod db 

Add the below prod TNS
entry in the clone server to connect PROD

On Target – clone server - TESTDB

vi $TNS_ADMIN/tnsnames.ora
PROD=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.4.7)(PORT=1527))
                (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.4.8)(PORT=1527))
                (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.4.9)(PORT=1527))
            (CONNECT_DATA=
                (SERVICE_NAME=PROD)
                (INSTANCE_NAME=PROD1)
            )
        )

tnsping PROD
Check the tnsping on prod and the clone server….it should be ok…..then start the rmn duplicate …..on both source and target.

 INFORMATIONAL NOTE FOR REFRENCE: (never got such error in clone(AUX))

(UR=A)
If UR=A is not added,you will get following error during connection to target in nomount state in next steps:-

ORA-12528: TNS:listener: all appropriate instances are blocking new connections

The (UR=A) clause for TNS connect strings was created in response to an enhancement request.
This clause can be inserted into the “(CONNECT_DATA=” section of a TNS connect string and allow a privileged or administrative user to connect via the listener even when the service handler is blocking connections for non-privileged users. This feature is introduced since Oracle 10g

Register in listener
SQL>alter system register;

Add the TNS entries in the
PROD tnsnames.ora for the clone server service name ex:

On Source EBS Instance - PROD

AHCTEST= (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.4.12)(PORT=1522))
                (CONNECT_DATA=(SID= AHCTEST))
            )

tnsping ahctest
 

On Target – clone server – TESTDB:

 
  sqlplus / as sysdba

startup nomount

show parameter spfile;
show parameter control_files
show parameter plsql_code_type
show parameter sga_max_size
show parameter sga_target
show parameter pga%target
show parameter shared_p
show parameter db_create_f
show parameter db_reco
show parameter log_archive_dest_1

show parameter cluster_database
show parameter convert

convert used if restore DB from bkp pcs
convert parameter keep empty
ß+DATA,/oradata/AHC/datafile
ß+RECO,/oradata/AHC/fra
ß+RECO, BPDATA

ß Cluster = false
convert parameters keep empty convert used if restore DB from bkp pcs

Orapw
check, Alert log file location:

SELECT INSTANCE_NAME, STATUS,
DATABASE_STATUS FROM V$INSTANCE;

select * from v$pwfile_users;

SELECT INST_ID,value from gv$diag_info where name='Diag Trace';

select 'tailf '||d.value||'/alert_'||i.INSTANCE_NAME||'.log' from gv$diag_info d,v$instance i where d.name='Diag Trace';

INSTANCE_NAME    STATUS       DATABASE_STATUS

---------------- ------------ -----------------

AHCTEST          STARTED      ACTIVE

SQL>

USERNAME                      SYSDB SYSOP SYSAS SYSBA SYSDG SYSKM     CON_ID

----------------------------- ----- ----- ----- ----- ----- ----- ----------

SYS                           TRUE  TRUE  FALSE FALSE FALSE FALSE          0

SQL>

VALUE

----------------------------------------------------------------------------

/u01/app/oramgr/diag/rdbms/ahctest/AHCTEST/trace

SQL> 

grep diagnostic_dest $ORACLE_HOME/dbs/init${ORACLE_SID}.ora

tailf /u01/app/oramgr/diag/rdbms/ahctest/AHCTEST/trace/alert*.log

[oratest@ebstestdb ~]$ ps -ef | grep -i oratest |wc -l

40

RMAN Connection to Prod for
Cloning:

If there is a restriction in getting PROD sys password then use existing password file for RMAN Duplicate with below steps : (Copy orapwd from ASM to local File system)
[grid@ebstestdb~]$ srvctl config database -d PROD
. . . . . 
Spfile: +DATA/spfilePROD.ora
Password file: +DATA/orapwPROD
Domain:
. . . . . 
[grid@proddb01 ~]$
ASMCMD [+data] > pwcopy +DATA/orapwPROD /home/grid/orapwPROD
[oramgr@ebsproddb1 ~]$ cp /home/grid/orapwPROD $ORACLE_HOME/dbs
[oramgr@ebsproddb1 ~]$ scp /home/grid/orapwPROD oracle@ebstestdb:$ORACLE_HOME/dbs/orapwAHCTEST

To update password file in ASM:
[oracle@prod01 ~]$ srvctl config database -d $ORACLE_UNQNAME
Password file: +DATA/NEREG/PASSWORD/pwdnereg.316.1132231531
Domain:

[oracle@prod01 dbs]$ orapwd file=+DATA/NEREG/PASSWORD/orapwNEREG password=59awestridge_1 dbuniquename='NEREG' entries=30 force=y
[oracle@prod01 ~]$ srvctl config database -d $ORACLE_UNQNAME
Password file: +DATA/NEREG/PASSWORD/orapwnereg


Domain:[oratest@ebstestdb ~]$ cd $ORACLE_HOME/dbs 
[oratest@ebstestdb ~]$ orapwd file=$ORACLE_HOME/dbs/orapwAHCTEST password=sys4nwr entries=30 force=y
[oratest@ebstestdb ~]$

Always try to execute long running scripts in a screen or tmux Session to avoid network disconnection issue:
Mode: 
To restore from RMAN Backup Follow from here:


Source database Should be in Archivelog Mode:
-free -h
- echo 3 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a && printf '\n%s\n' ' Ram-cache and the swap get cleared'
-rman target sys@PROD    <- need to check/correct sys connection without password
-rman target sys@PROD nocatalog auxiliary sys@AHCTEST

-rman target sys/sys4nwr@PROD
-rman target sys/sys4nwr@PROD nocatalog auxiliary sys/sys4nwr@AHCTEST |& tee ~/rman_dup.txt
Recovery Manager: Release 12.1.0.2.0 - Production on Mon Sep 28 10:27:42 2020

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PROD (DBID=1307206531)
using target database control file instead of recovery catalog
connected to auxiliary database: AHCTEST (not mounted)
RMAN>
RMAN> SHOW DEVICE TYPE;
RMAN> SHOW CHANNEL;
RMAN> SHOW SNAPSHOT CONTROLFILE NAME;
RMAN>
click here for details 
check show all; for parallelism make it 10 if not 10 (should be able to show source database configuration)

RMAN configuration parameters for database with db_unique_name PROD are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 5 DAYS;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 15 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '+RECOC1/rmanbkp/%U';
CONFIGURE MAXSETSIZE TO 16200 M;
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DATAC1/PROD/CONTROLFILE/snapcf_PROD1.f';
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DATAC1/PROD/CONTROLFILE/snapcf_PROD1.f';


RMAN Duplication for Target Database (Active DB Cloning):
srvctl config database
srvctl remove database -d TEST  (before duplicate remove any registered database)

RMAN> (approx. 12 Mins at parallelism 15)
RMAN> (approx. 14 Mins at parallelism 10)
RMAN> duplicate database to AHCTEST from active database nofilenamecheck;
RMAN> duplicate database to UPGTEST from active database nofilenamecheck;
RMAN> duplicate database to AHCOCI11 from active database USING COMPRESSED BACKUPSET nofilenamecheck; (approx. 55 Mins for compressed bkp @10 PARALLELISM)


If unable to update parallelism configuration due to any reason run  manually for RMAN session;
RMAN> show all;
RMAN> show AUXILIARY CHANNEL;
RMAN> RUN {
allocate auxiliary channel t1 type disk; # channels in Run{ will be released by it self when run command complete given tasks by closing "}"
allocate auxiliary channel t2 type disk;
allocate auxiliary channel t3 type disk;
allocate auxiliary channel t4 type disk;
allocate auxiliary channel t5 type disk;
duplicate database to AHCTEST from active database nofilenamecheck;}

The above clone command restore datafile and current(16…?) archives form backup set

Make sure you backup a new pfile after completing RMAN
create pfile='/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/initAFTER_RMAN.ora' from spfile;
RMAN> duplicate database to ECCAR from active database nofilenamecheck dorecover; <- this command will not work gives below error

Starting Duplicate Db at 11-FEB-21
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 02/11/2021 14:50:05
RMAN-05501: aborting duplication of target database
RMAN-05571: DORECOVER specified for non standby duplication

RMAN>

duplicate database to TESTUAT from active database section size 500M using compressed backupset; --- from standby db
This will start restoring the database to target clone server…..check the data top folder to confirm…

After the rman duplicate is over…delete password files if created also the tns service names from both prod and clone server……. 


RMAN Restore From Backup Pieces:
srvctl config database
srvctl remove database -d TEST  (before duplicate remove any registered database)

If restoring form RMAN Backup Pieces then follow:
Incase if no psu is applied on Oracle Home then you may face error during duplication as mentioned in bleow note, apply patch 19503821
WARNING: failed to start ASMB after RAC Database on ASM converted to Single Instance Non-ASM Database (Doc ID 2138520.1)

select NAME from v$controlfile;
select file_name from dba_data_files;
select name from V$TEMPFILE;
select member from v$logfile;

[oratest@ebstestdb~]$ $ORACLE_HOME/OPatch/opatch lsinventory | grep 19503821

add below pfile parameters by modifying as per requirement -> create spfile and backup it:
cluster_database=false
db_file_name_convert='+DATA/PROD/DATAFILE','/u01/app/oracle/DCLONE/DATA','+DATA/PROD/TEMPFILE','/u01/app/oracle/DCLONE/DATA'
log_file_name_convert='+RECO/PROD/ONLINELOG','/u01/app/oracle/DCLONE/DATA'
#log_archive_dest_1='LOCATION=/u01/app/oracle/DCLONE/DATA/archive'


create spfile from pfile;
startup nomount
exit

[oratest@ebstestdb~]$ rman auxiliary /
run {
ALLOCATE AUXILIARY CHANNEL aux1 DEVICE TYPE DISK; # channles in Run{ will be released by it self when run command complete given tasks by closing "}"
ALLOCATE AUXILIARY CHANNEL aux2 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux3 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux4 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux5 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux6 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux7 DEVICE TYPE DISK;
duplicate database to AHCTEST backup location '/backup/rman';}

no need of providing Release channel if it allocated in Run{ once the duplicate command complete it automatically releases the channel and you wont get below error.
Lets make a try with out issuing any duplicate in RUN{
same will happen for normal channel and auxiliary channel.
RMAN> RUN {
allocate AUXILIARY channel t1 type disk;
allocate AUXILIARY channel t2 type disk;
allocate AUXILIARY channel t3 type disk;}

allocated channel: t1
channel t1: SID=4 device type=DISK

allocated channel: t2
channel t2: SID=2006 device type=DISK

allocated channel: t3
channel t3: SID=4013 device type=DISK
released channel: t1
released channel: t2
released channel: t3

RMAN> 

RMAN % calculation made as per allotted channels for backup/restore
SQL> 
col OPNAME for a30
set pagesize 1000
set linesize 200;
select OPNAME,SOFAR/TOTALWORK*100 PCT, trunc(TIME_REMAINING/60) MIN_RESTANTES,
trunc(ELAPSED_SECONDS/60) MIN_ATEAGORA
from v$session_longops where TOTALWORK>0 and OPNAME like '%RMAN%';


RMAN Duplicate Transfer Rate for duplicate
set linesize 200;
SELECT 'DUPLICATE/RESTORE THROUGHPUT',
round(SUM(v.value/1024/1024/1024),1) Giga_BYTES_SOFAR,
round(SUM(v.value /1024/1024)/nvl((SELECT MIN(elapsed_seconds)
FROM v$session_longops
WHERE OPNAME LIKE 'RMAN: aggregate input'
AND SOFAR != TOTALWORK AND elapsed_seconds IS NOT NULL ),SUM(v.value /1024/1024)),2) mbytes_per_sec,n.name
FROM gv$sesstat v, v$statname n, gv$session s
WHERE v.statistic#=n.statistic#
AND n.name = 'physical write total bytes' AND v.sid = s.sid AND v.inst_id=s.inst_id AND s.program like 'rman@%'
GROUP BY 'DUPLICATE/RESTORE THROUGHPUT',n.name;

--Query 2 for % complete check
SELECT SID, SERIAL#, CONTEXT, SOFAR, TOTALWORK, 
ROUND (SOFAR/TOTALWORK*100, 2) "% COMPLETE"
FROM V$SESSION_LONGOPS
WHERE OPNAME LIKE 'RMAN%' AND OPNAME NOT LIKE '%aggregate%'
AND TOTALWORK! = 0 AND SOFAR <> TOTALWORK;


SELECT 'DUPLICATE/RESTORE THROUGHPUT',
round(SUM(v.value/1024/1024),1) mbytes_sofar,
round(SUM(v.value /1024/1024)/nvl((SELECT MIN(elapsed_seconds)
FROM v$session_longops
WHERE OPNAME LIKE 'RMAN: aggregate input'
AND SOFAR != TOTALWORK AND elapsed_seconds IS NOT NULL ),SUM(v.value /1024/1024)),2) mbytes_per_sec,n.name
FROM gv$sesstat v, v$statname n, gv$session s
WHERE v.statistic#=n.statistic#
AND n.name = 'physical write total bytes' AND v.sid = s.sid AND v.inst_id=s.inst_id AND s.program like 'rman@%'
GROUP BY 'DUPLICATE/RESTORE THROUGHPUT',n.name;

Recreating Standby for Datagaurd Connection for RMAN:

rman target sys/sys4nwr@PROD auxiliary sys/sys4nwr@AHCSTBY1

RMAN> duplicate target database for standby from active database using compressed backupset nofilenamecheck dorecover;

If dorecover option is not defined, then archives logs will not be shifted to standby. You need to shift using DGBROKER by enabling standby or manually start MRP-recovery mode on standby:

duplicate database to TESTUAT from active database section size 500M using compressed backupset; --- from standby db

Verify datafiles and archives in standby & monitor Alert log

ls -ltr /oradata/AHC/datafiles/

ls -ltr /oradata/AHC/fra/AHCSTBY1/archivelog/

ls -ltr |wc -l

select count(*) from v$dba_data_files;

select INST_ID,value from gv$diag_info where name='Diag Trace';

Enable Flashback on Standby after RMAN duplicate complete

SQL> select LOG_MODE,FLASHBACK_ON,FORCE_LOGGING from v$database;

SQL> alter database flashback on;

SQL> alter database recover managed standby database cancel; 

SQL> alter database recover managed standby database disconnect from sessions;

Standby Database status and verification of DG Broker:

set linesize 200 pagesize 400;

col name for a10;

col DB_UNIQUE_NAME for a15;

select NAME,DB_UNIQUE_NAME,CONTROLFILE_TYPE,OPEN_MODE,DATABASE_ROLE,PROTECTION_MODE, DATAGUARD_BROKER from v$database;

From Primary Database enable standby DB

DGMGRL> show configuration;

Configuration - ahc_dg

  Protection Mode: MaxPerformance

  Members:

  PROD - Primary database

    ahcstby1 - Physical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS   (status updated 44 seconds ago)

DGMGRL> enable database ahcstby1;

Enabled.

DGMGRL>

Perform the post duplication steps to remove the 2nd node log files and undo tablespace

set linesize 200 pagesize 400;

col name for a10;

col DB_UNIQUE_NAME for a12;

select NAME,DB_UNIQUE_NAME,CONTROLFILE_TYPE,OPEN_MODE,DATABASE_ROLE,PROTECTION_MODE, DATAGUARD_BROKER from v$database;

SELECT INSTANCE_NAME, STATUS, DATABASE_STATUS FROM V$INSTANCE;

INSTANCE_NAME    STATUS       DATABASE_STATUS

---------------- ------------ -----------------

AHCTEST          OPEN         ACTIVE

SQL> 

SELECT INSTANCE_NAME, STATUS, DATABASE_STATUS FROM V$INSTANCE;--Instnc Detail

select count(*) from DBA_POLICIES;--security policies in DB

select * from dba_registry_sqlpatch;--latest BUNDLE PATCH (BP) in EBS DB

select * from ad_trackable_entities where abbreviation in('ad','txk','atg_pf','fnd','fwk') --,'au','gl','per');--Code Level

SQL> select name from v$controlfile;

NAME

--------------------------------------------------------------------------------

+DATA/cntrl_ecc_01.dbf

+DATA/cntrl_ecc_02.dbf

+DATA/cntrl_ecc_03.dbf

SQL>

SQL> select member from v$logfile;

MEMBER

--------------------------------------------------------------------------------

+DATA/AHCTEST/ONLINELOG/group_4.535.1052304065

+DATA/AHCTEST/ONLINELOG/group_4.532.1052304065

+DATA/AHCTEST/ONLINELOG/group_3.546.1052304065

+DATA/AHCTEST/ONLINELOG/group_3.684.1052304065

+DATA/AHCTEST/ONLINELOG/group_2.542.1052304065

+DATA/AHCTEST/ONLINELOG/group_2.543.1052304065

+DATA/AHCTEST/ONLINELOG/group_1.541.1052304065

+DATA/AHCTEST/ONLINELOG/group_1.402.1052304065

+DATA/AHCTEST/ONLINELOG/group_9.666.1052304047

+DATA/AHCTEST/ONLINELOG/group_9.669.1052304047

+DATA/AHCTEST/ONLINELOG/group_10.670.1052304047

MEMBER

--------------------------------------------------------------------------------

+DATA/AHCTEST/ONLINELOG/group_10.614.1052304047

+DATA/AHCTEST/ONLINELOG/group_11.671.1052304047

+DATA/AHCTEST/ONLINELOG/group_11.352.1052304049

+DATA/AHCTEST/ONLINELOG/group_12.663.1052304049

+DATA/AHCTEST/ONLINELOG/group_12.258.1052304049

16 rows selected.

SQL>

SQL> select group#,thread#,members,status from v$log;

GROUP# THREAD# MEMBERS STATUS

---------- ---------- ---------- ----------------

1 1 2 CURRENT

2 1 2 INACTIVE

3 1 2 INACTIVE

9 2 2 UNUSED

10 2 2 UNUSED

11 2 2 UNUSED

6 rows selected.

SQL>

SQL> select THREAD#, STATUS, ENABLED from v$thread;

   THREAD# STATUS ENABLED

---------- ------ --------

         1 OPEN   PUBLIC

         2 CLOSED PUBLIC

SQL> select group# from V$log where THREAD#=2;

    GROUP#

----------

9

10

11

12

SQL> select THREAD#, STATUS, ENABLED from v$thread;

select group# from V$log where THREAD#=2;

alter database disable
thread 2;
alter database drop
logfile group 9;
alter database drop
logfile group 10;
alter database drop
logfile group 11;
alter database drop
logfile group 12;
select
THREAD#, STATUS, ENABLED from v$thread;

   THREAD# STATUS ENABLED

---------- ------ --------

         1 OPEN   PUBLIC

SQL> 

show parameter undo 

select tablespace_name from dba_tablespaces where TABLESPACE_NAME='APPS_UNDOTS2';

TABLESPACE_NAME

------------------------------

APPS_UNDOTS2

34 rows selected.

SQL> drop tablespace APPS_UNDOTS2 including contents and
datafiles;

Tablespace dropped.

SQL> select tablespace_name,file_name,status from dba_temp_files;

12 rows selected.

2)	To update the library files (Optional)

$ cd [RDBMS ORACLE_HOME]/appsutil/install/[CONTEXT NAME]

$ sqlplus “/ as sysdba” @adupdlib.sql so 

3)	 TO configure application context inside the database (Optional) 

$ cd [RDBMS ORACLE_HOME]/appsutil/clone/bin

$ perl adcfgclone.pl dbconfig [Database target context file]

Where Database target context file is: [RDBMS ORACLE_HOME]/appsutil/[Target CONTEXT_NAME].xml.

Adop Prepare Phase Fails While Calling ad_zd.prepare" With Error "ORA-04031: unable to allocate 256 bytes of shared memory" (Doc ID 2043199.1)

SQL> 

show parameter sga

show parameter pga

show parameter recovery

alter system set sga_max_size=16G scope=spfile sid='*';

alter system set sga_target=12G scope=spfile sid='*';

alter system set pga_aggregate_target=4G scope =spfile sid='*';

alter system set db_recovery_file_dest_size=1500G scope =spfile sid='*';

Solution for adop prepare:

1.Set sga_target = 6g and pga_aggregate_target = 6g for adop.

2.Restart the database.

3.run adop phase=prepare again.

Disable archive log mode on
TARGET clone server

Disable Archivelog mode on TARGET clone Server:
SQL> select LOG_MODE,FLASHBACK_ON,FORCE_LOGGING from v$database;

LOG_MODE     FLASHBACK_ON       FORCE_LOGGING
------------ ------------------ ---------------------------------------
ARCHIVELOG   YES                YES

SQL>
SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /oradata/AHC/testdata/archive
Oldest online log sequence     1
Next log sequence to archive   1
Current log sequence           1
SQL> shutdown immediate  ß startup mount force;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size                  2922264 bytes
Variable Size             654313704 bytes
Database Buffers          402653184 bytes
Redo Buffers               13852672 bytes
Database mounted.
SQL> select name,open_mode from v$database;

NAME      OPEN_MODE
--------- --------------------
AHCTEST   MOUNTED

SQL>
SQL> alter database noarchivelog;
alter database noarchivelog
*
ERROR at line 1:
ORA-38774: cannot disable media recovery - flashback database is enabled

SQL> select name,open_mode from v$database;

NAME      OPEN_MODE
--------- --------------------
ECCTEST   MOUNTED

SQL>
SQL> select LOG_MODE,FLASHBACK_ON,FORCE_LOGGING from v$database;

LOG_MODE     FLASHBACK_ON       FORCE_LOGGING
------------ ------------------ ---------------------------------------
ARCHIVELOG   YES                YES

SQL>
SQL> alter database flashback off;
Database altered.
SQL> alter database noarchivelog;
Database altered.
SQL> alter database open;
Database altered.

SQL> select LOG_MODE,FLASHBACK_ON,FORCE_LOGGING from v$database;

LOG_MODE     FLASHBACK_ON       FORCE_LOGGING
------------ ------------------ ---------------------------------------
NOARCHIVELOG NO                 YES

SQL> show edition
EDITION
------------------------------
V_20230116_1150

Alter session set edition =V_20230116_1150;

show parameter sga
show parameter pga
show parameter recovery

FND_NODES clear before
running autoconfig:

Clear fnd nodes before running autoconfig on db to avoid error in apps cloning at autconfig step
“txkGenADOPWrapper.pl” (Doc ID 2064223.1)
Running AutoConfig generates ORA-04063: Package Body "APPS.AD_ZD_ADOP" Has Errors (Doc ID 2175698.1)


SQL> conn apps/appsr12 
SQL> 
SQL> select release_name from apps.fnd_product_groups;

RELEASE_NAME
--------------------------------------------------
12.2.9

SQL>
SQL> select NODE_NAME from fnd_nodes;

NODE_NAME
--------------------------------------------------------------------------------
EBSPRODDB1
EBSPRODDB2
EBSPRODAPP1
EBSPRODAPP3
EBSPRODAPP4
AUTHENTICATION

SQL> select NODE_NAME from fnd_oam_context_files;
SQL>
How to Synchronize the AD tables FND_NODES, ADOP_VALID_NODES, and FND_OAM_CONTEXT_FILES in 12.2 when adop fails. (Doc ID 2064223.1)
SQL> select NODE_NAME from ADOP_VALID_NODES; -- prod apps entries
SQL> select NODENAME from ad_nodes_config_status; -- prod apps entries
SQL> select NODE_NAME from fnd_oam_context_files; --3:DB(template,metadata,xml) 8:apps( 4-RFS, 4-PFS )

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL>
SQL> commit;

[oratest@ebstestdb ~]$ $ORACLE_HOME/OPatch/opatch lsinventory


Finally run autoconfig on DB tier to fix all the configuration
[oratest@ebstestdb ~]$ 
[oratest@ebstestdb ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh
[oratest@ebstestdb AHCTEST_ebstestdb]$ adautocfg.sh
This should complete successfully


oraoci@ebstestdb:~$ sqlplus / as sysdba
SQL> conn apps/appsr12
Connected.
SQL>
SQL>
SQL> select NODE_NAME from fnd_nodes;

NODE_NAME
--------------------------------------------------------------------------------
EBSTESTDB

SQL>
select NODE_NAME from ADOP_VALID_NODES; --check status before apps tier cloning
select NODENAME from ad_nodes_config_status; --check status before apps tier cloning
select NODE_NAME from fnd_oam_context_files; --check status before apps tier cloning

To fix adop ETCC Codelevel checker unzip patch and run atleast 1 time on cloned instance:
Running ADOP Fails With Error "ETCC Not Run In The Database Node" (Doc ID 2191678.1)

[oratest@ebstestdb:/u01/app/oratest]$ unzip p17537119_R12_GENERIC.zip
SQL> select NODE_NAME from apps.TXK_TCC_RESULTS;--etcc checkdb, checkmt ran or not 
SQL> select * from apps.TXK_TCC_RESULTS;--etcc checkdb, checkmt ran or not
[oratest@ebstestdb:~]$ sh $ORACLE_HOME/appsutil/etcc/checkDBpatch.sh  ß mandatory, if not adop will fail when ever used on instance
[oratest@ebstestdb: ~]$ ls -ltrh $ORACLE_HOME/appsutil/etcc/check*

[applmgr@ebstestapp:~]$ sh checkMTpatch.sh  ß mandatory, if not adop will fail when ever used on instance

Reset system and Custom Schemas password & Expdp Directory in new Instance:

----------- Check system password status for apps patching and expdp:
SQL> conn / as sysdba
SQL> select username,account_status,EXPIRY_DATE from dba_users where username like 'SYSTEM';
SQL> ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;
SQL> alter user system identified by manager;


Custom schemas password change if any
SQL> 
SQL> alter user ajar_live identified by ajar_live;
SQL> alter user ajar_live account unlock;
SQL> 

expdp logical directory creation
SQL> SELECT * FROM dba_directories where DIRECTORY_NAME='EXPDP';

Replace Existing EXPDP
SQL> create or replace directory expdp as '/u01/app/oracle/expdp';
Directory created.

Drop and create EXPDP
SQL> drop directory expdp;
Directory dropped.
SQL>
SQL> create directory expdp as '/u01/app/oraar/expdp';
Directory created.
SQL> exit
[oratest@ebstestdb:~$ mkdir -p /u01/app/oracle/expdp
[oratest@ebstestdb:~$ chown oraar:oinstall /u01/app/oracle/expdp
[oratest@ebstestdb:~$ chmod -R 777 /u01/app/oracle/expdp/
[oratest@ebstestdb:~$





Optional change in Cloned DB Instance:
Setting SQL prompt in SQL*Plus Oracle along with user,connection, date & time (sql beautifier)
[oratest@ebstestdb ~]$ cp $ORACLE_HOME/sqlplus/admin/glogin.sql $ORACLE_HOME/sqlplus/admin/glogin.sql_bkp

>$vi $ORACLE_HOME/sqlplus/admin/glogin.sql
Go--set sqlprompt "_user'@'_connect_identifier 'on' _date> "
--alter session set nls_date_format='dd-Mon-yyyy hh24:mi:ss';
--alter session set nls_date_format='dd/mm/yyyy hh:mi:sspm';

Creating DB alert log link in user's home with customize script:
select 'ln -s '||d.value||'/alert_'||i.INSTANCE_NAME||'.log ~/alert_'||i.INSTANCE_NAME||'.log' from gv$diag_info d,v$instance i where d.name='Diag Trace';
--select INST_ID,value from gv$diag_info where name='Diag Trace';-- alert log file location

SQL> !tail -2f $ORACLE_HOME/sqlplus/admin/glogin.sql


Gather SYS schema statistics: in parallel of duualfs    --------------( 10 + 5 + 1) mins in total
Mandatory to execute one by one else command will not execute at once
DBMS_STATS.GATHER_SCHEMA_STATS got "'DBMS_STATS.AUTO_SAMPLE_SIZE' must be declared" (Doc ID 2804534.1)
SQL> show user
USER is "SYS"
SQL> select OWNER, TABLE_NAME, LAST_ANALYZED from dba_tab_statistics where table_name like 'X$%' order by last_analyzed;--1162

SQL> begin
dbms_stats.gather_schema_stats('SYS',options=>'GATHER STALE', estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE,method_opt => 'FOR ALL COLUMNS SIZE AUTO',cascade => TRUE);
 end;
/
SQL> exec dbms_stats.gather_fixed_objects_stats;
SQL> exec dbms_stats.gather_dictionary_stats;
SQL>

[oratest@ebstestdb ~]$ ln -s /u01/app/oracle/diag/rdbms/uat/UAT/trace/alert_UAT.log ~/alert_UAT.log
 

Opatch update in .env file:

[oramgr@ebsproddb1:~$ opatch version
-bash: opatch: command not found
[oramgr@ebsproddb1:~$
[oratest@ebstestdb:~$ which opatch
which: no opatch in (/u01/app/oratest/product/12.1.0.2/dbhome_1/perl/bin:/u01/app/oratest/product/12.1.0.2/dbhome_1/perl/bin:/u01/app/oratest/product/12.1.0.2/dbhome_1/bin:/usr/ccs/bin:/usr/sbin:/u01/app/oratest/product/12.1.0.2/dbhome_1/jdk/jre/bin:/u01/app/oratest/product/12.1.0.2/dbhome_1/bin:/usr/bin:/usr/sbin:.:.)
[oratest@ebstestdb:~$ vi $ORACLE_HOME/AHCTEST_ebstestdb.env (update PATH)

:$ORACLE_HOME/OPatch:$PATH

[oratest@ebstestdb ~]$
[oratest@ebstestdb ~]$ which opatch
/u01/app/oratest/product/12.1.0.2/dbhome_1/OPatch/opatch
[oratest@ebstestdb ~]$
[oratest@ebstestdb ~]$ opatch version
OPatch Version: 12.2.0.1.24

OPatch succeeded.
[oratest@ebstestdb ~]$ $ORACLE_HOME/OPatch/opatch lsinventory | more

. . . . . 

. . . . .

Patch description:  "Database PSU 12.1.0.2.210119,Oracle JavaVM Component (JAN2021)"

Database Cloning is
completed Successfully

SQL> select owner, TRIGGER_NAME,DESCRIPTION,STATUS from dba_triggers where triggering_event like '%LOGON%'; 

Check the status of trigger it was disable because of below error I manually enable with system user

Checking the port pool 3

done: Port Pool 3 is free

Report file located at /u01/dev/fs2/inst/apps/DEVPDB_ahcdevcloudapp/admin/out/portpool.lst

The new APPL_TOP context file has been created :

  /u01/dev/fs2/inst/apps/DEVPDB_ahcdevcloudapp/appl/admin/DEVPDB_ahcdevcloudapp.xml

Check Clone Context logfile /u01/dev/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0803170553.log for details.

FMW Pre-requisite check log file location : /u01/dev/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Running: FMW pre-req check...

Configuring: Run file system....

LogFile located at /u01/dev/fs1/inst/apps/DEVPDB_ahcdevcloudapp/admin/log/clone/run/RCloneApplyAppstier_08031706.log

ERROR: Failed to configure the target system,

please check the logfile in : /u01/dev/fs1/inst/apps/DEVPDB_ahcdevcloudapp/admin/log/clone

Enter the APPS password : Enter the APPS password : Error while executing /u01/dev/fs1/EBSapps/comn/clone/bin/alter_ebs_logon_trigger.sql

Failed to enable EBS_LOGON trigger

[applmgr@ahcdevcloudapp bin]$

Application Cloning Starts from here

Now Configure APPS Tier(Start Apps Cloning):

root@ebstestapp:~# tar -xvzf EBSapps_14112020.tar -C /u01/AHCTEST/fs1 (approx. 3hrs+ )

tar -xvzf 12cdb1_23062021.tar.gz -C /u01/app/oracle/product/12.1.0.2 (
untar with root only )

Fix Inventory Location and central/global
inventory.xml file update

Inventory location should be inside EBS ORACLE_HOME(10.1.2)

Update Clone instance Inventory location should be in ORACLE_HOME: or

Best to point it local Inv to direct central inventory location (As Msb Said) – loc set in file /var/opt/oracl/oraInst.loc)

Inventory Location Fixup on APPSTIER:

Best Approach is to point Location of local Inventory to Central Inventory to avoid issue said by MSB

root@ebstestapp:~# cat /etc/oraInst.loc

root@ebstestapp:~# cat /var/opt/oracle/oraInst.loc

inventory_loc=/export/home/applahc/oraInventory

inst_group=dba

[appltest@ebstestapp ~]$

[appltest@ebstestapp ~]$ cd /export/home/applahc/oraInventory/ContentsXML/

[appltest@ebstestapp ~]$

[appltest@ebstestapp ContentsXML]$ cp inventory.xml inventory.xml_bkp

[appltest@ebstestapp ContentsXML]$

Update Global inventory.xmlfile and delete entry’s of Clone instance from .xml file, if using any same existing SID’s and leave last 2 lines

</HOME_LIST>

</INVENTORY>

Once the Central Inventory backup is done, we need to update it.

If you are using Same SID and Same Path. For an instance which is creating for the first time no need to delete anything.

[appltest@ebstestapp ~]$ cat /export/home/applahc/oraInventory/ContentsXML/inventory.xml

. . . . .

<HOME NAME="AHCTEST_TOOLS__u01_AHC_fs1_EBSapps_10_1_2" LOC="/u01/AHC/APPS122/fs1/EBSapps/10.1.2" TYPE="O" IDX="8"/>

</HOME_LIST>

</INVENTORY>

root@ebstestapp:~#

[appltest@ebstestapp ~]$ cat /u01/AHCTEST/fs1/EBSapps/10.1.2/oraInst.loc

inventory_loc=/u01/AHC/oraInventory <--Point location to Central Inventory.

inst_group=dba

[appltest@ebstestapp ~]$ vi /u01/AHCTEST/fs1/EBSapps/10.1.2/oraInst.loc

[appltest@ebstestapp ~]$

[appltest@ebstestapp ~]$ cat /u01/AHCTEST/fs1/EBSapps/10.1.2/oraInst.loc

inventory_loc=/export/home/applahc/oraInventory

inst_group=dba

[appltest@ebstestapp ~]$

For the below need to try wirh Global Inventory remove home if already exist,

Try next time with attach and detach command, anyways above will also work:

[oraoci@rac1 bin]$ $ORACLE_HOME/oui/bin/runInstaller -silent -attachHome -invPtrLoc /var/opt/oracle/oraInst.loc ORACLE_HOME="/u01/app/oraoci/product/19.3.0.0/dbhome_1" ORACLE_HOME_NAME="OraGI19Home1"

[oraoci@rac1 ~]$

[oraoci@rac1 bin]$ $ORACLE_HOME/oui/bin/runInstaller -detachHome /u01/app/oraoci/product/19.3.0.0/dbhome_1 -invPtrLoc /var/opt/oracle/oraInst.loc

[oraoci@rac1 bin]$

----------------------------------------------------------------

Method-1 : If you have only one application server running on this node then you can simple delete global inventory it will be recreated automatically by adcfg script

If Didn’t work the above Centrol inventory then 

Setting up oraInventory before adcfg on apps:

Before mv oraInst.loc in 10.1.2 home and remove old entries from global inventory

This will be recreated if there is only one application running on server. 

[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ cd /u01/AHCTEST/fs1/EBSapps/10.1.2/
[appltest@ebstestapp 10.1.2]$ mv oraInst.loc oraInst.loc_03082021
[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ find /u01/AHCTEST/fs1 -iname oraInst.loc*
/u01/AHCTEST/fs1/EBSapps/10.1.2/oraInst.loc_03082021
[appltest@ebstestapp ~]$ cat /etc/oraInst.loc
inventory_loc=/home/applmgr/oraInventory
inst_group=oinstall
[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ cd /home/applmgr/oraInventory
[appltest@ebstestapp oraInventory]$ ls -ltr
total 32
drwxrwx---. 2 applmgr oinstall   22 Jun 24 07:42 oui
drwxrwx---. 7 applmgr oinstall 4096 Jun 24 08:56 backup
-rw-rw----. 1 applmgr oinstall  303 Jun 24 08:58 oraInstaller.properties
drwxrwx---. 2 applmgr oinstall 4096 Jun 24 09:00 Contents
-rw-rw----. 1 applmgr oinstall   37 Jun 24 09:00 install.platform
drwxrwx---. 2 applmgr oinstall 4096 Aug  3 17:38 ContentsXML
drwxrwx---. 2 applmgr oinstall 8192 Aug  3 17:53 logs
[appltest@ebstestapp oraInventory]$ rm -rf *
[appltest@ebstestapp oraInventory]$
[appltest@ebstestapp oraInventory]$ ls -ltr
total 0
[appltest@ebstestapp oraInventory]$

Method-2 : Other Way is to Create directory inside ORACLE_HOME only..
[appltest@ebstestapp ~]$
[appltest@ebstestapp ~]$ cat /u01/AHCTEST/fs2/EBSapps/10.1.2/oraInst.loc
#Best is inventory_loc= /export/home/applahc/oraInventory 
#Above loc is central inv. As per /var/opt/oracle/oraInst.loc
inventory_loc=/u01/AHCTEST/fs2/EBSapps/10.1.2/oraInventory
inst_group=dba

[appltest@ebstestapp ~]$ mkdir -p /u01/AHCTEST/fs2/EBSapps/10.1.2/oraInventory
[appltest@ebstestapp ~]$ chmod -R 777 /u01/AHCTEST/fs2/EBSapps/10.1.2/oraInventory

root@ebstestapp:~# chown -R appltest:dba /u01/AHCTEST

Soft Links Permission Error:

root@ebstestapp:~# ls -ltr /u01/AHCTEST/fs2/EBSapps/10.1.2/webcache/examples

total 255

-rw-rw-r-- 1 applecc dba 70018 Sep 15 2004 IsWebCacheWorking.ear

-rw-rw-r-- 1 applecc dba 2679 Jan 2 2007 invalidate.txt

-rw-rw-r-- 1 applecc dba 4329 Jan 2 2007 invalidate.sql

-rw-rw-r-- 1 applecc dba 896 Jan 2 2007 invalidate.sh

-rw-rw-r-- 1 applecc dba 2309 Jan 2 2007 invalidate.pl

-rw-rw-r-- 1 applecc dba 8345 Jan 2 2007 invalidate.c

-rw-rw-r-- 1 applecc dba 1860 Jan 2 2007 inv-trigger.sql

-rw-rw-r-- 1 applecc dba 2691 Jan 2 2007 Invalidate.java

-rw-rw-r-- 1 applecc dba 626 Jan 2 2007 EncodeBase64.java

-rw-rw-r-- 1 applecc dba 71 Jan 2 2007 README.examples.txt

-rw-rw-r-- 1 applecc dba 24496 Jan 2 2007 WCSInvalidate.java

-rw-rw-r-- 1 applecc dba 885 Jan 2 2007 SyncObject.java

-rw-rw-r-- 1 applecc dba 735 Jan 2 2007 SSLVerifier.java

lrwxrwxrwx 1 applahc dba 62 Nov 20 16:46 readme.examples.html -> /u01/AHCTEST/fs2/EBSapps/10.1.2/webcache/docs/readme.examples.html

root@ebstestapp:~#

applahc if got no connection with database during apps-cgf then set the sqlnet.ora parameters from prod or backup

After this perform the apps tier cloning as usual…

Change .profile …uncomment or point to the prod env file  
Check Permissions on directory and Files before start adcfg
root@ebstestapp:~# su – appalr
[appltest@ebstestapp ~]$  vi .profile
#. /u01/AHCTEST/EBSapps.env run
[appltest@ebstestapp ~]$  cd /u01/AHCTEST/fs2/EBSapps/comn/clone/bin 
[appltest@ebstestapp bin]$  
[appltest@ebstestapp bin]$ ls -ltr
total 578
-rwxr-xr-x   1 appltest dba        48268 Feb 11 08:56 adclonectx.pl
-rwxr-xr-x   1 appltest dba       100066 Feb 11 08:56 adclone.pl
-rwxr-xr-x   1 appltest dba         5101 Feb 11 08:56 adchkutl.sh
-rwxr-xr-x   1 appltest dba       127745 Feb 11 08:56 adcfgclone.pl
-rwxr-xr-x   1 appltest dba        10887 Feb 11 08:56 adaddnode.pl
[appltest@ebstestapp bin]$

Run Perl Script for Application Tier Configuration:

[appltest@ebstestapp bin]$ set <- crosscheck no parameter should set

Always try to run long running scripts in a screen or tmux Session to avoid network disconnection issue:

[root@ebstestapp ~]# tmux -f ~/.tmux.conf (force load all customizing in tmux session)

[root@ebstestapp ~]# tmux ls ----> tmux attach -t 0 ----> copy/paste shift + selection to paste Shift + R.clk paste

[root@ebstestapp ~]# cat ~/.screenrc ----> enable scrolling option

# Enable scrolling

termcapinfo xterm* ti@:te@

[root@ebstestapp ~]# screen -S dualfs ----> screen -d -r dualfs ----> re-attach screen session

RAM Cache Swap Clearing use echo 3 option on test scheduled in crontab daily at 3 AM:

# free -h -w

# echo 1 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a && printf '\n%s\n' ' Ram-cache and the swap get cleared'

free -h -w ----(if required free up the memory of server with echo 3 command)

export TIMEDPROCESS_TIMEOUT=-1

export CONFIG_JVM_ARG="-Xms1024m -Xmx2048m"

export CONFIG_JVM_ARG="-Xms2048m -Xmx4096m"

echo $CONFIG_JVM_ARG

SQL> select * from all_objects where status='INVALID' and owner='APPS' and object_name='AD_ZD_ADOP';--{else autoconfig will fail in dualfs}

SQL> select NODE_NAME from apps.fnd_nodes; --{clear PROD entries before dualfs, else will fail will as show here}

ERROR: txkWfClone.sh execution failed, exit code 1

if you scroll up the logs you may find some more errors

ERRORMSG: *******FATAL ERROR*******

PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)

TIME : Wed Oct 9 14:50:36 2019

FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]

ERRORMSG: Invalid APPS database user credentials.

SQL> exec fnd_conc_clone.setup_clean;

SQL> commit;

[oratest@ebstestdb ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh

time perl adcfgclone.pl appsTier dualfs

(Working day total approx. 3Hrs ={2Hr:RFS + 1Hr+:PFS})

(Weekend day total approx. 2Hrs ={50Min:RFS + 1Hr+:PFS})

Redwood Shores, California, USA

Oracle E-Business Suite Rapid Clone

Version 12.2

adcfgclone Version 120.63.12020000.62

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] :

Running: Context clone...

Log file located at /u01/AHCTEST/fs1/EBSapps/comn/clone/bin/CloneContext_0212140124.log

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [ebstestapp] :

Target System Database SID : AHCTEST (UATPDB/DEVPDB)

Target System Database Server Node [ebstestapp] : ebstestdb

Target System Database Domain Name [areefdba.com.sa] :

Target System Base Directory : /u01/AHCTEST

Target System Base Directory set to /u01/AHCTEST

Target System Current File System Base set to /u01/AHCTEST/fs1

Target System Other File System Base set to /u01/AHCTEST/fs2

Target System Fusion Middleware Home set to /u01/AHCTEST/fs1/FMW_Home

Target System Other File System Fusion Middleware Home set to /u01/AHCTEST/fs2/FMW_Home

Target System Web Oracle Home set to /u01/AHCTEST/fs1/FMW_Home/webtier

Target System Other File System Web Oracle Home set to /u01/AHCTEST/fs2/FMW_Home/webtier

Target System Appl TOP set to /u01/AHCTEST/fs1/EBSapps/appl

Target System Other File System Appl TOP set to /u01/AHCTEST/fs2/EBSapps/appl

Target System COMMON TOP set to /u01/AHCTEST/fs1/EBSapps/comn

Target System Other File System COMMON TOP set to /u01/AHCTEST/fs2/EBSapps/comn

Target System Instance Home Directory [/u01/AHCTEST] :

Target System Current File System Instance Top set to /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp

Do you want to preserve the Display [ebsprodapp4:0.0] (y/n) : n

Target System Display [ebstestapp:0.0] :

Target System Root Service [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 15 (here we can use any port for url 8000 later it will ask exact port of DATABASE 1522)

Checking the port pool 15

done: Port Pool 15 is free

Report file located at /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/out/portpool.lst

RC-40201: Unable to connect to Database AHCTEST.

Enter the Database listener port [1523] : 1521 (nc -zv uatdb01 1521 , nc -zv uatdb01-vip 1521 , lsof -i TCP:1521 , also check db is up or not)

Target System proxy port [80] :

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/tmp

2. /usr/tmp

3. /u01/app/oraar/product/12.1.0.2/dbhome_1/appsutil/outbound/AHCTEST_ebstestdb

4. /usr/tmp

Choose a value which will be set as APPLPTMP value on the target node [1] :

The new APPL_TOP context file has been created :

/u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/appl/admin/AHCTEST_ebstestapp.xml

Check Clone Context logfile /u01/AHCTEST/fs1/EBSapps/comn/clone/bin/CloneContext_0212140124.log for details.

Creating Patch file system context file.....

Log file located at /u01/AHCTEST/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0212140340.log

Target System Other File System Instance Top set to /u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp

Target System Port Pool [0-99] : 16

Checking the port pool 16

done: Port Pool 16 is free

Report file located at /u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/admin/out/portpool.lst

The new APPL_TOP context file has been created :

/u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/appl/admin/AHCTEST_ebstestapp.xml

Check Clone Context logfile /u01/AHCTEST/fs1/EBSapps/comn/clone/bin/CloneContextPatch_0212140340.log for details.

FMW Pre-requisite check log file location : /u01/AHCTEST/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Running: FMW pre-req check...

Configuring: Run file system.... (approx. 50 Mins for RUN FS)

LogFile located at /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/run/RCloneApplyAppstier_02121403.log

Configuring: Patch file system.... (approx. 1Hrs for PATCH FS)

LogFile located at /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/patch/RCloneApplyAppstier_02121501.log

Do you want to startup the Application Services for ECCAR? (y/n) [n] : n

Services not started

[appltest@ebstestapp bin]$

It will complete normal

0% à 50% -à after 30 mins 51% à 65% à 73% ---------completed apply….

It will ask to start the services say NO……

If autoconfig in dual fs failed verify with below log

Running AutoConfig generates ORA-04063: Package Body "APPS.AD_ZD_ADOP" Has Errors (Doc ID 2175698.1)

grep -i 'error \|ORA-\|failed' $INST_TOP/admin/log/*/adconfig.log

important and priority xml parameters need to update:

if faced any issue in autoconfig after editing in xml file this can be applied later after start of application

Remove https login from context file after completing dualfs in OCI

[applmgr@ebstestapp ~]$ cp $CONTEXT_FILE ~

[applmgr@ebstestapp ~]$ grep -i 'sslterminator\|https\|site_name' $CONTEXT_FILE (update ssl setting before running autoconfig replace with below line / add yellow part)

Check whether given Application ports RFS/PFS including weblogic server ports are open or not in OCI:

[root@ebstestapp ~]# sudo systemctl status firewalld

[root@ebstestapp ~]# sudo firewall-cmd --list-all

public (active)

target: default

icmp-block-inversion: no

interfaces: ens3

sources:

services: dhcpv6-client ssh

ports: 8002/tcp 7003/tcp 7021/tcp

protocols:

masquerade: no

forward-ports:

source-ports:

icmp-blocks:

rich rules:

[root@ebstestapp ~]#

files to monitor during adcfgclone on AppsTier:

cat /export/home/applahc/oraInventory/ContentsXML/inventory.xml (Central Inv)
new homes of instances will be updated as per given local inventory details.

Log File for WL-Admin Server, EBS Domain, opmn, nodemanager
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/EBS_domain.log
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/access.log
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.out
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer-diag*.log
tail -100f $FMW_HOME/wlserver_10.3/common/nodemanager/nmHome1/nodem*.log
tail -100f $IAS_ORACLE_HOME/instances/E*OHS1/d*/logs/OPMN/opmn/opmn.log

tail -100f $IAS_ORACLE_HOME/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log   Still some Error related to SSL

Run File System:
tail -f /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/run/RCloneApplyAppstier_02121403.log (Autoconfig update in RFS in this File)(Approx Time RCloneApplyAppstier= 1Hr 3 mins )
Re-trying for 6 time[s]. 
. . . .
Enter the IMAP domain name [NoImapDomain]:<--4 Mins, autocfg, Patch FS start


For regular
Logs in EBS:
1. EBS Cloning check logs under run/patch
tailf $INST_TOP/admin/log/clone/run/RCloneApplyAppstier*.log
tailf $INST_TOP/admin/log/clone/run/fmwT2PApply/CLONE*.log
tailf $INST_TOP/admin/log/clone/run/wlsT2PApply/CLONE*.log
tailf $INST_TOP/admin/log/clone/run/ohsT2PApply/CLONE*.log


2. For Multi Node Config Files to monitor
find $IAS_ORACLE_HOME/instances -iname EBS_web*.log
ls -ltrh $IAS_ORACLE_HOME/instances/EBS_web*/diagnostics/logs/OHS/EBS_web/EBS_web*.log

find $IAS_ORACLE_HOME/instances -type f \( -iname \apps.conf -o -iname \mod_wl_ohs.conf \)

find $IAS_ORACLE_HOME/instances -iname apps.conf
ls -ltrh $IAS_ORACLE_HOME/instances/EBS_web*OHS*/config/OHS/EBS_web*/apps.conf

find $IAS_ORACLE_HOME/instances -iname mod_wl_ohs.conf
ls -ltrh $IAS_ORACLE_HOME/instances/EBS_web*/config/OHS/EBS_web*/mod_wl_ohs.conf


3. For fs_clone creation no run/patch forlder is there:
tailf $INST_TOP/admin/log/clone/run/FSCloneApplyAppstier*.log
tailf $INST_TOP/admin/log/clone/fmwT2PApply/CLONE*.log
tailf $INST_TOP/admin/log/clone/wlsT2PApply/CLONE*.log
tailf $INST_TOP/admin/log/clone/ohsT2PApply/CLONE*.log


tail -f /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/run/fmwT2PApply/CLONE2021-02-12_14-03-57_1047025890.log (% of Complete update here)
(Approx Time fmwT2Apply= 25Mins)
Extracting all the homes from the archive file. Extraction time will vary depending upon the archive size ...
Operation is 11% complete, in "38" seconds.
Operation is 21% complete, in "84" seconds.

tail -f /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/run/wlsT2PApply/CLONE2021-02-12_14-21-42_1319751649.log (Approx Time wlsT2Apply = 40 Mins )
. . . .
Configuring the Datasource -  EBSDataSource <-- 15 Mins in RFS/PFS,
From here total time is 15+23 Mins. To completes

tail -f /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/run/AdminServer.log

tail -f /u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/log/clone/run/ohsT2PApply/CLONE2021-02-12_14-55-16_455292676.log

Patch File System:
/u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/admin/log/clone/patch/RCloneApplyAppstier_11201936.log

cd /u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/admin/log/clone/patch/fmwT2PApply

tail -f /u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/admin/log/clone/patch/fmwT2PApply/

################################################################################################################

If auto config Fails with error below check adautoconfig.log for “txkGenADOPWrapper.pl” (Doc ID 2064223.1)

ls -ltrh $INST_TOP/admin/log/*/adconfig.log

select * from fnd_oam_context_files --still target node not updated

select * from fnd_nodes --still target node not updated

select * from adop_valid_nodes --still target node not updated

select * from ad_nodes_config_status --still target node not updated

select node_id, platform_code, support_db D, support_cp C, support_admin A, support_forms F,

support_web W, node_name, server_id, server_address, domain, webhost, virtual_ip

from fnd_nodes order by node_id;

How to Synchronize the AD tables FND_NODES, ADOP_VALID_NODES, and FND_OAM_CONTEXT_FILES in 12.2 when adop fails. (Doc ID 2064223.1)

SQL> sqlplus apps/appsr12_2_9

SQL>

SQL> select NODE_NAME from fnd_nodes;

SQL> exec fnd_conc_clone.setup_clean;

SQL> commit;

- Run autoconfig dbTier and then (select NODE_NAME from fnd_nodes;)

- Run autoconfig on apps runfile system (select NODE_NAME from fnd_nodes;)

- Then disable ebslogon trigger from runFS

- Then source patch file system with new user session and run auto config

- Then enable ebslogon trigger from runFS

- Then change apps password and run autoconfig on dbTier first and apps Run file Tier

Open duplicate Session disable logon Trigger with apps or system user

Disabling Trigger with apps user (Doc ID 2357063.1)

Disabling Trigger with System User

. EBSapps.env Run

- As apps user disable logon trigger

SQL> SELECT ad_zd.get_edition_type from dual;

GET_EDITION_TYPE

-------------------------------------

RUN

SQL> select owner, TRIGGER_NAME,DESCRIPTION,STATUS from dba_triggers where triggering_event like '%LOGON%';

SQL> exec ad_zd.ALTER_LOGON_TRIGGER('DISABLE');

SQL>

SQL> commit;

Now run autoconfig with the patch env sourced

applahc@ebstestapp:~$ . /u01/AHCTEST/EBSapps.env patch

applahc@ebstestapp:~$ echo $FILE_EDITION

patch

applahc@ebstestapp:~$ cd $ADMIN_SCRIPTS_HOME/

applahc@ebstestapp:~$ pwd

should run adconfig for Patch FS

applahc@ebstestapp:~$ ./adautocfg.sh

SQL> exec ad_zd.ALTER_LOGON_TRIGGER('ENABLE');

SQL> commit;

Re-run AutoConfig in run fs once again.

applahc@ahcsc1zioclient010301:~$ sqlplus system/manager1

[oracle@ahcuatclouddb ~]$ . $ORACLE_HOME/UATPDB_ahcuatclouddb.env

[oracle@ahcuatclouddb ~]$

sqlplus system/manager1@UATPDB

SQL> show user

USER is "SYSTEM"

SQL>

SQL> select * from dba_triggers where triggering_event like '%LOGON%';

SQL> alter trigger ebs_logon disable;

Now run autoconfig with the patch env sourced

applahc@ebstestapp:~$ . /u01/AHCTEST/EBSapps.env patch

applahc@ebstestapp:~$ echo $FILE_EDITION

patch

applahc@ebstestapp:~$ cd $ADMIN_SCRIPTS_HOME/

applahc@ebstestapp:~$ pwd

should run adconfig for Patch FS

applahc@ebstestapp:~$ ./adautocfg.sh

applahc@ebstestapp:~$ sqlplus system/manager1

SQL> alter trigger ebs_logon enable;

################################################################################################################

Then connect the Database and perform the post clone procedures like before …to cancel the scheduled requests….

Change the .profiles to current *.env files on the clone server….

1) Change apps password using FNDCPAAS……then run autoconfig on apps tier

ADOP Tables for node information configured with Single/Multi Apps Node:

1)     select * from fnd_nodes;--(only 3:AUTHENTICATION, DBNODE, APPSNODE)
1.1.1) select * from ADOP_VALID_NODES;  -- prod apps entries (Doc ID 2064223.1)
1.1.2) select * from ad_nodes_config_status; -- (prod apps entries
1.1.3) select * from fnd_oam_context_files; --(Context File-{fs1,fs2,DB} TEMPLATE,METADATA) 
1.1.4) select distinct(PATH) from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') in ('run','patch');-- check RFS/PFS context files uploaded to db or not
1.1.4) select distinct(PATH),EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') as FILE_SYSTEM from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') in ('run','patch') -- check RFS/PFS context files uploaded to db or not

1.1.5) select * from apps.TXK_TCC_RESULTS;--(etcc checkdb ran or not)
1.1.6) select THREAD#, STATUS, ENABLED from v$thread;--(disable thread 2 if not required)
1.1.7) select ad_zd.get_edition_type from dual;
1.1.8) select FND_WEB_SEC.VALIDATE_LOGIN('SYSADMIN','welcome1') from dual;--(crosscheck after FNDCPASS)
1.1.9) select * from all_objects where status='INVALID' and owner='APPS'-- and object_name='AD_ZD_ADOP';--(
1.1.9) select to_char(RESETLOGS_TIME, 'DD-MON-YYYY, hh24:mi:ss') from v$database;--(when database was cloned)

1.2) -- check schedule request.
select request_id from fnd_concurrent_requests--(Doc ID 152209.1)
where status_code in ('Q','I') --To find all scheduled requests (Doc ID 170107.1)
and requested_start_date > SYSDATE
and hold_flag = 'N';

1.3) -- check any pending wf mails
select notification_id, status, mail_status, begin_date
from WF_NOTIFICATIONS
where status = 'OPEN' and mail_status = 'MAIL';
 
2.1) --cancel schedule requests (no. of Records Updated = 4)
update fnd_concurrent_requests
set phase_code='C',
status_code='D'
where phase_code = 'P'
and (status_code = 'I' OR status_code = 'Q')
and requested_start_date >= SYSDATE
and hold_flag = 'N'
 
2.2) --cancel schedule requests (no. of Records Updated = 3)
UPDATE fnd_concurrent_requests
SET phase_code = 'C', status_code = 'X'
WHERE status_code ='T'
OR phase_code = 'R'
 
2.3) -- cancel schedule requests (no. of Records Updated = 178)
UPDATE fnd_concurrent_requests
SET hold_flag = 'Y'
WHERE phase_code = 'P'
AND status_code in ('Q','I')
 
3.1) -- update Mail status to sent (no. of Records Updated = 179)
update FND_CONCURRENT_REQUESTS set phase_code='C',status_code='D' where phase_code='P'

3.2) -- update Mail status to sent (no. of Records Updated = 30162)
select count(*) from WF_NOTIFICATIONS where status='OPEN' and mail_status='MAIL' --and message_type in ('WFTESTS','UMXLHELP') --and RECIPIENT_ROLE='MAREEF'
update WF_NOTIFICATIONS set mail_status='SENT' where status='OPEN' and MAIL_STATUS='MAIL'

3.3) -- update Mail status to sent (no. of Records Updated=0)
update WF_NOTIFICATIONS set mail_status='SENT' where status='OPEN' and MAIL_STATUS='MAIL'

3.4) -- Optional if required update Mail status to sent (no. of Records Updated=2LC)
How to Purge WFERROR (System: Error) Workflow Items? (Doc ID 804622.1)
select count(*) from wf_notifications w where  w.RECIPIENT_ROLE='SYSADMIN' and w.status='OPEN';-- to check errors after sysadmin login
--UPDATE wf_notifications set status='CLOSED' , mail_status='SENT', end_date=sysdate where recipient_role='SYSADMIN';

3.5) -- check above select statment for any pending wf mails
;
4.1) select ad_zd.get_edition_type from dual;
GET_EDITION_TYPE
-----------------------------------------------------------------------------
RUN
SQL>

4) –- If found any more from above select then perform below updates also
update wf_notifications
 set mail_status = 'SENT'
 where end_date is not null
 and status = 'CLOSED'
 and MAIL_STATUS = 'MAIL';
  
update WF_NOTIFICATIONS set mail_status = 'SENT' where mail_status = 'MAIL'; 

select notification_id, status, mail_status, begin_date 
from WF_NOTIFICATIONS
where status = 'OPEN' and mail_status = 'MAIL';

sqlplus apps/apps@db @$FND_TOP/patch/115/sql/wfntfqup.sql apps apps applsys
select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

Creation of EBSapps.env file:

applupg@ebstestapp:/u01/AHCTEST/fs2$ find . -iname EBSapps.env

find $INST_TOP/admin/ -iname EBSapps.env

/u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/admin/out/02121456/EBSapps.env

[appltest@ebstestapp ~]$ cp $INST_TOP/admin/out/02121456/EBSapps.env /u01/AHC/

[appltest@ebstestapp ~]$

[appltest@ebstestapp ~]$ vi /export/home/appltest/.profile

. /u01/AHCTEST/EBSapps.env run

[appltest@ebstestapp ~]$ ln -s /oracle/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/admin/out/09180958/EBSapps.env ~/EBSapps.env

To fix adop ETCC Codelevel checker patch

It is necessary to run check db atleast 1 time on cloned instance, else adop prepare phase will fail
[oracle@clouduatdb ~]$ ls -ltrh $ORACLE_HOME/appsutil/etcc/check*
[oracle@clouduatdb etcc]$ . $ORACLE_HOME/UATPDB_clouduatdb.env
[oracle@clouduatdb etcc]$ echo $ORACLE_SID
UATPDB


While Applying AD Patch, Getting “The following middle tier fixes have not been applied” Even After Applying ETCC Patches (Doc ID 2958876.1)
select * from applsys.TXK_TCC_RESULTS;
truncate table applsys.TXK_TCC_RESULTS;

[oracle@clouduatdb etcc]$ unzip p17537119_R12_GENERIC.zip
oraar@ebstestdb:~ $ sh $ORACLE_HOME/appsutil/etcc/checkDBpatch.sh ß mandatory, if not adop will fail when ever used on instance
[appltest@ebstestapp ~]$ ./checkMTpatch.sh

col CHECK_MESSAGE for a55;
select a.component_name,a.check_date,a.check_message from applsys.TXK_TCC_RESULTS a order by a.check_date;

adop check

[appltest@ebstestapp ~]$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.11)
Session Id: 14
Command: status
Output: /u01/AHCTEST/fs_ne/EBSapps/log/adop/14/20210212_170040/adzdshowstatus.out
===============================================================

File System Synchronization Type: Full

adop exiting with status = 0 (Success)
[appltest@ebstestapp ~]$

ADOP apply phase always fails if option is no for the below
Adop apply fails with " Do you wish to apply this patch now [No] ? No *" (Doc ID 2355751.1)
[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ grep -A 6 MISSING_TRANSLATED_VERSION  $APPL_TOP/admin/${TWO_TASK}/adalldefaults.txt
       MISSING_TRANSLATED_VERSION
  %%END_OF_TOKEN%%

  %%START_OF_VALUE%%
       Yes
  %%END_OF_VALUE%%
## End of Defaults Record
[appltest@ebstestapp ~]$
if ADOP Cleanup Fails with below error refer doc
Cleanup Phase Is Successful But Has Some Errors And Warnings In The Log File: [WARNING] Could not drop covered object (Doc ID 2496457.1)
--Getting Error Could not drop covered object while Running Adop Cleanup phase (Doc ID 2511377.1) Alter session set edition = V_20190109_1434;
--select * from dba_editions
select co.edition_name, co.owner, co.object_type, co.object_name --count(*)
from dba_objects_ae co, database_properties run /* run edition name */
where run.property_name = 'DEFAULT_EDITION'
and co.object_type <> 'NON-EXISTENT'
and co.edition_name is not null
and co.edition_name < run.property_value
and exists
( select null from dba_objects_ae ro /* replacement object */
where ro.owner = co.owner
and ro.object_name = co.object_name
and ro.namespace = co.namespace
and ro.edition_name > co.edition_name
and ro.edition_name <= run.property_value )
and not exists
( select null from sys.dependency$ dep /* dependents */
where dep.p_obj# = co.object_id )
order by co.edition_name desc, co.owner, co.object_name;





Verify User Profile
OraInventory & Opatch:

root@ebstestapp:~# su - applmgr

Oracle Corporation SunOS 5.11 11.4 October 2020

#########################

# EBS TEST APP #

#########################

You have new mail.

E-Business Suite Environment Information

----------------------------------------

RUN File System : /u01/AHCTEST/fs1/EBSapps/appl

PATCH File System : /u01/AHCTEST/fs2/EBSapps/appl

Non-Editioned File System : /u01/AHCTEST/fs_ne

DB Host: ebstestapp.areefdba.com.sa Service/SID: AHCTEST

Sourcing the RUN File System ...

[appltest@ebstestapp ~]$

[appltest@ebstestapp ~]$ which opatch

/u01/AHCTEST/fs1/EBSapps/10.1.2/OPatch/opatch

[appltest@ebstestapp ~]$

[appltest@ebstestapp ~]$ opatch lsinventory

Oracle Interim Patch Installer version 1.0.0.0.64

OPatch succeeded.

[appltest@ebstestapp ~]$

[appltest@ebstestapp ~]$ $ORACLE_HOME/OPatch/opatch lsinventory

. . . . .

OPatch succeeded.

[appltest@ebstestapp ~]$

Run File System:

Verify User Profile OraInventory & Opatch:

Central Inventory should update 8 more IDX home in inventory.xml file

root@ebstestapp:~#

root@ebstestapp:~# cat /u01/AHCTEST/fs1/EBSapps/10.1.2/oraInst.loc

inventory_loc=/export/home/applahc/oraInventory

inst_group=dba

root@ebstestapp:~#

[appltest@ebstestapp ~]$ cat /export/home/applahc/oraInventory/ContentsXML/inventory.xml

New Clone Instance IDX are added to Central Inv. As per given Local Inv Location)

. . . . .

<HOME NAME="OH200942157" LOC="/u01/AHCTEST/fs1/FMW_Home/oracle_common" TYPE="O" IDX="9">

</REFHOMELIST>

</HOME>

<HOME NAME="OH826029475" LOC="/u01/AHCTEST/fs1/FMW_Home/webtier" TYPE="O" IDX="10">

</DEPHOMELIST>

</HOME>

<HOME NAME="OH634527727" LOC="/u01/AHCTEST/fs1/FMW_Home/Oracle_EBS-app1" TYPE="O" IDX="11"/>

<HOME NAME="AHCTEST_TOOLS__u01_AHCTEST_fs1_EBSapps_10_1_2" LOC="/u01/AHCTEST/fs1/EBSapps/10.1.2" TYPE="O" IDX="12"/>

<HOME NAME="OH267948910" LOC="/u01/AHCTEST/fs2/FMW_Home/oracle_common" TYPE="O" IDX="13">

</REFHOMELIST>

</HOME>

<HOME NAME="OH1823101828" LOC="/u01/AHCTEST/fs2/FMW_Home/webtier" TYPE="O" IDX="14">

</DEPHOMELIST>

</HOME>

<HOME NAME="OH665547534" LOC="/u01/AHCTEST/fs2/FMW_Home/Oracle_EBS-app1" TYPE="O" IDX="15"/>

<HOME NAME="AHCTEST_TOOLS__u01_AHCTEST_fs2_EBSapps_10_1_2" LOC="/u01/AHCTEST/fs2/EBSapps/10.1.2" TYPE="O" IDX="16"/>

root@ebstestapp:~#

root@ebstestapp:~# ps -ef |grep -i applahc|wc -l

Sample steps to re-create appsutil directory for database:

IF necessary to recreate appsutil directory follow here

apploci@ebsprodapp3:~$ perl $AD_TOP/bin/admkappsutil.pl

apploci@ebsprodapp3:~$ scp $INST_TOP/admin/out/appsutil.zip oraoci@192.168.4.12:/u01/app/oraoci/product/12.1.0.2/dbhome_1

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1$ mv appsutil appsutil_old

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1$ unzip -o appsutil.zip

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil_old$ cp jre-7u251-solaris-sparc*.tar.gz /u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil

Create jre folder from previous appsutil directory:

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil$ tar -xvzf jre-7u251-solaris-sparc.tar.gz

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil$ tar -xvzf jre-7u251-solaris-sparcv9.tar.gz

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil$ mv jre1.7.0_251 jre

Create Contextfile for Database Tier:

oraoci@ebsproddb1:~$ cd $ORACLE_HOME/appsutil/bin

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/bin$

perl $ORACLE_HOME/appsutil/bin/adbldxml.pl appsuser=apps appspass=appspswd template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp out=$ORACLE_HOME/appsutil/PRODPDB_ebsproddb1.xml

Enter Database Service Name: PRODPDB

Enter the value for Display Variable: :0.0

Create Scripts directory:

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/bin$ sh adconfig.sh contextfile=$ORACLE_HOME/appsutil/PRODPDB_ebsproddb1.xml

Run AutoConfig on the <RDBMS ORACLE_HOME>:

perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh

oraoci@ebsproddb1:/u01/app/oraoci/product/12.1.0.2/dbhome_1/appsutil/scripts/PRODPDB_ebsproddb1$ adautocfg.sh

Normal Application Cloning Ends here

APPS Tier Cloning is Completed Successfully and achieved below:

- Successful run Auto config on Patch file System without ERROR during adcfgclone.

- Inventory Issue Resolved

- ADOP issue resloved

- Weblogic Deny Line issue resolved

Remove https(ssl) setting form context file before autoconfig:
IF necessary take backup of xml file for reference:
before running autoconfig, when if updated xml variable sslterminator then autoconfig failed
[applmgr@ahcdevcloudapp ~]$ cp $CONTEXT_FILE ~


If required change apps password, then update in weblogic also:
After FNDCPASS
[appltest@ebstestapp ~]$ cd $FND_TOP/bin
[appltest@ebstestapp bin]$ sqlplus system/manager
[appltest@ebstestapp bin]$ 
[appltest@ebstestapp bin]$ FNDCPASS apps/appspswd 0 Y system/manager SYSTEM APPLSYS ahctestm7
[appltest@ebstestapp bin]$ sqlplus apps/ahcsctestm7
[appltest@ebstestapp bin]$ 
[appltest@ebstestapp bin]$ history -d 1030 (to delete apps prod password from history)

IF EBS Database is CDB/PDB then source PDB before running autoconfig:
[oracle@ahcdevdb ~]$ . $ORACLE_HOME/DEVPDB_ahcdevdb.env
[oracle@ahcdevdb ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh

Run autoconfig on db tier first then on apps tier (no need on Patch FS.)
nextime try to update context variable before running autoconfig on apps to save time 


Deny Line for weblogic access,
ports check for application and Weblogic in OCI servers
Open port for Newly created Apps(Compute) in OCI:  (only on new server installation or new/extra APPS Port)
sudo firewall-cmd --list-all
systemctl status firewalld
sudo firewall-cmd --permanent --add-port=8001/tcp
sudo firewall-cmd --reload
now start the services…..

also open the weblogic console port….
sudo firewall-cmd --permanent --add-port=7002/tcp
sudo firewall-cmd –reload

[appltest@ebstestapp ~]$
[appltest@ebstestapp ~]$ grep deny $EBS_DOMAIN_HOME/config/config.xml  ß deny
    <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule> 

Weblogic Admin server access with deny line: (replace existing line with below)
 
apploci@ebstestapp:~$ grep s_wls_admin_console_access_nodes $CONTEXT_FILE
         <s_wls_admin_console_access_nodes oa_var="s_wls_admin_console_access_nodes">192.168.140.200</s_wls_admin_console_access_nodes>
apploci@ebstestapp:~$
apploci@ebstestapp:~$
         <s_wls_admin_console_access_nodes oa_var="s_wls_admin_console_access_nodes"/>

cp $CONTEXT_FILE ~
grep s_wls_admin_console_access_nodes $CONTEXT_FILE
<s_wls_admin_console_access_nodes oa_var="s_wls_admin_console_access_nodes">192.168.140.200</s_wls_admin_console_access_nodes> 
grep -i 'sslterminator\|https\|sitename' $CONTEXT_FILE
<sslterminator oa_var="s_enable_sslterminator" customized="yes">#</sslterminator>

grep site_name $CONTEXT_FILE
grep s_wfmail_agent $CONTEXT_FILE
grep s_webhost $CONTEXT_FILE
grep s_domainname $CONTEXT_FILE
grep s_webport $CONTEXT_FILE
grep s_login_page $CONTEXT_FILE
Ex: http://myhost.mydomain.com:8000

apploci@ebstestapp:~$ adautocfg.sh
apploci@ebstestapp:~$ adadminsrvctl.sh stop
apploci@ebstestapp:~$ adadminsrvctl.sh start
 
apploci@ebstestapp:~$ grep deny $EBS_DOMAIN_HOME/config/config.xml
    <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>
apploci@ebstestapp:~$

[appltest@ebstestapp ~]$ ps -ef |grep -i applmgr |wc -l
[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ adadminsrvctl.sh start  ß approx. 2 Mins

grep login $CONTEXT_FILE
grep wls_admin $CONTEXT_FILE

grep url_protocol $CONTEXT_FILE
grep s_webhost $CONTEXT_FILE
grep s_wls_admin_domain $CONTEXT_FILE
grep s_wls_adminport $CONTEXT_FILE

Weblogic Console Url with echo command:
echo "http://"$(cat $CONTEXT_FILE | grep s_webhost | cut -d '>' -f2 | cut -d '<' -f1)"."$(cat $CONTEXT_FILE | grep s_wls_admin_domain | cut -d '>' -f2 | cut -d '<' -f1)":"$(cat $CONTEXT_FILE | grep s_wls_adminport | cut -d '>' -f2 | cut -d '<' -f1)"/console"

http://ebstestapp.areefdba.com.sa:7004/console

Method-1:
Console --> Domain Structure --> Services --> Data Source --> EBSDataSource --> Connection Pools --> Password

Method-2:
[appltest@ebstestapp ~]$ perl $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl
Please select from list of valid options
        updateDSPassword - Update WebLogic Datasource Password
        updateDSJdbcUrl  - Update WebLogic Datasource Connection String
Enter Your Choice : updateDSPassword
Enter the full path of Applications Context File [DEFAULT - /u01/UAT/fs2/inst/apps/UAT_uatapp01/appl/admin/UAT_uatapp01.xml]:
Enter weblogic admin server password:

Enter the APPS user password: apps_new_password
---------------------------------------------
Performing Steps for option=updateDSPassword
---------------------------------------------
Updating WLS DataSource Password
--------------------------------
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Connecting to t3://areefdba.sa:7011 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'EBS_domain_UAT'.
Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Domain Update successful
------------------------
[appltest@ebstestapp ~]$

If required to update JVM Mem. for OACORE
- oacore_server1 --> (tab)Server Start --> Arguments: -Xms2048m -Xmx4096m
adnodemgrctl.sh stop/start oacore_server1
ps -ef | grep oacore_server1
ps -ef | grep Xmx4096
Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2 (Doc ID 1905593.1)
Note: For the oacore service, Oracle recommends up to 150-200 concurrent users per JVM of size 2 GB.

[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ adstrtal.sh apps/apps (worked by start apps after pswd updated in WLC with out stoping admin server to same time)
[appltest@ebstestapp ~]$ adadminsrvctl.sh stop 

[appltest@ebstestapp ~]$ ps -ef |grep -i applmgr|wc -l 
[appltest@ebstestapp ~]$


Important Variable of
Context file which need update after clone:

1. Context File variables update for Regular Clone.
cp $CONTEXT_FILE ~
grep s_wls_admin_console_access_nodes $CONTEXT_FILE 
grep sslterminator $CONTEXT_FILE
grep site_name $CONTEXT_FILE
grep s_wfmail_agent $CONTEXT_FILE
grep s_frmNetworkRetries $CONTEXT_FILE
grep s_forms_catchterm $CONTEXT_FILE
grep s_wls_admin_console_access_nodes $CONTEXT_FILE

grep deny $EBS_DOMAIN_HOME/config/config.xml
echo $FORMS_WEB_CONFIG_FILE  (or) grep networkRetries $FORMS_WEB_CONFIG_FILE
grep networkRetries $ORA_CONFIG_HOME/10.1.2/forms/server/appsweb.cfg
echo $FORMS_CATCHTERM 
(deny line, networkRetries=30, catchterm=0)(Doc ID 745711.1) (Doc ID 744143.1) 

grep sslterminator $CONTEXT_FILE  ß update # to stop redirecting to https in login page
grep deny $EBS_DOMAIN_HOME/config/config.xml  (for weblogic access page)
grep networkRetries $ORA_CONFIG_HOME/10.1.2/forms/server/appsweb.cfg  (value 30)
grep FORMS_CATCHTERM $ORA_CONFIG_HOME/10.1.2/forms/server/default.env  (value 0)
grep s_wls_admin_console_access_nodes $CONTEXT_FILE   (provide access to Weblogic console with a static IP PC)

###############################################################################################
###############################################################################################
2. Context File variables update for Workflow mailer Account update from itinfocloudp to itinfocloudt

cp $CONTEXT_FILE ~
grep -A 12 "<oa_workflow_server>" $CONTEXT_FILE  (List all values of WFM like user,smtp,imap)

grep s_javamailer_imaphost $CONTEXT_FILE
grep s_javamailer_imapdomainname $CONTEXT_FILE
grep s_wf_admin_role $CONTEXT_FILE

grep s_javamailer_reply_to $CONTEXT_FILE
grep s_javamailer_imap_user $CONTEXT_FILE
grep s_javamailer_outbound_user $CONTEXT_FILE

grep s_smtphost $CONTEXT_FILE
grep s_smtpdomainname $CONTEXT_FILE

grep s_wfmail_agent $CONTEXT_FILE  (for WFM Profile)
grep s_endUserMonitoringURL $CONTEXT_FILE
grep s_webhost $CONTEXT_FILE
grep s_domainname $CONTEXT_FILE
grep s_webport $CONTEXT_FILE
echo "http://"$(cat $CONTEXT_FILE | grep s_webhost | cut -d '>' -f2 | cut -d '<' -f1)"."$(cat $CONTEXT_FILE | grep s_domainname | cut -d '>' -f2 | cut -d '<' -f1)":"$(cat $CONTEXT_FILE | grep s_webport | cut -d '>' -f2 | cut -d '<' -f1)
grep s_wfmail_agent $CONTEXT_FILE
grep s_login_page $CONTEXT_FILE
Ex: http://myhost.mydomain.com:8000

Workflow Mailer Profile (Doc ID 2493004.1)
Make Sure if you have load balancer configured for the application, then you have to provide site value as per your primary node and remove "s" from https if ssl is also configured use below variable of context file and build value as shown in example:
grep s_wfmail_agent $CONTEXT_FILE   (for WFM Profile)
grep s_webhost $CONTEXT_FILE
grep s_domainname $CONTEXT_FILE
grep s_webport $CONTEXT_FILE
Ex: http://myhost.mydomain.com:8000
WF: Workflow Mailer Framework Web Agent
Site: URL without s in https: grep login $CONTEXT_FILE

--A good index should return 2 rows, 1 for the CORRID, 1 for the MSGID on table column level per table(IN, OUT)
select * from all_ind_columns where table_name in ('WF_NOTIFICATION_IN','WF_NOTIFICATION_OUT');--,'WF_NOTIFICATIONS');--to check index over table columns

###############################################################################################
###############################################################################################
3. Context File variables update for DMZ configuration multi Node:
click here to add application node & DMZ server setup

Oracle E-Business Suite Release 12.2 Configuration in a DMZ (Doc ID 1375670.1)

cp $CONTEXT_FILE ~  (Master node contextfile, always bkp before modiying)
hostname –f  (Master node fqdn)
 
Shared storage for EBS R12.2 Server
grep s_shared_file_system $CONTEXT_FILE

Services need to be disabled on DMZ Server
grep s_batch_status $CONTEXT_FILE
grep s_adminserverstatus $CONTEXT_FILE
grep s_web_admin_status $CONTEXT_FILE
 
For Weblogic Ports Check:
grep s_atName $CONTEXT_FILE
grep wls_oacoreport $CONTEXT_FILE
grep wls_oafmport $CONTEXT_FILE
grep wls_formsport $CONTEXT_FILE
grep wls_forms-c4wsport $CONTEXT_FILE
grep s_adminserverstatus $CONTEXT_FILE (to check primary node from all EBS nodes)
Only the WLS Admin Server/Primary node will have this enabled.
This is the master node from where you run adop.

find $IAS_ORACLE_HOME/instances -iname EBS_web*.log (check multinode status)
find $IAS_ORACLE_HOME/instances -iname apps.conf  (Multi-node Server config detils file)
find $IAS_ORACLE_HOME/instances -iname mod_wl_ohs.conf

find $IAS_ORACLE_HOME/instances -type f \( -iname \apps.conf -o -iname \mod_wl_ohs.conf \)

###############################################################################################
###############################################################################################
4. IF TEST EBS DB is RAC the imp. Varaibles for DB & Apps:

RAC TEST DB Context variables
grep remote_listener $CONTEXT_FILE
grep cluster_database $CONTEXT_FILE
grep scanName $CONTEXT_FILE
grep scanPort $CONTEXT_FILE
grep scanUpdateFlag $CONTEXT_FILE


Apps Context variables for TEST RAC DB
grep s_dbhost $CONTEXT_FILE
grep cmanhost $CONTEXT_FILE
grep s_tools_twotask $CONTEXT_FILE
grep s_weboh_twotask $CONTEXT_FILE
grep s_apps_jdbc_connect_alias $CONTEXT_FILE
grep CP_TWOTASK $CONTEXT_FILE


###############################################################################################
###############################################################################################
5. weblogic Cluster, MS and their respective ports of MS
grep _cluster $CONTEXT_FILE
grep oacore_server $CONTEXT_FILE
grep oa_managed_server $CONTEXT_FILE
grep _server_ports $CONTEXT_FILE

[appltest@ebstestapp ~]$ grep _cluster $CONTEXT_FILE  (list all clusters registered in weblogic)
[appltest@ebstestapp ~]$ grep oacore_server $CONTEXT_FILE
[appltest@ebstestapp ~]$ grep oa_managed_server $CONTEXT_FILE (lists all types of MS)
[appltest@ebstestapp ~]$ grep _server_ports $CONTEXT_FILE (list all MS along with ports)
[appltest@ebstestapp ~]$

###############################################################################################
###############################################################################################
6. if PROD has SSO, then disable after Clone else fsclone will failwith below erros:
[applmgr@ebsprodapp3 ~]$ find $ADOP_LOG_HOME/30 -iname FSCloneApplyAppstier*.log
1. ERROR: Managed Server's are not in sync between file system context and DB context
2. Error while deploying application accessgate

grep s_oaea_managed_servers $CONTEXT_FILE
grep s_oaea_clustername $CONTEXT_FILE
[appltest@ebstestapp ~]$ grep _cluster $CONTEXT_FILE
[appltest@ebstestapp ~]$ grep oa_managed_server $CONTEXT_FILE

- Verify Managed Servers List in WebLogic console as per application Context file
and delete unwanted/additional MS/Cluster configured from WL Console which not attached to any machines (where machine name is empty)

- Delete accessgate deployment from Weblogic console which was created for PROD SSO
follow SSO disable steps here
Cloning Oracle E-Business Suite Release 12.2 Environments integrated with Oracle Access Manager 11gR2 (11.1.2) and Oracle E-Business Suite AccessGate (Doc ID 1614793.1)
Integrating Oracle E-Business Suite Release 12.2 with Oracle Internet Directory 11gR1 (Doc ID 1371932.1)

1. Disable SSO profile
APPS_SSO_LDAP_SYNC - PROFILE NAME is :  Application SSO LDAP Synchronization

2. Remove all references to the original OID/SSO instance.
$FND_TOP/bin/txkrun.pl -script=SetSSOReg -removereferences=Yes

3. weblogic Console delete below all SSO configuration
- Envinoment --> Managed Servers (OAEA)
- Envinoment --> Clusters (oaea_cluster)
- Services --> Data Sources (OAEADataSource)
- EBS_domain_UAT --> Deployments --> accessgate

4. Delink all Oracle E-Business Suite users that were linked to the original Oracle Internet Directory users
[applmgr@uatapp01 ~]$ time sqlplus apps @$FND_TOP/patch/115/sql/fndssouu.sql %

5. Remove the managed server and port from the mod_wl_ohs.conf configuration:
copy a backup of below .conf files
find $IAS_ORACLE_HOME/instances -type f \( -iname \apps.conf -o -iname \mod_wl_ohs.conf \)
grep s_wls_oaeaport $CONTEXT_FILE

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -accessgate=http://uatapp01.areefdba.edu.sa:7011

###############################################################################################
###############################################################################################
7. if autoconfig failes with error:
[Autoconfig Fails With "Timed out( 300000 ): Interrupted Exception" Error (Doc ID 2177265.1)
[appltest@ebstestapp ~]$ free -h
[appltest@ebstestapp ~]$ ls -ltrh $INST_TOP/admin/log/*/adconfig.log

###############################################################################################
###############################################################################################
8. if Multinode EBS for Parllel Concurrent Processing (PCP):

[appltest@ebstestapp ~]$ grep s_appldcp $CONTEXT_FILE
         <APPLDCP oa_var="s_appldcp">ON</APPLDCP>

Profile Options at Site Level:
Concurrent:TM Transport Type --> QUEUE
Concurrent:PCP Instance Check --> ON

[appltest@ebstestapp ~]$


You always need to run autoconfig again if updated any further in xml file


Start Application Services with new password: 
[appltest@ebstestapp ~]$ ps -ef |grep -i applmgr|wc -l 
[appltest@ebstestapp ~]$ grep deny $EBS_DOMAIN_HOME/config/config.xml

[appltest@ebstestapp ~]$ adstrtal.sh apps/ahcsctestm7   ß New Password approx. 4 Mins

[appltest@ebstestapp ~]$  ps -ef |grep -i applmgr|wc -l 
310
[appltest@ebstestapp ~]$ adcmctl.sh status apps/ahcsctestm7

You are running adcmctl.sh version 120.19.12020000.7

Internal Concurrent Manager is Active.

adcmctl.sh: exiting with status 0


adcmctl.sh: check the logfile /u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/logs/appl/admin/log/adcmctl.txt for more information ...

[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ adopmnctl.sh status

You are running adopmnctl.sh version 120.0.12020000.2

Checking status of OPMN managed processes...

Processes in Instance: EBS_web_OHS1
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
EBS_web                          | OHS                |    9370 | Alive


adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /u01/AHCTEST/fs2/inst/apps/AHCTEST_ebstestapp/logs/appl/admin/log/adopmnctl.txt for more information ...

[appltest@ebstestapp ~]$

If you got any error like below while opening forms, Stop apps then recreate jar files from adadmin options(1---> 4 Generate product JAR files) (approx 30 Mins for jar files generation)
E-Business Suite R12.2 : Using Java Web Start Get Error 'JAR Resources In JNLP File Are Not Signed By Same Certificate' (Doc ID 2464716.1)
[appltest@ebstestapp ~]$ adstpall.sh apps/appspswd
[appltest@ebstestapp ~]$ adadmin

  1.    Generate Applications Files menu
  4.    Generate product JAR files
Do you wish to force regeneration of all jar files? [No] ? YES
tailf $NE_BASE/EBSapps/log/adadmin/log/adadmin.log

Once Start the Application Change Sys pswd and Reduce Conc Mngr:
 
 If Source instance was configured with SSL then remove setting on Cloned instance
[appltest@ebstestapp ~]$
[appltest@ebstestapp ~]$ grep -i 'sslterminator\|https' $CONTEXT_FILE (not worked at this step need to check again ssl setting before running autoconfig replace with below line or add yellow)
<sslterminator oa_var="s_enable_sslterminator" customized="yes">#</sslterminator>

or else update whole below line
<sslterminator oa_var="s_enable_sslterminator" customized="yes">#</sslterminator>
<sslterminator oa_var="s_enable_sslterminator" customized="yes">#</sslterminator>

Manually reach near to autoconfig update from html page:
http://ebstestapp.areefdba.com.sa:8002/OA_HTML/weboam/oam/adconfig/adAppsCtxtFilesTable$target=AHCTEST
[applmgr@ebstestapp ~]$


[appltest@ebstestapp ~]$
[appltest@ebstestapp ~]$ ps -ef |grep -i applahc|wc -l
310
[appltest@ebstestapp ~]$
[appltest@ebstestapp ~]$ cd $FND_TOP/bin
[appltest@ebstestapp bin]$ FNDCPASS apps/appspswd 0 Y system/manager USER SYSADMIN welcome1
[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ echo $APPLTMP
Update Temp Directory in Application from front end
XML PUBLISHER AdministratoràAdministrationà Generalà Temporary directoryà Site Value: echo $APPLTMP

-	oacore_server1 (-Xms2048m -Xmx4096m)
-	Sysadmin password expiry to none
-	Standard Manager
-	SCHEDULE MANAGER
-	Order Management Custom -SRS
-	Java Color Scheme  (teal-dev, red-uat, khaki-pdbtest)
-	WF: Workflow Mailer Framework Web Agent  (mailer profile)
-	 - TEST WFM MAIL update correct mailer password
-	Clear Cache (Functional
Administrator)
-	echo $APPLTMP (XML
Pub.– Home – Administration)
-	Change Sysadmin Password (Using
FNCPASS)
-	Check DB Invalid Objects 
-	Corporate Branding Image for Oracle Applications (if
Logo is used)
-	alter user system identified by manager;
-	site name, https & other variables update in xml
-	OACORE Addition or deleteion  (Doc ID 1905593.1)

Alerts to be disabled:
-	TA_Missed_IN_Alert 
-	TA_Missed_OUT_Alert 
-	ahc_Iqama_Alert 
-	AHC_Emp_Termination_Daily 
-	AHC_Hired_Employee_info 


[appltest@ebstestapp ~]$
[appltest@ebstestapp ~]$ ps -ef |grep -i appltest|wc -l  
144
[appltest@ebstestapp ~]$ 

Update JAVA Color from query and clear cache to take effect.
UPDATE fnd_profile_option_values erm
SET erm.PROFILE_OPTION_VALUE='RED'--BLUE,KHAKI,OLIVE,PURPLE,RED,STANDARD,TEAL,TITANIUM
WHERE erm.PROFILE_OPTION_ID in ( SELECT erm2.PROFILE_OPTION_ID
FROM FND_PROFILE_OPTIONS_TL erm1,FND_PROFILE_OPTIONS erm2
WHERE erm1.USER_PROFILE_OPTION_NAME = 'Java Color Scheme' and
erm2.PROFILE_OPTION_NAME = erm1.PROFILE_OPTION_NAME);


If required upload missing RFS/PFS context filies entries to database in fnd_oam_context_files table 
SQL> select distinct(PATH),EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') as FILE_SYSTEM from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') in ('run','patch') -- check RFS/PFS context files uploaded to db or not


. ~/EBSapps.env Patch
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=$CONTEXT_FILE logfile=~/patchctxupload.log



If required update Workflow mailer details:
System administrator -> Oracle Applications Manager -> Workflow -> Site Map -> Notification Mailer -> Workflow Notification Mailer -> Edit

Mail Account with MS Cloud (29-06-2021)
Outbound EMail Account    
Server Name Host Name of the machine: smtp.office365.com
Inbound EMail Account
Server Name Host Name of the machine: outlook.office365.com
Username for the Outbound Server: itinfocloudp@areefdba.com
Password for Outbound Server (SMTP): passwords XXXX
Reply-To Address The address of the e-mail: itinfocloudp@areefdba.com
Reply Connection Security: STARTTLS


Bursting Program for Pay Slip with cloud email account (Salary slip.PDF Atch)-->
Senario-2 Bursting Program for Pay Slip (Salary slip.PDF Atch)-->
-XML Pub. Admin--> Home--> Data Definition->Code (Executable Name): 
XXAHC_PAY_SLIP_XML
XXAJAR_PAY_SLIP_XML
XXAHC_PAY_CASH_SLIP_XML

-Bursting Control File: XML template, Update Location of (.rtf) in XML
<xapi:email id="GradeMsgId"
server="smtp.office365.com" port="587" from="itinfocloudt@areefdba.com.sa">
<xapi:template type="rtf" location="/u01/AHCTEST/fs1/inst/apps/AHCTEST_ebstestapp/appltmp/Salary_Template_Design.rtf"/>

[appltest@ebstestapp ~]$ cp Salary_Template_Design.rtf $APPLTMP
[appltest@ebstestapp ~]$ 
[appltest@ebstestapp ~]$ cat $XDO_TOP/resource/xdodelivery.cfg
[appltest@ebstestapp ~]$ vi $XDO_TOP/resource/xdodelivery.cfg
<?xml version="1.0" encoding="UTF-8" ?>
<config xmlns="http://xmlns.oracle.com/oxp/delivery/config">
<servers>
<server name="smtpserver1" type="smtp_email" default="true">
<host>smtp.office365.com</host>
<port>587</port>
<secureConnection>tls</secureConnection>
<username>itinfocloudt@areefdba.com.sa</username>
<password>XXXXXXXX</password>
</server>
</servers>
<properties>
<property name="ds-temp-dir">/u01/AHC/fs2/inst/apps/AHCTEST_ebstestapp/appltmp</property>
<property name="ds-buffering">true</property>
<property name="SMTP_CONTENT_TYPE:String">"application/pdf"</property>
</properties>
</config>
[appltest@ebstestapp ~]$ vi $XDO_TOP/resource/xdodelivery.cfg
[appltest@ebstestapp ~]$
-Conc.Program: AHC Employee Salary Bank Payment Slip(XML)


Apply Latest ETCC Patch on DB and AppsTier for any missing patch:
oratest@ebstestdb:ETCC]$ unzip p17537119_R12_GENERIC.zip -d /u01/app/oratest/ETCC
oratest@ebstestdb:ETCC]$ ./checkDBpatch.sh
[appltest@ebstestapp ETCC]$ ./checkMTpatch.sh


Sqlnet.ora parameters for OCI DB for RDBMS Home & Client Home:
Update the sqlnet.ora in $ORACLE_HOME/network/admin/sqlnet.ora
Update encrypt location of target by comparing source
#################################
# Definition for CDB sqlnet.ora
#################################
#ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jxxx)))
#ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/AHCCDEV_jxxx)))
 
SQLNET.ENCRYPTION_SERVER=REQUIRED
SQLNET.CRYPTO_CHECKSUM_SERVER=REQUIRED
SQLNET.ENCRYPTION_TYPES_SERVER=(rc4_40,AES256,AES192,AES128)#(Doc ID 2777856.1)
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=(SHA1)
#--- Client sqlnet.ora paramters
#SQLNET.ENCRYPTION_CLIENT=REQUIRED
#SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED
#SQLNET.ENCRYPTION_TYPES_CLIENT=(rc4_40,AES256,AES192,AES128)
#SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=(SHA1)
 
NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME)
SQLNET.EXPIRE_TIME=10
SQLNET.INBOUND_CONNECT_TIMEOUT=60
SQLNET.ALLOWED_LOGON_VERSION_SERVER=10
 
IFILE=/u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/sqlnet_ifile.ora

###################################################################################################
Important sql Commands to check users session in EBS Multinode Architecture
###################################################################################################
-- Count of Active Users per Application node
col NODE_NAME for a25;
col HOW_MANY_USER_SESSIONS for a35;
select a.node_name, 'Number of user sessions : ' || count(distinct b.session_id) How_many_user_sessions 
from apps.fnd_nodes a,apps.icx_sessions b
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'),
NULL,limit_time, 0,limit_time,apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate
and a.node_id=b.node_id and counter < limit_connects group by a.node_name;

NODE_NAME                 HOW_MANY_USER_SESSIONS
------------------------- -----------------------------------
EBSTESTAPP                Number of user sessions : 46

-- Number of Active Users Details as per organization

select a.node_name,u.USER_NAME,u.USER_ID,pap.EMPLOYEE_NUMBER,pap.FULL_NAME,org.NAME,pay.payroll_name,a.LAST_MONITORED_TIME,u.LAST_LOGON_DATE

from apps.fnd_nodes a,apps.icx_sessions b,fnd_user u,per_all_people_f pap,hr_all_organization_units org,per_all_assignments_f paa,pay_all_payrolls_f pay

where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'

and org.organization_id(+)=paa.organization_id

and paa.PERSON_ID = pap.PERSON_ID

and sysdate between paa.EFFECTIVE_START_DATE and paa.EFFECTIVE_END_DATE

and (last_connect + decode(apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'),

NULL,limit_time,0,limit_time,apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24)>sysdate

and a.node_id=b.node_id and counter < limit_connects

and u.USER_ID = b.USER_ID

and sysdate between pap.EFFECTIVE_START_DATE and pap.EFFECTIVE_END_DATE

and u.EMPLOYEE_ID = pap.PERSON_ID

and pay.payroll_id(+) = paa.payroll_id

and pay.effective_end_date(+)>sysdate

order by pay.payroll_name, org.name, pap.employee_number

grep command to search a error word from same file name from multiple folders

[oracle@rac1 ~]$ ls -ltrh $INST_TOP/admin/log/*/adconfig.log

[oracle@rac1 ~]$ grep -r Interrupted $INST_TOP/admin/log/*/adconfig.log

How to Start/Stop EBS services in Oracle Apps R12.2 on all app nodes through single command

[applmgr@ebstestapp ~]$ adstpall.sh -mode=allnodes apps/apps_password

[applmgr@ebstestapp ~]$ adstrtal.sh -mode=allnodes apps/apps_password

[applmgr@ebstestapp ~]$ adstrtal.sh -msimode apps/apps_password

[applmgr@ebstestapp ~]$ adstpall.sh -mode=allnodes -skipNM -skipAdmin apps/apps

[applmgr@ebstestapp ~]$ adstrtal.sh -mode=allnodes -skipNM -skipAdmin apps/apps

[applmgr@ebstestapp ~]$ ls -ltrh $INST_TOP/logs/appl/admin/log/adst*

need to verify can be deleted unwanted other than clone folder or not:
ls -ltrh $RUN_BASE/EBSapps/comn/*/FMW/FMW_Home.jar
ls -ltrh $PATCH_BASE/EBSapps/comn/*/FMW/FMW_Home.jar

###################################################################################################

APPS Tier Cloning is Completed Successfully:

Log Files Location to look for any errors after Clone:
 -bash-5.
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/EBS_domain.log
tail -100f $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.out
tail -100f $FMW_HOME/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.log
tail -100f $IAS_ORACLE_HOME/instances/EBS_web_OHS1/diagnostics/logs/OPMN/opmn/opmn.log
tail -100f $IAS_ORACLE_HOME/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log

Verify list of applied patches on Oracle Homes & JDK/JRE Version :

oramgr@ebstestdb:~ $

$ORACLE_HOME/OPatch/opatch lspatches

$ORACLE_HOME/OPatch/opatch lsinventory |grep applied

$ORACLE_HOME/OPatch/opatch lsinventory |grep description

Check Patch Conflict methods with current applied patches to HOME:

1. Manual Command to check patch Conflict is go to patch folder and run:

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./

2. Run patch conflict check online, with option "Analyze with opatch" prior downloading patch (useful if having Big patch, check conf. without downloading patch)

$ORACLE_HOME/OPatch/opatch lsinventory > Analyze_with_OPatch_Online.txt

- click Analyze with opatch (near Download Button)

- Attach/upload the above txt file to the patch with which need to check conflicts

- click "Analyze for Conflict"

###############################################################################################
###############################################################################################

-- Oracle_Home , Middel ware Home Weblogic Applied Patches check

-- EBS HOME 1 (EBS Apps Home 10.1.2)

echo $ORACLE_HOME

/u01/OCI/fs1/EBSapps/10.1.2

$ORACLE_HOME/OPatch/opatch lsinventory | grep applied

-- EBS HOME 2 (Middel ware home version, applied patches)

echo $OHS_VERSION

11.1.1.9.0

echo $OHS_SUB_VERSION

11.1.1.9

export ORACLE_HOME=$IAS_ORACLE_HOME

export ORACLE_HOME=$FMW_HOME/webtier

$ORACLE_HOME/OPatch/opatch lsinventory | grep WebTier

Oracle WebTier and Utilities CD 11.1.1.9.0

$ORACLE_HOME/OPatch/opatch lsinventory | grep applied

-- EBS HOME 3 : (oracle common home)

export ORACLE_HOME=$FMW_HOME/oracle_common

$ORACLE_HOME/OPatch/opatch lsinventory | grep Common

Oracle AS Common Toplevel Component 11.1.1.2.0

$ORACLE_HOME/OPatch/opatch lsinventory | grep applied

-- EBS HOME 4: Weblogic Home patches check (for weblogic patching click here)

Method-1a

[oraerp@rac1 scripts]$

. $FMW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh

java weblogic.version

Method-1b

. $FMW_HOME/user_projects/domains/EBS_domain/bin/setDomainEnv.sh

java weblogic.version

Method-2 go to bsu folder and run

cd $FMW_HOME/utils/bsu

bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view

Current JDK and JRE Version

[applmgr@ebsdevapp ~]$ $ADJVAPRG -version

java version "1.7.0_321"

Java(TM) SE Runtime Environment (build 1.7.0_321-b08)

Java HotSpot(TM) Server VM (build 24.321-b08, mixed mode)

[applmgr@ebsdevapp ~]$

[oracle@ebsdevdb ~]$ $ORACLE_HOME/appsutil/jre/bin/java -version

java version "1.7.0_321"

Java(TM) SE Runtime Environment (build 1.7.0_321-b08)

Java HotSpot(TM) 64-Bit Server VM (build 24.321-b08, mixed mode)

[oracle@ebsdevdb ~]$

Apply latest etcc patch Patch 17537119: EBS Technology Codelevel Checker(ETCC)

latest opatch 6880880

If required make instance cages as per EBS-DB Core Licences:
Instance Caging for CPU allocation:
Configuring and Monitoring Instance Caging (Doc ID 1362445.1)
ODA (Oracle Database Appliance): Howto Configuring and Monitoring Instance Caging (Doc ID 1474290.1)
CPU Resource Usage in Linux and Unix ( Doc ID 466996.1 )
How is CPU_COUNT Determined? ( Doc ID 1448389.1 )

SQL> select value from v$osstat where stat_name = 'NUM_CPUS';
     VALUE
----------
        32

Enable Instance Caging:
SQL> alter system set cpu_count = 4;
SQL> alter system set resource_manager_plan = 'default_plan';

Monitor Instance Caging:
SQL> select name,instance_caging from v$rsrc_plan;
SQL> select to_char(begin_time, 'HH24:MI') time, sum(avg_running_sessions) avg_running_sessions, sum(avg_waiting_sessions) avg_waiting_sessions from v$rsrcmgrmetric_history group by begin_time order by begin_time;
SQL >

"avg_running_sessions" is the average number of running sessions for this minute. If avg_running_sessions is much smaller than cpu_count, the instance is not fully utilizing its cpu_count allocation. cpu_count could be decreased without affecting performance.

"avg_waiting_sessions" is the average number of sessions waiting to be scheduled for this minute. If avg_waiting_sessions is consistently bigger than 0, the performance of the instance could be improved by increasing cpu_count by this amount.
SQL> select * from V$RSRCMGRMETRIC
SQL> select * from v$rsrcmgrmetric_history

V$RSRCMGRMETRIC_HISTORY displays a history (the last one hour) of resource manager metrics, taken from V$RSRCMGRMETRIC. When a resource plan is set, this history is cleared and restarted. This view provides information about resources consumed and wait times per consumer group.
The columns for V$RSRCMGRMETRIC_HISTORY are the same as those for V$RSRCMGRMETRIC.




Known Issues of Weblogic Server Startup:
Issue Reported:1
Refrence Blog
Not able to bring up the Weblogic Admin server due to abnormal shutdown  or server reboot
- [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
adstrtal.sh apps/apps
. . . .
ERROR: Skipping startup of oacore_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of forms-c4ws_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of oafm_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of forms_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of oaea_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.

adstrtal.sh: Exiting with status 1
vi $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt
tailf $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.out
tailf $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log
tailf $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer-diagnostic.log

---------------------------------------------------
Solution:
1. Go to DOMAIN_HOME/servers/<SERVERNAME>/data/ldap/ldapfiles and remove the EmbeddedLDAP.lok lock file
2. Delete the *.dat file under  DOMAIN_HOME/servers/<SERVERNAME>/data/store/default and DOMAIN_HOME/servers/<SERVERNAME>/data/store/diagnostics/
3. Remove the contents under  DOMAIN_HOME/servers/<SERVERNAME>/tmp and DOMAIN_HOME/servers/<SERVERNAME>/cache
4. Now start the server (adadminsrvctl.sh start)

---------------------------------------------------
List of servers (Admin,Managed) are to delete files 1,2,3
1.AdminServer 2.oacore_server1 3.oafm_server1 4.forms_server1
[applmgr@ebsuatapp ~]$ ls -ltrh $EBS_DOMAIN_HOME/servers
total 12K
drwxr-xr-x.  3 applmgr oinstall  1 Jul 27  2021 forms-c4ws_server1
drwxr-xr-x.  3 applmgr oinstall  1 Jul 27  2021 oaea_server1
drwxr-----.  3 applmgr oinstall  1 Jul 27  2021 domain_bak
drwxr-xr-x. 11 applmgr oinstall  9 Mar 23 07:36 forms_server1
drwxr-xr-x. 11 applmgr oinstall  9 Mar 23 07:37 oafm_server1
drwxr-xr-x. 11 applmgr oinstall  9 Mar 23 07:37 oacore_server1
drwxr-----. 12 applmgr oinstall 10 Mar 23 07:44 AdminServer
-rw-r-----.  1 applmgr oinstall 14 Mar 23 08:06 AdminServerTag

try to rename the existing files but not to delete:
find $EBS_DOMAIN_HOME/servers -iname EmbeddedLDAP.lok*   --ldap folder
find $EBS_DOMAIN_HOME/servers -iname _WLS*SERVER*0.DAT*  --default folder
find $EBS_DOMAIN_HOME/servers -iname WLS_DIAG*.DAT*   --diagnostics folder


mv $EBS_DOMAIN_HOME/servers/AdminServer/tmp $EBS_DOMAIN_HOME/servers/AdminServer/tmp_old
mv $EBS_DOMAIN_HOME/servers/AdminServer/cache $EBS_DOMAIN_HOME/servers/AdminServer/cache_old
mkdir -p $EBS_DOMAIN_HOME/servers/AdminServer/tmp $EBS_DOMAIN_HOME/servers/AdminServer/cache

mv $EBS_DOMAIN_HOME/servers/oacore_server1/tmp $EBS_DOMAIN_HOME/servers/oacore_server1/tmp_old
mv $EBS_DOMAIN_HOME/servers/oacore_server1/cache $EBS_DOMAIN_HOME/servers/oacore_server1/cache_old
mkdir -p $EBS_DOMAIN_HOME/servers/oacore_server1/tmp $EBS_DOMAIN_HOME/servers/oacore_server1/cache

mv $EBS_DOMAIN_HOME/servers/oafm_server1/tmp $EBS_DOMAIN_HOME/servers/oafm_server1/tmp_old
mv $EBS_DOMAIN_HOME/servers/oafm_server1/cache $EBS_DOMAIN_HOME/servers/oafm_server1/cache_old
mkdir -p $EBS_DOMAIN_HOME/servers/oafm_server1/tmp $EBS_DOMAIN_HOME/servers/oafm_server1/cache

mv $EBS_DOMAIN_HOME/servers/forms_server1/tmp $EBS_DOMAIN_HOME/servers/forms_server1/tmp_old
mv $EBS_DOMAIN_HOME/servers/forms_server1/cache $EBS_DOMAIN_HOME/servers/forms_server1/cache_old
mkdir -p $EBS_DOMAIN_HOME/servers/forms_server1/tmp $EBS_DOMAIN_HOME/servers/forms_server1/cache

re-verify all files again
ls -ltrh $EBS_DOMAIN_HOME/servers/AdminServer/data/ldap/ldapfiles/*.lok
ls -ltrh $EBS_DOMAIN_HOME/servers/oacore_server1/data/store/default/*.DAT
ls -ltrh $EBS_DOMAIN_HOME/servers/oacore_server1/data/store/diagnostics/*.DAT
ls -ltrh $EBS_DOMAIN_HOME/servers/oafm_server1/tmp
ls -ltrh $EBS_DOMAIN_HOME/servers/oafm_server1/cache

mv $EBS_DOMAIN_HOME/servers/forms_server1/tmp $EBS_DOMAIN_HOME/servers/forms_server1/tmp_old
mkdir -p $EBS_DOMAIN_HOME/servers/forms_server1/tmp

mv $EBS_DOMAIN_HOME/servers/forms_server1/cache $EBS_DOMAIN_HOME/servers/forms_server1/cache_old
mkdir -p $EBS_DOMAIN_HOME/servers/forms_server1/cache


Issue Reported:2
during Stop/start of apps services getting below error as weblogic password is invalid.
also unable to login to Weblogic console.
need to bounce all the related sevices of EBS such as apps,db_listner, database


[applmgr@devapp01 ~]$ adstpall.sh apps/apps

ERROR: Skipping shutdown of oacore_server1 as the WebLogic password is invalid.
ERROR: Skipping shutdown of oacore_server4 as the WebLogic password is invalid.
ERROR: Skipping shutdown of forms_server1 as the WebLogic password is invalid.
ERROR: Skipping shutdown of oacore_server8 as the WebLogic password is invalid.
ERROR: Skipping shutdown of oafm_server1 as the WebLogic password is invalid.
[applmgr@devapp01 ~]$ pkill -u applmgr 

[oramgr@devdb ~]$ $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/addbctl.sh stop immediate
[oramgr@devdb ~]$ $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/addlnctl.sh stop $ORACLE_SID
[oramgr@devdb ~]$ pkill -u oramgr

[oramgr@devdb ~]$ $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/addbctl.sh start
[oramgr@devdb ~]$ $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/addlnctl.sh start $ORACLE_SID
[applmgr@devapp01 ~]$  adstrtall.sh apps/apps


echo "http://"$(cat $CONTEXT_FILE | grep s_webhost | cut -d '>' -f2 | cut -d '<' -f1)"."$(cat $CONTEXT_FILE | grep s_wls_admin_domain | cut -d '>' -f2 | cut -d '<' -f1)":"$(cat $CONTEXT_FILE | grep s_wls_adminport | cut -d '>' -f2 | cut -d '<' -f1)"/console"


WebLogic User Keeps Getting Locked Out with the Following Error BEA-090078 User WebLogic In Security Realm Myrealm Has Had 5 Invalid Login Attempts, Locking Account (Doc ID 2652897.1)

- Lock and edit
Security Realms
myrealm
User Lockout (tab)
Lockout Enabled (uncheck)
Save
Release Configuration

Thanks for Reading

Regards,

Mohammed Areefuddin.

Suggested Topics :

Linux	DATABASE	RMAN	RAC	EBS
5.7 Install	DATABASE Install		11201 Install	R1226 Install
Open Filer Instl	Active DataGaurd	Rman DataGuard	12102 RAC INSTAL	R1229 M7 Clone
DNS in Linux 7	WIN-LINUX MIGRATE	RMAN DB CLONE	RAC DataGuard	Pluggable DB Clone
lpadmin cups	ASMCMD ASM Migrt	RMAN ASM Migrt	RAC DB Clone	appsutil for DB
	11204 M-Upgrade		RAC - RAC Clone	JDK JRE upgrade
	12102 M-Upgrade		12C-19C RAC UPGRADE	Add EBS Node
	Flashback DB			ADOP in R12.2.9
				12102 EBS UPGRADE
Linux Basics	DATABASE Basics	RMAN Basics	RAC Basic	EBS Basics

Wednesday, December 15, 2021

EBS R12.2.9 Full Clone With RMAN Duplicate m7

No comments:

Post a Comment