Sunday, December 19, 2021

EBS R12.2.9 Cloning for Pluggable EBS Database




Hi, in this topic we learn how to create Full Clone of EBS Pluggable Database & Application release EBS R12.2.9 on OCI.


################################################################################################################
################################################################################################################

Cloning Oracle E-Business Suite Release 12.2 with Multitenant Database using Rapid Clone (Doc ID 2552208.1) 

Screen:
[root@ebscloudprodapp3 ~]# screen -ls
There is a screen on:
        29206.pts-0.ebscloudprodapp3    (Attached)
1 Socket in /var/run/screen/S-root.

[root@ebscloudprodapp3 ~]# screen -d -r 29206.pts-0.ebscloudprodapp3

----------------
https://my-orcl.blogspot.com/2018/04/linux-basics-commands.html

[root@rac1 ~]# yum -y install tmux
[root@rac1 ~]# rpm -qa tmux
tmux-1.8-4.el7.x86_64
[root@rac1 ~]# 
[root@rac1 ~]# tmux -f ~/.tmux.conf (force load all customizing in tmux session)
[root@rac1 ~]# tmux ls
0: 1 windows (created Tue Dec 14 06:27:17 2021) [200x57] (attached) 
[root@rac1 ~]# tmux at -t 0
[root@rac1 ~]#


Copy command from OCI -> On-premise Servers
[root@prod1-oci ~]# scp root@192.0.100.44:/home/oracle/expdp/AJ_LIVE_01022022.dmp.tar.gz .

Copy command from On-premise -> OCI Servers
[root@rac1-onprem ~]# scp -i /home/applmgr/ssh-key-20.key opc@172.168.53.10:/u02/2022-02-15-dev-fs1.tar.gz .

Mount command for /u02 in OCI Server
[root@clouduatdb ~]# sudo mkdir -p /u02
[root@clouduatdb ~]# sudo mount 172.168.10.49:/ebsfss01 /u02
[root@clouduatdb ~]# cat /etc/fstab
##NFS Share for u02
172.168.10.49:/ebsfss01 /u02 nfs defaults,nofail 0 0

################################################################################################################
################################################################################################################
Specially for OCI Database: DBCLI Commands in DATABASE Install
To check list of RDBMS homes(only) on servers from root:
OCI db home CPU patches methods  Parts Need to check and verify
Use the cliadm update-dbcli command to update the database CLI with the latest new and updated commands.


[root@prod1 ~]# dbcli describe-system -d
[root@prod1 ~]# dbcli describe-latestpatch
[root@prod1 ~]# dbcli describe-component
System Version
---------------
21.4.2.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        19.10.0.0.0           19.13.0.0
DB                                        12.1.0.2.210119       12.1.0.2.211019

[root@prod1 ~]#

Run Adpreclone Scripts on db and Apps Tier
[applmgr@ebscloudprodapp3 ~]$
[applmgr@ebscloudprodapp3 ~]$ adpreclone.pl appsTier (approx. >30Mins)
[root@ebscloudprodapp3 fs2]# tar -cvzf /u02/CLONE_FILES/EBSapps_fs2_`date +%d-%m-%y`.tar.gz EBSapps
[root@ebscloudprodapp3 fs2]#

[opc@prod1 ~]$ sudo su - oracle
Last login: Tue Dec 14 08:46:12 +03 2021
$ORACLE_SID=PRODCDB1
. $ORACLE_HOME/PRODPDB_prod1.env
[oracle@prod1 ~]$ free -hw
              total        used        free      shared     buffers       cache   available
Mem:            58G         38G         10G        3.8G        368M        9.3G         13G
Swap:           15G        157M         15G
[oracle@prod1 ~]$
[oracle@prod1 ~]$ . $ORACLE_HOME/PRODPDB_prod1.env
[oracle@prod1 ~]$ echo $ORACLE_SID
PRODPDB
[oracle@prod1 ~]$ cd $ORACLE_HOME/appsutil/scripts/PRODPDB_prod1
[oracle@prod1 PRODPDB_prod1]$ adpreclone.pl dbTier
[root@prod1 12.1.0.2]# 
[root@prod1 12.1.0.2]# pwd
/u01/app/oracle/product/12.1.0.2
[root@prod1 12.1.0.2]# 
[root@prod1 12.1.0.2]# tar -cvzf 12cdb1_cloud_`date +%d-%m-%y`.tar.gz dbhome_1 
[root@prod1 12.1.0.2]#




Prepare RMAN Backup for Container DB:
[opc@prod1 ~]$ sudo su - oracle
Last login: Tue Aug  3 11:25:26 +03 2021
$ORACLE_SID=PRODCDB1
. $ORACLE_HOME/PRODPDB_prod1.env
[oracle@prod1 ~]$ 
[oracle@prod1 ~]$ echo $ORACLE_SID
PRODCDB1
[oracle@prod1 ~]$ 
[oracle@prd1 ~]$ cat $ORACLE_HOME/dbs/initPRODCDB1.ora
SPFILE='+DATA/PRODCDB/spfilePRODCDB.ora'
[oracle@prd1 ~]$ sqlplus / as sysdba

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/PRODCDB_JED164/PARAMETER
                                                 FILE/spfile.268.1069690857
SQL>

[oracle@prod1 ~]$ rman target /

Recovery Manager: Release 12.1.0.2.0 - Production on Tue Jun 29 08:15:30 2021
Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.
connected to target database: PRODCDB (DBID=3723779434)

RMAN>
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name PRODCDB_JED164 are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/u02/rman/clone/%F';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO BACKUPSET PARALLELISM 1;
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT   '%d_%I_%U_%T_%t' PARMS  'SBT_LIBRARY=/opt/oracle/dcs/commonstore/oss/PRODCDB_jed164/libopc.so ENV=(OPC_PFILE=/opt/oracle/dcs/commonstore/oss/PRODCDB_jed164/d9499162-c087-419f-80c9-289b3b4f47e0/opc_PRODCDB_jed164.ora)';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE ON;
CONFIGURE ENCRYPTION ALGORITHM 'AES256';
CONFIGURE COMPRESSION ALGORITHM 'LOW' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECO/PRODCDB_JED164/controlfile/snapcf_PRODCDB_jed164.f';

RMAN>


RMAN BACKUP SCRIPT as per PROD Configuration (approx. time  17 mins)
#Test this updated script without changing exisitng RMAN Parameters
RUN {
#configure controlfile autobackup format for device type disk to '/u02/rman/clone/%F';
#configure controlfile autobackup on;
allocate channel t1 type disk; #channels in Run{ will be released by it self when run command complete given tasks by closing "}"
allocate channel t2 type disk;
allocate channel t3 type disk;
allocate channel t4 type disk;
allocate channel t5 type disk;
allocate channel t6 type disk;
sql 'alter system archive log current';
backup AS COMPRESSED BACKUPSET full filesperset 4 format "/u02/rman/clone/%d_FULL_%T_%s_%p.bak" database plus archivelog;
backup current controlfile format '/u02/rman/clone/control1_file_%d_%T_%s_%p.ctl';
BACKUP SPFILE format "/u02/rman/clone/SPFILE_%d_%T_%s_%p.bak";}



#Previously using Script
RUN {configure controlfile autobackup format for device type disk to '/u02/rman/clone/%F';
configure controlfile autobackup on;
allocate channel t1 type disk; #channels in Run{ will be released by it self when run command complete given tasks by closing "}"
allocate channel t2 type disk;
allocate channel t3 type disk;
allocate channel t4 type disk;
allocate channel t5 type disk;
allocate channel t6 type disk;
sql 'alter system archive log current';
backup AS COMPRESSED BACKUPSET full filesperset 4 format "/u02/rman/clone/%d_FULL_%T_%s_%p.bak" database plus archivelog;
BACKUP SPFILE format "/u02/rman/clone/SPFILE_%d_%T_%s_%p.bak";}



RMAN> report need backup;
RMAN> list backup summary;
RMAN> crosscheck backup;

Create directories as per backup script and change permission:
[root@clouduatdb 12.1.0.2]#  mkdir -p /u02/rman/clone
[root@clouduatdb 12.1.0.2]# chown -R oracle:oinstall /u02/rman/clone




[root@clouduatdb ~]# ps -ef | grep UATCDB |wc -l
94
[root@clouduatdb ~]# ps -ef | grep oracle |wc -l
120
[root@clouduatdb ~]#  Stop CDB with immediate
[opc@clouduatdb ~]$ sudo su - oracle
Last login: Tue Aug  3 11:43:06 UTC 2021 on pts/0
[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb ~]$ srvctl config database
[grid@clouduatdb ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [363229741] and the complete list of patches [29340594 32067171 32218454 32218663 32222571 32240590 ] have been applied on the local node. The release patch string is [19.10.0.0.0].
[grid@clouduatdb ~]$ 
[oracle@prod1 clone]$ srvctl config database
PRODCDB_jed164
[oracle@prod1 clone]$ srvctl config database -d PRODCDB_jed164 
[oracle@prod1 clone]$ 
[oracle@prod1 clone]$ srvctl status database -d PRODCDB_jed164
Instance PRODCDB1 is running on node prod1
Instance PRODCDB2 is running on node prod2
[oracle@prod1 clone]$ 
[oracle@rac1 ~]$ srvctl config database
[oracle@rac1 ~]$ srvctl status database -d $ORACLE_UNQNAME
[oracle@rac1 ~]$ srvctl status instance -d $ORACLE_UNQNAME -i $ORACLE_SID
[oracle@rac1 ~]$ srvctl status instance -d $ORACLE_UNQNAME -n $HOSTNAME

[oracle@rac1 ~]$ srvctl remove database -db $ORACLE_UNQNAME
[oracle@rac1 ~]$ srvctl remove database -d $ORACLE_UNQNAME -f -y
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status listener
[oracle@rac1 ~]$ srvctl config scan
[oracle@rac1 ~]$ srvctl status scan
[oracle@rac1 ~]$ srvctl status scan_listener

[oracle@clouduatdb ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Aug 3 11:47:26 2021

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
[oracle@clouduatdb ~]$ ps -ef | grep tns
root        64     2  0 May03 ?        00:00:00 [netns]
oracle    2030   540  0 12:59 pts/0    00:00:00 grep --color=auto tns
grid     89260     1  0 Jun24 ?        00:05:51 /u01/app/19.0.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid     94840     1  0 Jun23 ?        00:19:26 /u01/app/19.0.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
[oracle@clouduatdb ~]$ 
[oracle@clouduatdb ~]$ lsnrctl stop UATCDB
[oracle@clouduatdb ~]$ ps -ef | grep UATCDB |wc -l

[oracle@clouduatdb ~]$ ps -ef | grep oracle |wc -l
28
[oracle@clouduatdb ~]$ exit

[opc@clouduatdb ~]$ sudo su - grid
Last login: Tue Aug  3 11:49:20 UTC 2021
[grid@clouduatdb ~]$ asmcmd -p
[grid@clouduatdb ~]$ asmcmd -p
ASMCMD [+] > lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304   2097152  1308276                0         1308276  0             Y  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304    417792   384108                0          384108              0             N  RECO/
ASMCMD [+] >
ASMCMD [+] > cd data
ASMCMD [+data] > ls
UATCDB/
AHCCDEV_JED1RX/
ASM/
dbSys4rutiopq/
orapwasm
orapwasm_backup
ASMCMD [+data] >
 
ASMCMD [+] > cd data/UATCDB*/data*
ASMCMD [+data/UATCDB] >
ASMCMD [+data/UATCDB] > ls
BFC7541A9666806AE0533C0AA8AC1270/
C22549CD17D50517E0533C0AA8AC365B/
CHANGETRACKING/
DATAFILE/
TEMPFILE/
ASMCMD [+data/UATCDB] > (Delete all files in UATCDB FOLDER in ASM)
ASMCMD [+] > lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304   2097152  2087344                0         2087344              0             Y  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304    417792   384108                0          384108              0             N  RECO/
ASMCMD [+] >
ASMCMD [+] > 

worked below drop db option instead physically deleting CRD files from ASM/OS
this deletes C-R-D files plus spfile
select NAME from v$controlfile;
select file_name from dba_data_files;
select name from V$TEMPFILE;
select member from v$logfile;

col DEST_NAME for a25;
col DESTINATION for a50;
set linesize 200;
select dest_name,status,destination from v$ARCHIVE_DEST;

SQL> shut immediate
SQL> startup mount exclusive restrict
SQL> drop database;


Tar -xcvzf oraclehome on prod db….copy to /u02/CLONE_FILES
[root@clouduatdb CLONE_FILES]# tar -xvzf 12cdb1_23062021.tar.gz -C /u01/app/oracle/product/12.1.0.2 ( untar with root only )

[oracle@clouduatdb ~]$ ls -ltr $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle asmadmin 329282584 Apr 16 06:07 oracle

[root@clouduatdb 12.1.0.2]# chown -R oracle:oinstall dbhome_1
[root@clouduatdb 12.1.0.2]# ls -ltr $ORACLE_HOME/bin/oracle  (change group,permission to asmadmin later)
-rwsr-s--x 1 oracle oinstall 329282848 Jun 28 10:53 oracle

[root@clouduatdb ~]# chgrp asmadmin /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle (with root user)
[root@clouduatdb ~]# chmod 6751 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle (changes to red highlight with 6751 permission )

[root@clouduatdb 12.1.0.2]# su  - oracle
Last login: Mon Jun 28 10:18:29 UTC 2021 on pts/2
[oracle@clouduatdb ~]$ cat ~/.bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1; export ORACLE_HOME
PATH=$PATH:/u01/app/oracle/product/12.1.0.2/dbhome_1/bin; export PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib; export LD_LIBRARY_PATH
ORACLE_UNQNAME=UATCDB;export ORACLE_UNQNAME
ORACLE_SID=UATCDB; export ORACLE_SID
## WARNING!! Modifying this file can cause failures in API/CLI provided by Cloud Tooling!!
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/
[oracle@clouduatdb dbhome_1]$ pwd
/u01/app/oracle/product/12.1.0.2/dbhome_1
[oracle@clouduatdb dbhome_1]$


4.1.3 Create Context File
4.1.3.1 Create the Context File for a Single-Node Database
[oracle@clouduatdb ~]$ ps -ef | grep -i tns
[oracle@clouduatdb ~]$ ps -ef | grep -i CDB,PDB,oracle-user
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/appsutil/clone/bin
[oracle@clouduatdb bin]$ 
[oracle@clouduatdb bin]$ ls -ltrh $ORACLE_HOME/appsutil/*.xml*
[oracle@clouduatdb bin]$ 
perl adclonectx.pl contextfile=$ORACLE_HOME/appsutil/PRODPDB_prod1.xml template=$ORACLE_HOME/appsutil/template/adxdbctx.tmp

                     Copyright (c) 2011, 2015 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adclonectx Version 120.30.12020000.22

Running:
/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/clone/bin/../jre/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -classpath /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/clone/bin/../jlib/ojdbc6.jar:/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/clone/bin/../jlib/xmlparserv2.jar:/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/clone/bin/../jlib/java: oracle.apps.ad.context.CloneContext  -e /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/PRODPDB_prod1.xml -tmpl /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/template/adxdbctx.tmp
Enter the APPS password : apps password

Log file located at /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/clone/bin/CloneContext_0627084219.log

Provide the values required for creation of the new Database Context file.

Target System Hostname (virtual or normal) [clouduatdb] :

It is recommended that your inputs are validated by the program.
However you might choose not to validate them under following circumstances:

        -If cloning a context on source system for a remote system.
        -If cloning a context on a machine where the ports are taken and
         you do not want to shutdown the services at this point.
        -If cloning a context but the database it needs to connect is not available.

Do you want the inputs to be validated (y/n) [n] ? :

Target System Base Directory : /u01/app/oracle

Target Instance is RAC (y/n) [y] : n

Target System CDB Name : UATCDB  DEVCDB

Target System PDB Name : UATPDB  DEVPDB

Oracle OS User [oracle] :

Oracle OS Group [oinstall] :

Role separation is supported y/n [n] ? : n

Target System utl_file_dir Directory List : /usr/tmp

Number of DATA_TOP's on the Target System [3] : 1

Target System DATA_TOP Directory 1 : +DATA

Target System RDBMS ORACLE_HOME Directory [/u01/app/oracle/12.1.0] : /u01/app/oracle/product/12.1.0.2/dbhome_1

Do you want to preserve the Display [localhost:10.0] (y/n)  : n

Target System Display [clouduatdb:0.0] :

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 0
Report file located at /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/temp/portpool.lst

New context path and file name [/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/UATPDB_clouduatdb.xml] :
The new database context file has been created :
  /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/UATPDB_clouduatdb.xml
contextfile=/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/UATPDB_clouduatdb.xml
Check Clone Context logfile /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/clone/bin/CloneContext_0627084219.log for details.

Verify unique name of CDB 
ls -ltrh $ORACLE_HOME/appsutil/*.xml*
ll $ORACLE_HOME/dbs/init*.ora* $ORACLE_HOME/*.env*    Still .ora file will not be created at this point
grep unique $ORACLE_HOME/appsutil/UATPDB_clouduatdb.xml
<cdb_unique_name oa_var="s_cdb_unique_name">UATCDB</cdb_unique_name>



(Set Local/update Global) Inventory setup before run adscfgclone on dbTier:

Local Inventory Setup:
Local Inventory Setup:
[oracle@uatclouddb bin]$ cat $ORACLE_HOME/oraInst.loc
#Oracle Installer Location File Location
#Mon Apr 12 16:11:35 AST 2021
inst_group=oinstall
inventory_loc=/u01/app/oracle/product/12.1.0.2/dbhome_1/oraInventory
[oracle@uatclouddb bin]$
[oracle@uatclouddb bin]$ mkdir -p $ORACLE_HOME/oraInventory
Global Inventory Setup:
Rm oraInst.loc
Remove oracle home entry from inventory.xml in oraInventory at /u01/app/oraInventory  to avoid error RC at adcfgclone run:

[oracle@clouduatdb ~]$  mv $ORACLE_HOME/oraInst.loc $ORACLE_HOME/oraInst.loc_orig 

[oracle@clouduatdb ~]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[oracle@clouduatdb ~]$ cd /u01/app/oraInventory/ContentsXML/ 
[oracle@clouduatdb ContentsXML]$  cp inventory.xml  inventory.xml_bkp
[oracle@clouduatdb ~]$  vi  /u01/app/oraInventory/ContentsXML/inventory.xml
<HOME NAME="UATPDB_DB__u01_app_oracle_product_12_1_0_2_dbhome_1" LOC="/u01/app/oracle/product/12.1.0.2/dbhome_1" TYPE="O" IDX="2"/>





4.1.4 Configure the Database Technology Stack
[oracle@clouduatdb ~]$ ls -ltr $ORACLE_HOME/bin/oracle 
-rwsr-s--x 1 oracle oinstall 329282848 Jun 28 10:53 oracle

In order to avoaid below error in adcfgclone.pl: 
[oracle@uatclouddb bin]$ 
mkdir -p $ORACLE_HOME/oraInventory
vi $ORACLE_HOME/oraInst.loc
Goinventory_loc=/u01/app/oracle/product/12.1.0.2/dbhome_1/oraInventory

AutoConfig is exiting with status 1

WARNING: RC-50013: Fatal: Instantiate driver did not complete successfully.
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/appsutil/clone/bin
[oracle@clouduatdb bin]$ ps -ef | grep -i UATCDB 

NOTE: If intended to use grid port 1521 then Verify 1521 port is free (grid listener)
(if you got any error 1521 port already in use then check grid listener port and stop grid listener and retry dbTechStack)
[oracle@clouduatdb bin]$ lsnrctl status
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.138.9.13)(PORT=1521)))

[grid@clouduatdb bin]$ lsnrctl stop
[oracle@clouduatdb bin]$ ps -ef | grep -i tns
[oracle@clouduatdb bin]$ lsnrctl status

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))  (need to reverify this line port 1521 status  with grid & oracle user)
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
[oracle@clouduatdb bin]$ 
perl adcfgclone.pl dbTechStack $ORACLE_HOME/appsutil/UATPDB_clouduatdb.xml (worked. approx. time 3 mins)
. . . . .
. . . . .
. . . . .
. . . . .
Completed Apply...
Fri Mar 11 18:54:01 2022

Starting database listener for DEVPDB:
Running:
/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/scripts/DEVPDB_ahcclouddevdb/addlnctl.sh start AHCCDEV
Logfile: /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/log/DEVPDB_ahcclouddevdb/addlnctl.txt

You are running addlnctl.sh version 120.4


Starting listener process AHCCDEV ...


LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 11-MAR-2022 18:54:01

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/DEVPDB_ahcclouddevdb/listener.ora
Log messages written to /u01/app/oracle/product/12.1.0.2/dbhome_1/admin/DEVPDB_ahcclouddevdb/diag/tnslsnr/ahcclouddevdb/ahccdev/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ahcclouddevdb.developmentsn.ahccloudprdvcn1.oraclevcn.com)(PORT=1531)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ahcclouddevdb.developmentsn.ahccloudprdvcn1.oraclevcn.com)(PORT=1531)))
STATUS of the LISTENER
------------------------
Alias                     AHCCDEV
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                11-MAR-2022 18:54:02
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/DEVPDB_ahcclouddevdb/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/admin/DEVPDB_ahcclouddevdb/diag/tnslsnr/ahcclouddevdb/ahccdev/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ahcclouddevdb.developmentsn.ahccloudprdvcn1.oraclevcn.com)(PORT=1531)))
Services Summary...
Service "DEVPDB" has 1 instance(s).
  Instance "DEVPDB", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

addlnctl.sh: exiting with status 0

addlnctl.sh: check the logfile /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/log/DEVPDB_ahcclouddevdb/addlnctl.txt for more information ...


Running ETCC to check status of DB technology patches...
. . . . .
. . . . .
. . . . .
. . . . .
[oracle@clouduatdb bin]$

.ora, env files get created and listener also started. also re-check at what stage listener status starts dectected by CDB SID
[oracle@clouduatdb bin]ps -ef | grep tns
[oracle@clouduatdb bin]ps -ef | grep $ORACLE_SID
oracle   39371     1  0 13:40 ?        00:00:00 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/tnslsnr UATCDB -inherit
oracle   42585  9986  0 13:42 pts/0    00:00:00 grep --color=auto UATCDB
[oracle@clouduatdb ~]$ lsnrctl status  (check where are the services of listener is loaded)

[oracle@clouduatdb bin]$
[oracle@clouduatdb ~]$ . $ORACLE_HOME/UATPDB_clouduatdb.enc
[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATPDB
[oracle@clouduatdb ~]$ lsnrctl status   ($ORACLE_SID,UATCDB,UATPDB  need to re-verify )
[oracle@ahcclouddevdb ~]$ lsnrctl status $ORACLE_SID

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 11-MAR-2022 19:08:31

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ahcclouddevdb)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=DEVPDB)(INSTANCE_NAME=AHCCDEV)))
STATUS of the LISTENER
------------------------
Alias                     AHCCDEV
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                11-MAR-2022 18:54:02
Uptime                    0 days 0 hr. 14 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/DEVPDB_ahcclouddevdb/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/admin/DEVPDB_ahcclouddevdb/diag/tnslsnr/ahcclouddevdb/ahccdev/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ahcclouddevdb.developmentsn.ahccloudprdvcn1.oraclevcn.com)(PORT=1531)))
Services Summary...
Service "DEVPDB" has 1 instance(s).
  Instance "DEVPDB", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@ahcclouddevdb ~]$
[oracle@clouduatdb ~]$ 
ls -ltrh $ORACLE_HOME/dbs/init*.ora* $ORACLE_HOME/*.env*
grep unique $ORACLE_HOME/dbs/init*.ora* $ORACLE_HOME/*.env*
 
[oracle@clouduatdb bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
[oracle@clouduatdb bin]$ cat $ORACLE_HOME/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="UATPDB_DB__u01_app_oracle_product_12_1_0_2_dbhome_1" LOC="/u01/app/oracle/product/12.1.0.2/dbhome_1" TYPE="O" IDX="1"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@clouduatdb bin]$

Instance Startup Fails With ORA-00205 After Applying Patch/Relinking (Doc ID 1166697.1) 
change oracle binary permission after the above step:
[oracle@clouduatdb ~]$ ls -ltr $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 329282848 Jun 28 10:53 oracle

Important now to change to ASMADMIN:
(now it is mandatory to set before txk scripts, set with root user only)
[root@clouduatdb bin]#  
ls -ltrh /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle
chgrp asmadmin /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle   
chmod 6751 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle
ls -ltrh /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle


4.1.5 Create the listener.ora and tnsnames.ora for the Target Database
Create the listener.ora and tnsnames.ora files for the target CDB by running the following commands:
1. Set the environment.
On Unix:
$ cd <ORACLE_HOME>/appsutil
$ source ./txkSetCfgCDB.env -dboraclehome=<ORACLE_HOME>
2. Generate the listener.ora and tnsnames.ora.
$ cd <ORACLE_HOME>/appsutil/bin
$ perl txkGenCDBTnsAdmin.pl -dboraclehome=<ORACLE_HOME> -cdbname=<Name of the target container database> \
-cdbsid=<SID of the target container database> -dbport=<Target DB port> -outdir=$ORACLE_HOME/appsutil/log \
-israc=<yes/no> [-virtualhostname=<virtual hostname>]

[oracle@clouduatdb ~]$ . $ORACLE_HOME/UATCDB_clouduatdb.env
[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb ~]$ lsnrctl status $ORACLE_SID   (verified pdb listner is up with source pdb env)
TNS-01101: Could not find listener name or service name UATCDB

[oracle@clouduatdb ~]$ lsnrctl status

[oracle@clouduatdb ~]$ cd $ORACLE_HOME/appsutil
[oracle@clouduatdb appsutil]$ 
. txkSetCfgCDB.env -dboraclehome=/u01/app/oracle/product/12.1.0.2/dbhome_1

Oracle Home being passed: /u01/app/oracle/product/12.1.0.2/dbhome_1
[oracle@clouduatdb appsutil]$


[oracle@clouduatdb appsutil]$ srvctl status listener  (verifed)
Listener LISTENER is enabled
Listener LISTENER is running on node(s): ahcclouddevdb
[oracle@clouduatdb appsutil]$ cd $ORACLE_HOME/appsutil/bin
[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb ~]$
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status LISTENER
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status $ORACLE_SID (need to re-verify )
[oracle@clouduatdb bin]$ (below command worked  tested and verified)
perl txkGenCDBTnsAdmin.pl -dboraclehome=$ORACLE_HOME -cdbname=UATCDB -cdbsid=UATCDB -dbport=1521 -outdir=$ORACLE_HOME/appsutil/log -israc=no

1perl txkGenCDBTnsAdmin.pl -dboraclehome=/u01/app/oracle/product/12.1.0.2/dbhome_1 -cdbname=UATCDB -cdbsid=UATCDB -dbport=1521 -outdir=$ORACLE_HOME/appsutil/log -israc=no
. . . . . . .
. . . . . . .
. . . . . . .
Inside updateCDBSqlNetOra()…
Updating the IFILE entry…



Exiting from the script.
Ended: Mon Jun 28 11:02:54 UTC 2021


[oracle@clouduatdb bin]$ 
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status LISTENER

From section 4.1.5 PDB listener will also be detected by CDB env (verifed)
[oracle@clouduatdb UATPDB_clouduatdb]$
[oracle@clouduatdb UATPDB_clouduatdb]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status $ORACLE_SID (verifed)

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 11-MAR-2022 19:20:53

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ahcclouddevdb.developmentsn.ahccloudprdvcn1.oraclevcn.com)(PORT=1531)))
STATUS of the LISTENER
------------------------
Alias                     AHCCDEV
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                11-MAR-2022 18:54:02
Uptime                    0 days 0 hr. 26 min. 51 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/DEVPDB_ahcclouddevdb/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/admin/DEVPDB_ahcclouddevdb/diag/tnslsnr/ahcclouddevdb/ahccdev/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ahcclouddevdb.developmentsn.ahccloudprdvcn1.oraclevcn.com)(PORT=1531)))
Services Summary...
Service "DEVPDB" has 1 instance(s).
  Instance "DEVPDB", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

[oracle@clouduatdb bin]$ cd $ORACLE_HOME/appsutil/scripts/UATPDB_clouduatdb
[oracle@clouduatdb UATPDB_clouduatdb]$ . adcdblnctl.sh start UATCDB

basename: invalid option -- 'b'
Try 'basename --help' for more information.
Logfile: /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/log/UATPDB_clouduatdb/adcdblnctl.txt

You are running  version 120.0.12020000.3
Starting listener process UATCDB ...
Listener UATCDB has already been started.

Waiting for service UATPDB to be available...
 -- 21 times retrys for about 5 Mins--
Waiting for service UATPDB to be available...

Listener service UATPDB not started.
: exiting with status 1

: check the logfile /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/log/UATPDB_clouduatdb/adcdblnctl.txt for more information ...

Here oracle user will get log out automatically
-Start the Grid Listner now (1521 port should not be seen in grid listener after started
[opc@clouduatdb ~]$ sudo su – grid
[grid@clouduatdb ~]$ lsnrctl status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
[grid@clouduatdb ~]$
[grid@clouduatdb ~]$ lsnrctl start

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 21-MAR-2022 12:19:19

Copyright (c) 1991, 2021, Oracle.  All rights reserved.

Starting /u01/app/19.0.0.0/grid/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /u01/app/19.0.0.0/grid/network/admin/listener.ora
Log messages written to /u01/app/grid/diag/tnslsnr/ahcclouddevdb/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                21-MAR-2022 12:19:19
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/19.0.0.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/ahcclouddevdb/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
The listener supports no services
The command completed successfully


[opc@clouduatdb ~]$ sudo su – oracle (if not login manual entry)
[oracle@clouduatdb ~]$ lsnrctl status $ORACLE_SID
[oracle@clouduatdb ~]$ lsnrctl status UATCDB

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 29-JUN-2021 09:10:31

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     UATCDB
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                29-JUN-2021 08:38:52
Uptime                    0 days 0 hr. 31 min. 39 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/UATPDB_clouduatdb/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/admin/UATPDB_clouduatdb/diag/tnslsnr/clouduatdb/uatcdb/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
Services Summary...
Service "UATPDB" has 1 instance(s).
  Instance "UATPDB", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@clouduatdb ~]$


As pdb not restored we will get error for pdb
Will correct the listener file later, as it is using lisnter inside of context:


[opc@clouduatdb ~]$ sudo su  - oracle
Last login: Mon Jun 28 11:08:45 UTC 2021
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/dbs
[oracle@clouduatdb dbs]$
[oracle@clouduatdb dbs]$ echo $ORACLE_SID
UATCDB 
[oracle@clouduatdb dbs]$ cp initUATPDB.ora initUATCDB.ora
[oracle@clouduatdb dbs]$ cp initUATPDB.ora initUATCDB.ora
[oracle@clouduatdb dbs]$ 
[oracle@clouduatdb dbs]$ sqlplus / as sysdba 
SQL> create spfile from pfile;
File created.
SQL> create pfile from spfile;
File created.
SQL> exit
Disconnected
[oracle@clouduatdb dbs]$ mv spfileUATCDB.ora spfileUATCDB.ora_bkp
[oracle@clouduatdb dbs]$ mv spfileUATCDB.ora spfileUATCDB.ora_bkp [oracle@clouduatdb dbs]$ ls -ltr

Generate initUATCDB.ora and edit the required parameter name and remove local_listener and modify control file
*.control_files='+DATA/cntrl01a.dbf' #directory should be created before for NON-ASM
#*.local_listener='UATCDB_LOCAL' # if got error in startup then delete local line
*.log_archive_dest_1='LOCATION=+DATA'  
*.plsql_code_type='NATIVE'# Default 11i setting #(verified and checked RMAN updated as per source DB)

[oracle@clouduatdb dbs]$ sqlplus / as sysdba 

SQL> startup nomount pfile='initUATCDB.ora' #(mandatory to start db with pfile only spfile will give error)
SQL> startup nomount pfile='initUATCDB.ora'
SQL> 
SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string
SQL>
show parameter spfile;
SELECT INSTANCE_NAME, STATUS, DATABASE_STATUS FROM V$INSTANCE;
select * from v$pwfile_users;

SELECT value from v$diag_info where name='Diag Trace';
exit

grep diagnostic_dest $ORACLE_HOME/dbs/initCDBTEST.ora
find /u01/app/oracle/diag/rdbms/ -iname alert*.log
tailf /u01/app/oracle/diag/rdbms/ahccdev_jed1rx/AHCCDEV/trace/alert*.log


Inorder to fix below error in RMAN Restore update the wallet as the database is in OCI by default Transparent Data Encryption(TDE) is enabled: 
 (Make sure u bkp existing wallets of server, Required only once when created new DB servers in OCI, Once copied to new server then no need repeated in next clone)

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 06/27/2021 09:41:31
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19870: error while restoring backup piece /u02/rman/clone/c-3723779434-20210627-1d
ORA-19913: unable to decrypt backup
ORA-28365: wallet is not open

[root@clouduatdb bin]# cd /opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh/
[root@clouduatdb UATCDB_jed1gh]# ls -ltr
total 100
-rw------- 1 oracle asmadmin 2555 Jun  2 09:39 ewallet_2021060209390371_defaultTag.p12
-rw------- 1 oracle asmadmin 5819 Jun  2 09:39 ewallet.p12_uatdefault
-rw------- 1 oracle asmadmin 5864 Jun  2 09:39 cwallet.sso_uatdefault
-rw-r--r-- 1 oracle asmadmin 2612 Jun 27 13:38 masterkey.bkp
-rw-r--r-- 1 oracle asmadmin 9211 Jun 27 13:39 ewallet_2021062713391205.p12
-rw-r--r-- 1 oracle asmadmin 2612 Jun 28 13:15 masterkey_exp.bkp
-rw------- 1 oracle asmadmin 9211 Jun 28 13:16 ewallet_2021062813163988.p12
-rw------- 1 oracle asmadmin 9211 Jun 28 13:16 ewallet.p12
-rw------- 1 oracle asmadmin 9256 Jun 28 13:16 cwallet.sso
[root@clouduatdb UATCDB_jed1gh]#

need to test: Make backup of existing c&e wallets, once RMAN restore completes then revert it back and try PSU patching checks from console
The above cwallet and ewallet is copied from source server to target to over come error “ORA-19913: unable to decrypt backup”


Update the sqlnet.ora in $ORACLE_HOME/network/admin/sqlnet.ora
Update encrypt location of target by comparing source
#################################
# Definition for CDB sqlnet.ora
#################################
#ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh)))
#ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/AHCCDEV_jed1rx)))
 
SQLNET.ENCRYPTION_SERVER=REQUIRED
SQLNET.CRYPTO_CHECKSUM_SERVER=REQUIRED
SQLNET.ENCRYPTION_TYPES_SERVER=(rc4_40,AES256,AES192,AES128)#(Doc ID 2777856.1)
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=(SHA1)
#--- Client sqlnet.ora paramters
#SQLNET.ENCRYPTION_CLIENT=REQUIRED
#SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED
#SQLNET.ENCRYPTION_TYPES_CLIENT=(rc4_40,AES256,AES192,AES128)
#SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=(SHA1)

NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME)
SQLNET.EXPIRE_TIME=10
SQLNET.INBOUND_CONNECT_TIMEOUT=60
SQLNET.ALLOWED_LOGON_VERSION_SERVER=10

IFILE=/u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/sqlnet_ifile.ora




4.1.6 Restore and Start the Target Database

RMAN RESTORE SCRIPT: (approx. Time: 35 mins)
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ ls -ltr $ORACLE_HOME/bin/oracle  (if not asmadmin then, restore of controlfile fails)
-rwsr-s--x 1 oracle asmadmin 329282584 Apr 16 06:07 oracle

[oracle@clouduatdb dbs]$ mkdir -p /u01/app/oracle/admin/UATCDB/adump
[oracle@clouduatdb dbs]$ mkdir -p /u01/app/oracle/admin/UATCDB/adump/ 

[oracle@clouduatdb dbs]$ 
[oracle@clouduatdb dbs]$ rman auxiliary /

Recovery Manager: Release 12.1.0.2.0 - Production on Mon Jun 28 11:33:19 2021

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to auxiliary database: UATPDB (not mounted)

RMAN> show AUXILIARY CHANNEL;
RMAN> host 'lsnrctl status';

#-UPDATED-restore db script (set db_unique_name) parameter if RMAN fails remove the line and try again.
RMAN>
run {
ALLOCATE AUXILIARY CHANNEL aux1 DEVICE TYPE DISK; # channels in Run{ will be released by it self when run command complete given tasks by closing "}"
ALLOCATE AUXILIARY CHANNEL aux2 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux3 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux4 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux5 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL aux6 DEVICE TYPE DISK;
duplicate database to UATCDB spfile
set db_unique_name='UATCDB_jed1gh# verifed & worked , check as per your OCI DB this line
set control_files '+DATA/cntrl01a.dbf' #directory should be created before for NON-ASM
set db_create_file_dest '+DATA/' 
SET CLUSTER_DATABASE='FALSE' 
set audit_file_dest='/u01/app/oracle/admin/UATCDB_jed1gh/adump
BACKUP LOCATION '/u02/rman/rman_dailybkp/' NOFILENAMECHECK;}

no need of providing Release channel if it allocated in Run{ once the duplicate command complete it automatically releases the channel and you wont get below error.
Lets make a try with out issuing any duplicate in RUN{
same will happen for normal channel and auxiliary channel.
RMAN> RUN {
allocate AUXILIARY channel t1 type disk;
allocate AUXILIARY channel t2 type disk;
allocate AUXILIARY channel t3 type disk;}

allocated channel: t1
channel t1: SID=4 device type=DISK

allocated channel: t2
channel t2: SID=2006 device type=DISK

allocated channel: t3
channel t3: SID=4013 device type=DISK
released channel: t1
released channel: t2
released channel: t3

RMAN>
. . . . . 
. . . . . 
Reenabling controlfile options for auxiliary database
Executing: alter database force logging
Executing: alter database enable block change tracking

contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

database opened

contents of Memory Script:
{
   sql clone "alter pluggable database all open";
}
executing Memory Script

sql statement: alter pluggable database all open
Finished Duplicate Db at 28-JUN-21

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of release command at 06/28/2021 12:37:28
RMAN-06012: channel: aux1 not allocated

RMAN> show AUXILIARY CHANNEL;
RMAN> exit

We can ignore the above error just channel got not released:
New alert log file will be started at 
SELECT value from v$diag_info where name='Diag Trace';

tailf /u01/app/oracle/diag/rdbms/uatcdb_jed1gh/UATCDB/trace/alert*
tailf /u01/app/oracle/diag/rdbms/devcdb/DEVCDB/trace/alert*

grep diagnostic_dest $ORACLE_HOME/dbs/initCDBTEST.ora
tailf /data/admin/oracle/diag/rdbms/cdbtest/CDBTEST/trace/alert*.log

Make sure you backup a new pfile after completing RMAN
create pfile='/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/initAFTER_RMAN.ora' from spfile;

ls -ltrh $ORACLE_HOME/dbs/initAFTER_RMAN.ora



After restoring apply patch p22747454_121020_Generic.zip if got MGD mismatch:
 
[oracle@clouduatdb ~]$ $ORACLE_HOME/OPatch/opatch lsinventory | grep 22747454
Patch  22747454     : applied on Mon Mar 21 13:20:32 UTC 2022
     22747454
[oracle@ahcclouddevdb ~]$
[oracle@clouduatdb ~]$ $ORACLE_HOME/OPatch/datapatch -verbose
If patch is already applied i.e using same db home for db-refresh/cloning then just run datapatch without shutdown on db

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PRODPDB                        READ WRITE YES
SQL>
SQL> select COMP_ID from DBA_REGISTRY where STATUS='VALID' order by 1;  --(CDB & PDB Should have same number of 18 rows in total)
SQL> select TIME, MESSAGE,ACTION from PDB_PLUG_IN_VIOLATIONS; -- check for error message of violations

TIME
---------------------------------------------------------------------------
MESSAGE
--------------------------------------------------------------------------------
ACTION
--------------------------------------------------------------------------------
28-JUN-21 12.37.23.572364 PM
Database option MGD mismatch: PDB installed version 12.1.0.2.0. CDB installed ve
rsion NULL.
Fix the database option in the PDB or the CDB


SQL> alter session set container=cdb$root;

Session altered.
SQL> alter pluggable database PRODPDB (close , open read write) – can test the warning in opening
Warning: PDB altered with errors. 
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PRODPDB                        READ WRITE YES

SQL> 
SQL> shut immediate

drwxr-xr-x 4 oracle oinstall  4096 Jun 20 07:29 22747454
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/appsutil/etcc/22747454
[oracle@clouduatdb 22747454]$
[oracle@clouduatdb 22747454]$ $ORACLE_HOME/OPatch/opatch lsinventory | grep 22747454
[oracle@clouduatdb 22747454]$
[oracle@clouduatdb 22747454]$ $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/12.1.0.2/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0.2/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.21
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatch/opatch2021-06-27_12-19-47PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   22747454

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Backing up files...
Applying interim patch '22747454' to OH '/u01/app/oracle/product/12.1.0.2/dbhome_1'

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patch 22747454 successfully applied.
Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatch/opatch2021-06-27_12-19-47PM_1.log

OPatch succeeded.
[oracle@clouduatdb 22747454]$ cd $ORACLE_HOME/OPatch/
[oracle@clouduatdb 22747454]$ $ORACLE_HOME/OPatch/opatch lsinventory | grep 22747454
Patch  22747454     : applied on Tue Dec 14 14:54:32 UTC 2021
     22747454 
[oracle@clouduatdb 22747454]$ 
[oracle@clouduatdb OPatch]$
[oracle@clouduatdb OPatch]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Jun 27 12:20:25 2021

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 3.0602E+10 bytes
Fixed Size                  3728688 bytes
Variable Size            1.0670E+10 bytes
Database Buffers         1.9864E+10 bytes
Redo Buffers               63377408 bytes
Database mounted.
Database opened.
SQL>
alter session set container=PRODPDB;
set linesize 200;
col DESCRIPTION for a80;
col ACTION_TIME for a31;
select PATCH_ID,ACTION_TIME,DESCRIPTION from dba_registry_sqlpatch;
exit
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options
[oracle@clouduatdb ~]$ $ORACLE_HOME/OPatch/datapatch -verbose (worked fine) (approx. 15 Mins)
[oracle@clouduatdb OPatch]$ cd $ORACLE_HOME/OPatch/
[oracle@clouduatdb OPatch]$ ./datapatch -verbose    
SQL Patching tool version 12.1.0.2.0 Production on Sun Jun 27 12:21:11 2021
Copyright (c) 2012, 2017, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/sqlpatch/sqlpatch_90541_2021_06_27_12_21_11/sqlpatch_invocation.log

Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Patch 18793246 (BASE OBJECT LOOKUP DURING DATAPUMP EXPORT CAUSES FULL TABLE SCAN PER OBJECT):
  Installed in the binary registry and PRODPDB CDB$ROOT PDB$SEED
Patch 22731026 (XTTS12102: DRG-10602 AFTER MIGRATION FROM RDBMS 11.2.0 TO 12.1.0):
  Installed in the binary registry and PRODPDB CDB$ROOT PDB$SEED
Patch 22747454 ():
  Installed in the binary registry only
Patch 31629625 (MERGE REQUEST ON TOP OF DATABASE BP 12.1.0.2.200714 FOR BUGS 31321438 31413047):
  Installed in the binary registry and PRODPDB CDB$ROOT PDB$SEED
Patch 31668915 (Database PSU 12.1.0.2.201020, Oracle JavaVM Component (OCT2020)):
  Installed in the binary registry and PRODPDB CDB$ROOT PDB$SEED
Patch 31838788 (MERGE REQUEST ON TOP OF DATABASE BP 12.1.0.2.201020 FOR BUGS 21117072 22611167):
  Not installed in the binary or the SQL registry
Bundle series DBBP:
  ID 201020 in the binary registry and ID 201020 in PDB CDB$ROOT, ID 201020 in PDB PDB$SEED, ID 201020 in PDB PRODPDB

Adding patches to installation queue and performing prereq checks...
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED PRODPDB
    Nothing to roll back
    The following patches will be applied:
      22747454 ()

Installing patches...
Patch installation complete.  Total patches installed: 3

Validating logfiles...
Patch 22747454 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/sqlpatch/22747454/21999295/22747454_apply_UATCDB_CDBROOT_2021Jun27_12_21_50.log (no errors)
Patch 22747454 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/sqlpatch/22747454/21999295/22747454_apply_UATCDB_PDBSEED_2021Jun27_12_24_01.log (no errors)
Patch 22747454 apply (pdb PRODPDB): SUCCESS
  logfile: /u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/sqlpatch/22747454/21999295/22747454_apply_UATCDB_PRODPDB_2021Jun27_12_24_01.log (no errors)
SQL Patching tool complete on Sun Jun 27 12:28:57 2021
[oracle@clouduatdb OPatch]$
[oracle@clouduatdb OPatch]$


Clear the Restricted mode on PDB:
[oracle@clouduatdb OPatch]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Jun 27 12:30:53 2021

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PRODPDB                        READ WRITE YES

SQL> alter pluggable database PRODPDB close;

Pluggable database altered.

SQL> alter pluggable database PRODPDB open read write;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PRODPDB                        READ WRITE NO
SQL>



Now the Pluggable database is opened normally without warning, so we can use it.
 
SQL> alter pluggable database PRODPDB close;

Pluggable database altered.

SQL> !ls -ltrh $ORACLE_HOME/dbs/*.xml*

SQL> alter pluggable database "PRODPDB" unplug into '/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/PRODPDB.xml';

Pluggable database altered.

[oracle@clouduatdb ~]$ ls -ltr $ORACLE_HOME/dbs/*.xml
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PRODPDB                        MOUNTED
SQL>
SQL> drop pluggable database PRODPDB;

Pluggable database dropped.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
SQL>

SQL> show parameter service

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      PRODPDB , PRODPDB_ebs_patch
SQL>
SQL>
SQL> create pluggable database "UATPDB" using '/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/PRODPDB.xml' nocopy  SERVICE_NAME_CONVERT=('ebs_PRODPDB','ebs_UATPDB');

SQL> create pluggable database "PDB" using '/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/PRODPDB.xml' nocopy  SERVICE_NAME_CONVERT=('ebs_PRODPDB','ebs_UATPDB');

Pluggable database created.

SQL>
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         MOUNTED
SQL>
SQL> alter session set container=cdb$root;
SQL> 
SQL> alter pluggable database UATPDB open read write;
SQL> alter pluggable database UATPDB open read write;

Warning: PDB altered with errors.

SQL> show pdbs
SQL>
SQL> alter pluggable database all save state instances=all;

Pluggable database altered.

SQL>
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE YES
SQL>


MGD Missmatch is corrected with the patch applied both CDB and PDB has 18 rows of record.
But still pluggable will in restricted mode…for wallet keys…..follow below procedure to fix…modify the below values for UAT …UATPDB and wallet location to /opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh 
/opt/oracle/dcs/commonstore/wallets/tde/AHCCDEV_jed1rx


Creating new Key and importing to PDB 
SQL> 
SQL> SELECT WRL_PARAMETER,STATUS,WALLET_TYPE FROM V$ENCRYPTION_WALLET;

WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE
------------------------------ --------------------
/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh/
OPEN                           AUTOLOGIN


SQL>
SQL> alter session set container=cdb$root;

Session altered.

SQL> alter pluggable database all save state instances=all;
SQL>
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "TesT_1234#" container=Current;
ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "TesT_1234#" container=Current
*
ERROR at line 1:
ORA-28354: Encryption wallet, auto login wallet, or HSM is already open

SQL>
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE CLOSE CONTAINER=ALL;

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE CLOSE identified by "TesT_1234#" CONTAINER=ALL;

keystore altered.

SQL>
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "TesT_1234#" container=Current;

keystore altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE YES
SQL>
SQL>
SQL> alter session set container= UATPDB;
SQL> alter session set container= UATPDB;

Session altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "TesT_1234#" container=Current;

keystore altered.

SQL>

[oracle@clouduatdb ~]$ ls -ltr /opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh
[oracle@clouduatdb UATCDB_jed1gh]$ ls -ltr
total 112
-rw------- 1 oracle asmadmin 2555 Jun  2 09:39 ewallet_2021060209390371_defaultTag.p12
-rw------- 1 oracle asmadmin 5819 Jun  2 09:39 ewallet.p12_uatdefault
-rw------- 1 oracle asmadmin 5864 Jun  2 09:39 cwallet.sso_uatdefault
-rw-r--r-- 1 oracle asmadmin 2612 Jun 27 13:38 masterkey.bkp
-rw-r--r-- 1 oracle asmadmin 9211 Jun 27 13:39 ewallet_2021062713391205.p12
-rw------- 1 oracle asmadmin 9211 Jun 28 13:16 ewallet_2021062813163988.p12
-rw-r--r-- 1 oracle asmadmin 2612 Jun 29 10:27 masterkey_exp.bkp
-rw------- 1 oracle asmadmin 9211 Jun 29 10:28 ewallet_2021062910281074.p12
-rw------- 1 oracle asmadmin 9211 Jun 29 10:28 ewallet.p12
-rw------- 1 oracle asmadmin 9256 Jun 29 10:28 cwallet.sso
[oracle@clouduatdb UATCDB_jed1gh]$


Worked red commands without giving UATCD_jed1gh or AHCCDEV_jed1rx folder

[oracle@clouduatdb tde]$ ls -ltr /opt/oracle/dcs/commonstore/wallets/tde
[oracle@clouduatdb tde]$ rm -rf /opt/oracle/dcs/commonstore/wallets/tde/masterkey*
SQL> 
SQL> ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "TesT_1234#" to '/opt/oracle/dcs/commonstore/wallets/tde/masterkey5_exp.bkp' IDENTIFIED BY "TesT_1234#";
SQL> ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "TesT_1234#" to '/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh/masterkey_exp.bkp' IDENTIFIED BY "TesT_1234#";

keystore altered.

SQL> ADMINISTER KEY MANAGEMENT IMPORT KEYS WITH SECRET "TesT_1234#" from '/opt/oracle/dcs/commonstore/wallets/tde/masterkey5_exp.bkp' IDENTIFIED BY "TesT_1234#" with backup;
SQL> ADMINISTER KEY MANAGEMENT IMPORT KEYS WITH SECRET "TesT_1234#" from '/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh/masterkey_exp.bkp' IDENTIFIED BY "TesT_1234#" with backup;

keystore altered.

SQL>
 SQL>show con_name

CON_NAME
------------------------------
UATPDB
SQL>
SQL> alter session set container=cdb$root;

Session altered.

SQL> alter pluggable database UATPDB close;
SQL> alter pluggable database UATPDB close;

Pluggable database altered.

SQL> alter pluggable database UATPDB open read write;
SQL> alter pluggable database UATPDB open read write;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE NO
SQL>

Now Both mgd and wallet corrections done at CDB & PDB level.
PDB opened normally without restricted mode.


SQL>
SQL> SELECT WRL_PARAMETER,STATUS,WALLET_TYPE FROM V$ENCRYPTION_WALLET;

WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE
------------------------------ --------------------
/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh/
OPEN                           PASSWORD


SQL>
SQL> alter session set container= UATPDB;
SQL> alter session set container= UATPDB;

Session altered.

SQL> SELECT WRL_PARAMETER,STATUS,WALLET_TYPE FROM V$ENCRYPTION_WALLET;

WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE
------------------------------ --------------------
/opt/oracle/dcs/commonstore/wallets/tde/UATCDB_jed1gh/
CLOSED                         UNKNOWN


SQL>
SQL> alter session set container=cdb$root;

Session altered.

SQL> shutdown immediate
ORA-65011: Pluggable database PRODPDB does not exist.
SQL> shut abort
SQL> startup
SQL> show pdbs  (if it opened in restricted mode again close/open PDB)

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE YES
SQL>
SQL> alter pluggable database UATPDB close;
SQL> alter pluggable database UATPDB close;

Pluggable database altered.

SQL> alter pluggable database UATPDB open read write;
SQL> alter pluggable database UATPDB open read write;

Pluggable database altered.

SQL> show pdbs  (now it should not be in restricted mode)

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE NO
SQL>
SQL> 
SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/12.1.0
                                                 .2/dbhome_1/dbs/spfileUATCDB.o
                                                 ra
SQL> 
SQL> show parameter service;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      PRODPDB , PRODPDB_ebs_patch
SQL>

Do not add ebs_patch service now because when we run autoconifg at final it gives error.
Add only red color line. Any ways autoconfig will create service with ebspatch but it will not update in init file:

[oracle@clouduatdb ~]$ grep service_names $ORACLE_HOME/dbs/initUATCDB.ora
[oracle@clouduatdb ~]$ 
[oracle@clouduatdb ~]$ grep service_names $ORACLE_HOME/dbs/initUATCDB.ora

SQL> alter system set service_names='UATPDB' scope=both;
SQL> alter system set service_names='UATPDB' scope=both;    (use this  command only didn’t get error in autoconfig)
SQL> 
SQL> alter system set service_names='UATPDB','UATPDB_ebs_patch' scope=both; (don’t use this ebs_patch command)
System altered.

SQL>
 SQL> show parameter service;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      UATPDB
SQL>

SQL> show parameter utl

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
create_stored_outlines               string
utl_file_dir                         string      /usr/tmp, /u01/OCI/fs1/inst/ap
                                                 ps/PRODPDB_ebscloudprodapp3/ap
                                                 pltmp, /u01/app/oracle/product
                                                 /12.1.0.2/dbhome_1/appsutil/ou
                                                 tbound/PRODPDB_prod1, /u02/OCI/
                                                 temp, /usr/tmp
SQL>

Update ult_file along with appltmp directory of Target APPS Server:
SQL> alter system set utl_file_dir='/usr/tmp','/u01/uat/fs1/inst/apps/UATPDB_uatcloudapp/appltmp','/usr/tmp' scope=spfile;

SQL> alter system set utl_file_dir='/usr/tmp','/u01/uat/fs1/inst/apps/UATPDB_uatcloudapp/appltmp','/usr/tmp' scope=spfile; (for dev appltmp location)

System altered.

SQL>
SQL> create pfile from spfile;

File created.

SQL>
[oracle@clouduatdb ~]$ grep service_names $ORACLE_HOME/dbs/initUATCDB.ora
*.service_names='UATPDB'
[oracle@clouduatdb ~]$ grep utl $ORACLE_HOME/dbs/initUATCDB.ora
*.utl_file_dir='/usr/tmp','/u01/uat/fs1/inst/apps/UATPDB_uatcloudapp/appltmp','/usr/tmp'
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ grep service_names $ORACLE_HOME/dbs/initUATCDB.ora
*.service_names='UATPDB'
[oracle@clouduatdb ~]$ grep utl $ORACLE_HOME/dbs/initUATCDB.ora
*.utl_file_dir='/usr/tmp','/u01/uat/fs1/inst/apps/UATPDB_uatcloudapp/appltmp','/usr/tmp'
[oracle@clouduatdb ~]$


SQL>
SQL> shut immediate;
ORACLE instance shut down.
SQL>

SQL> startup;
SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/12.1.0
                                                 .2/dbhome_1/dbs/spfileUATCDB.o
                                                 ra
SQL>
SQL> show parameter utl

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
create_stored_outlines               string
utl_file_dir                         string      /usr/tmp, /u01/uat/fs1/inst/ap
                                                 ps/UATPDB_uatcloudapp/applt
                                                 mp, /usr/tmp
SQL> show parameter service;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      UATPDB
SQL>

alter session set container=cdb$root;
show pdbs
alter pluggable database UATPDB close;
alter pluggable database UATPDB open read write;
show pdbs

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE YES
SQL>
SQL> alter pluggable database UATPDB close;  (verify by close and open if required)

Pluggable database altered.

SQL>
SQL> alter pluggable database UATPDB open read write;

Pluggable database altered.

SQL>
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE NO
SQL>


check the listener status ebs_patch added to service, it should not as we removed.
[oracle@clouduatdb ~]$ echo  $ORACLE_SID
UATCDB
[oracle@clouduatdb ~]$ lsnrctl status $ORACLE_SID
Service "UATPDB_ebs_patch" (Will not be there)

Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/listener.ora (need to update)
Services Summary...
Service "AHCICDBXDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 2 handler(s) for this service...
Service "AJARDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 2 handler(s) for this service...
Service "UATCDB_jed1gh" has 1 instance(s).
  Instance "UATCDB", status READY, has 2 handler(s) for this service...
Service "UATPDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 2 handler(s) for this service...
The command completed successfully

Most important Step to recreate listener and tns files for EBS: (approx. time 5 mins)
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/appsutil/clone/bin

defining $ORACLE_HOME has worked
[oracle@clouduatdb bin]$ perl adcfgclone.pl dbconfig $ORACLE_HOME/appsutil/UATPDB_clouduatdb.xml

[oracle@clouduatdb bin]$ perl adcfgclone.pl dbconfig /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/UATPDB_clouduatdb.xml

[oracle@clouduatdb ~]$ lsnrctl status $ORACLE_SID
Service "UATPDB_ebs_patch" (Will be added)
Service

If got any error then add this below parameter and retry, all the listener services should point to this listener obly:
SQL> alter system set listener_networks='(( NAME=net1)(LOCAL_LISTENER=(ADDRESS=(PROTOCOL=TCP)(HOST=192.172.9.13)(PORT=1531))))' scope=spfile;
SQL> shut immediate
SQL> startup

show pdbs
alter pluggable database UATPDB close;
alter pluggable database UATPDB open read write;
show pdbs
[oracle@clouduatdb bin]$ perl adcfgclone.pl dbconfig $ORACLE_HOME/appsutil/UATPDB_clouduatdb.xml




Now Run Autoconfig on database Tier:
[oracle@clouduatdb ~]$ . $ORACLE_HOME/UATPDB_clouduatdb.env
[oracle@clouduatdb ~]$ . $ORACLE_HOME/UATPDB_clouduatdb.env 

[oracle@clouduatdb ~]$ sqlplus apps/appspasswd@UATPDB
[oracle@clouduatdb ~]$ sqlplus apps/appsr12_2_9@UATPDB

SQL> select NODE_NAME from fnd_nodes;
NODE_NAME
--------------------------------------------------------------------------------
CLOUDUATDB

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> 

Before running autoconfig with check ebs_patch service added to litener or not
Need to check: If already there in service, high chance to get below error in autoconfig 
[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATPDB
[oracle@clouduatdb ~]$ lsnrctl status $ORACLE_SID
Service "UATPDB_ebs_patch" has 1 instance(s).

[oracle@clouduatdb ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh
[oracle@clouduatdb UATPDB_clouduatdb]$ adautocfg.sh
AutoConfig completed successfully.

Now ebs_patch named serveice should be added to listener by autoconfig.
If required now create a pfile and check service in pfile ebs_patch added or not.?
If autconfig completed successfully, then database cloning is completed:

Perform the post duplication steps to remove the 2nd node log files and undo tablespace
SQL> show user
USER is "SYS"
SQL> 
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE NO
SQL> 
SQL> select THREAD#, STATUS, ENABLED from v$thread;

   THREAD# STATUS ENABLED
---------- ------ --------
         1 OPEN   PUBLIC
         2 CLOSED PUBLIC

SQL> select group# from V$log where THREAD#=2;

    GROUP#
----------
         3
         4
SQL> alter session set container=cdb$root;

Session altered.

SQL>
SQL> select THREAD#, STATUS, ENABLED from v$thread;
select group# from V$log where THREAD#=2;
alter database disable thread 2;
alter database drop logfile group 3;
alter database drop logfile group 4;
select THREAD#, STATUS, ENABLED from v$thread;

   THREAD# STATUS ENABLED
---------- ------ --------
         1 OPEN   PUBLIC
SQL> 
SQL> show parameter undo

Check system password status for apps patching and expdp:
SQL> select username,account_status,EXPIRY_DATE from dba_users where username like 'SYSTEM';
SQL> ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;

To fix adop ETCC Codelevel checker patch
It is necessary to run check db atleast 1 time on cloned instance, else adop prepare phase will fail
[oracle@clouduatdb etcc]$ ls -ltrh $ORACLE_HOME/appsutil/etcc/check*
[oracle@clouduatdb etcc]$ unzip p17537119_R12_GENERIC.zip
[oracle@clouduatdb etcc]$ . $ORACLE_HOME/UATPDB_clouduatdb.env
[oracle@clouduatdb etcc]$ echo $ORACLE_SID
UATPDB
[oracle@clouduatdb etcc]$ sh $ORACLE_HOME/appsutil/etcc/checkDBpatch.sh ÃŸ mandatory, if not adop will fail when ever used on instance
applahc@ahcsc1zioclient020302:~/ETCC$ ./checkMTpatch.sh

[oratest@ebstestdb: ~]


Run autconfig again
[oracle@clouduatdb ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh
[oracle@clouduatdb u]$ adautocfg.sh
AutoConfig completed successfully.
################################################################################################################
################################################################################################################
Additional steps to reconfigure DB to Cluster

Update db uniqe name to rejoin cluster
SQL> create pfile='$ORACLE_HOME/dbs/initUATCDB_09022022.ora' from spfile;
SQL> alter system set db_unique_name='UATCDB_jed1gh' scope=spfile;
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE YES
SQL> alter pluggable database UATPDB close;

Pluggable database altered.

SQL> alter pluggable database UATPDB open read write;

Pluggable database altered.

SQL> sho pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 UATPDB                         READ WRITE NO
SQL>
[oracle@clouduatdb ~]$ cat /etc/oratab
#UATCDB:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
UATCDB_jed1gh:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ srvctl add database -d UATCDB_jed1gh -n UATCDB -o /u01/app/oracle/product/12.1.0.2/dbhome_1
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ srvctl add instance -d UATCDB_jed1gh -i UATCDB -n clouduatdb
[oracle@clouduatdb ~]$ echo $ORACLE_UNQNAME
UATCDB_jed1gh
[oracle@clouduatdb ~]$ srvctl config database -d $ORACLE_UNQNAME
Database unique name: UATCDB_jed1gh
Database name: 
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile:
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups:
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oinstall
Database instances: UATCDB
Configured nodes: clouduatdb
Database is administrator managed
[oracle@clouduatdb ~]$ 

[oracle@clouduatdb ~]$ srvctl status database -d $ORACLE_UNQNAME
Instance UATCDB is not running on node clouduatdb
[oracle@clouduatdb ~]$ srvctl start database -d $ORACLE_UNQNAME
alter session set container=cdb$root;
show pdbs
alter pluggable database DEVPDB close;
alter pluggable database DEVPDB open read write;
show pdbs


[oracle@clouduatdb appsutil]$ grep unique $ORACLE_HOME/appsutil/UATPDB_clouduatdb.xml
      <cdb_unique_name oa_var="s_cdb_unique_name">UATCDB_jed1gh</cdb_unique_name>
[oracle@clouduatdb appsutil]$
[oracle@clouduatdb ~]$ . $ORACLE_HOME/UATPDB_clouduatdb.env
[oracle@clouduatdb ~]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh
[oracle@clouduatdb UATPDB_clouduatdb]$ ./adautocfg.sh

Update SPFILE to ASM 
[oracle@clouduatdb ~]$ rman target /
RMAN> backup spfile;
RMAN> restore spfile to '+DATA';
RMAN>

ASMCMD [+data/uatcdb] > ls
BFC7541A9666806AE0533C0AA8AC1270/
C22549CD17D50517E0533C0AA8AC365B/
CHANGETRACKING/
CONTROLFILE/
DATAFILE/
TEMPFILE/
spflie
ASMCMD [+data/uatcdb] >

[oracle@clouduatdb ~]$ srvctl config database -d $ORACLE_UNQNAME
Spfile: +DATA/UATCDB_JED1GH/PARAMETERFILE/spfile.439.1098806737

[oracle@clouduatdb ~]$ srvctl stop database -d $ORACLE_UNQNAME
[oracle@clouduatdb ~]$ srvctl start database -d $ORACLE_UNQNAME
[oracle@clouduatdb ~]$ !sq
alter pluggable database DEVPDB close;
alter pluggable database DEVPDB open read write;
SQL> show parameter spfile;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/UATCDB_JED1GH/PARAMETERFILE/spfile.439.1098806737
SQL> 

[oracle@clouduatdb ~]$ ls -ltrh $ORACLE_HOME/dbs/spfile*.ora* $ORACLE_HOME/dbs/initUATCDB.ora
[oracle@clouduatdb ~]$ cp $ORACLE_HOME/dbs/initUATCDB.ora $ORACLE_HOME/dbs/initUATCDB.ora_bkp4
[oracle@clouduatdb ~]$ echo +DATA/UATCDB_JED1GH/PARAMETERFILE/spfile.439.1098806737 > $ORACLE_HOME/dbs/initUATCDB.ora
[oracle@clouduatdb ~]$ cat $ORACLE_HOME/dbs/initUATCDB.ora
+DATA/UATCDB_JED1GH/PARAMETERFILE/spfile.439.1098806737
[oracle@clouduatdb ~]$ 
[oracle@clouduatdb ~]$ srvctl modify database -d orclcdb -spfile '+DATA/UATCDB/spfile'

Update password file to ASM is still required (same as prod)



Reverify all details of DB
[root@clouduatdb ~]# dbcli describe-component
[root@clouduatdb ~]# dbcli list-jobs|tail -5
[root@clouduatdb ~]# dbcli describe-job -i <failed job id for xxx> -l Verbose
[root@clouduatdb ~]# dbcli describe-latestpatch
[root@clouduatdb ~]# dbcli list-dbstorages
[root@clouduatdb ~]# dbcli list-dbhomes
[root@clouduatdb ~]# dbcli list-databases
[root@clouduatdb ~]# dbcli describe-database -in UATCDB
[root@clouduatdb ~]# 
[oracle@clouduatdb ~]# srvctl config database
[oracle@clouduatdb ~]# srvctl config database -d $ORACLE_UNQNAME

################################################################################################################
################################################################################################################


Check below of the file for APPS Cloning initial Steps:
Correct inventory before running adcfg on apps

[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status UATCDB

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 03-AUG-2021 22:43:26

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     UATCDB
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                03-AUG-2021 21:12:30
Uptime                    0 days 1 hr. 30 min. 56 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/log/diag/tnslsnr/clouduatdb/UATCDB/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
Services Summary...
Service "PRODCDBXDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATCDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB_ebs_patch" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@clouduatdb UATPDB_clouduatdb]$ 

Check db alert log
[root@clouduatdb bin]# tailf /u01/app/oracle/diag/rdbms/UATCDB/UATCDB/trace/alert_UATCDB.log
There should no tns error and warnings like.
Fatal NI connect error 12170.
TNS-12535: TNS:operation timed out


If autoconfig is not completed successfully the follow below steps remove ebs_patch in spfile and retry
[oracle@clouduatdb bin]$ 
[oracle@clouduatdb bin]$ lsnrctl status uatcdb

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 29-JUN-2021 11:09:29

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     UATCDB
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                29-JUN-2021 08:38:52
Uptime                    0 days 2 hr. 30 min. 37 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/UATPDB_clouduatdb/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/admin/UATPDB_clouduatdb/diag/tnslsnr/clouduatdb/uatcdb/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
Services Summary...
Service "PRODCDBXDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATCDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB" has 2 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
  Instance "UATPDB", status UNKNOWN, has 1 handler(s) for this service...
Service "UATPDB_ebs_patch" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@clouduatdb bin]$


Autoconfig ERROR Solution/Fix (start listener with CDB )
Here the CDB Listener is using PDB File to fix it follow below:

If listener is not using $ORACLE_HOME/network/admin/listener.ora then autoconfig fails with error
WARNING: [CVM Error Report]
The following report lists errors encountered during CVM Phase
      <filename>  <return code where appropriate>
  /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/bin/txkSetADOPPatchSrvName.sh 1

 No. of scripts failed in CVM phase: 1



AutoConfig is exiting with status 1

[oracle@clouduatdb UATPDB_clouduatdb]$

[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ . $ORACLE_HOME/UATPDB_clouduatdb.env
[oracle@clouduatdb ~]$
[oracle@clouduatdb ~]$ echo $ORACLE_SID
UATPDB
[oracle@clouduatdb ~]$ cd $ORACLE_HOME/appsutil/scripts/UATPDB_clouduatdb
[oracle@clouduatdb UATPDB_clouduatdb]$ addlnctl.sh stop UATCDB
[oracle@clouduatdb UATPDB_clouduatdb]$ . $ORACLE_HOME/UATCDB_clouduatdb.env
[oracle@clouduatdb UATPDB_clouduatdb]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb UATPDB_clouduatdb]$ adcdblnctl.sh start UATCDB
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status uatcdb

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 30-JUN-2021 09:00:14

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     uatcdb
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                30-JUN-2021 05:36:55
Uptime                    0 days 3 hr. 23 min. 19 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/log/diag/tnslsnr/clouduatdb/uatcdb/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
Services Summary...
Service "PRODCDBXDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATCDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB_ebs_patch" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@clouduatdb UATPDB_clouduatdb]$


If any additional error in auto config due to Service names ebs_patch then stop and delete as below the autoconfig will create internally and will not shown in spfile/pfile:
ERROR DESCRIPTION:
(ORA-44303: service name exists
ORA-06512: at "SYS.DBMS_SERVICE_ERR", line 21
ORA-06512: at "SYS.DBMS_SERVICE", line 218
ORA-06512: at line 1 (DBD ERROR: OCIStmtExecute) at /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/bin/txkSetADOPPatchSrvName.pl line 511.

[oracle@clouduatdb ~]$ grep service_names $ORACLE_HOME/dbs/initUATCDB.ora

SQL> alter system set service_names='UATPDB' scope=both;
SQL> begin
  DBMS_SERVICE.STOP_SERVICE(service_name =>'UATPDB_ebs_patch');
  DBMS_SERVICE.DELETE_SERVICE(service_name =>'UATPDB_ebs_patch');
end;
/
SQL>
SQL> commit;

Commit complete.

SQL> create pfile from spfile;

File created.

SQL>
SQL> shut immediate
[oracle@clouduatdb ~]$ grep service_names $ORACLE_HOME/dbs/initUATCDB.ora
*.service_names='UATPDB'
[oracle@clouduatdb UATPDB_clouduatdb]$ echo $ORACLE_SID
UATCDB
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status uatcdb (check whether ebs_patch service added or not)
[oracle@clouduatdb UATPDB_clouduatdb]$ . $ORACLE_HOME/UATPDB_clouduatdb.env
[oracle@clouduatdb UATPDB_clouduatdb]$
[oracle@clouduatdb UATPDB_clouduatdb]$ perl $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/log/UATPDB_clouduatdb/08031734/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u01/app/oracle/product/12.1.0.2/dbhome_1

Value of s_dbcset is AR8MSWIN1256

Character set is not present in the allowed list. Need to add orai18n.jar to the CLASSPATH.

Library orai18n.jar exists.

Value of s_dbcset is AR8MSWIN1256

Character set is not present in the allowed list. Need to add orai18n.jar to the CLASSPATH.

Library orai18n.jar exists.
        Classpath                   : :/u01/app/oracle/product/12.1.0.2/dbhome_1/jdbc/lib/ojdbc6.jar:/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/java/xmlparserv2.jar:/u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/java:/u01/app/oracle/product/12.1.0.2/dbhome_1/jlib/netcfg.jar:/u01/app/oracle/product/12.1.0.2/dbhome_1/jlib/ldapjclnt12.jar:/u01/app/oracle/product/12.1.0.2/dbhome_1/jlib/orai18n.jar

        Using Context file          : /u01/app/oracle/product/12.1.0.2/dbhome_1/appsutil/UATPDB_clouduatdb.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.
[oracle@clouduatdb UATPDB_clouduatdb]$
[oracle@clouduatdb UATPDB_clouduatdb]$ lsnrctl status UATCDB

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 03-AUG-2021 17:36:38

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     UATCDB
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                03-AUG-2021 17:29:06
Uptime                    0 days 0 hr. 7 min. 31 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0.2/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/12.1.0.2/dbhome_1/log/diag/tnslsnr/clouduatdb/UATCDB/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=clouduatdb.uatsn.cloudprdvcn1.oraclevcn.com)(PORT=1521)))
Services Summary...
Service "PRODCDBXDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATCDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
Service "UATPDB_ebs_patch" has 1 instance(s).
  Instance "UATCDB", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@clouduatdb UATPDB_clouduatdb]$

Check db alert log
SELECT value from v$diag_info where name='Diag Trace';
[root@clouduatdb bin]# tailf /u01/app/oracle/diag/rdbms/UATCDB/UATCDB/trace/alert_UATCDB.log
There should no tns error and warnings.
Fatal NI connect error 12170.
TNS-12535: TNS:operation timed out





Below this not required for DB Cloning:
After the pdb status is o run below:
Cd $ORACLE_HOME/appsutil/install/UATPDB_clouduatdb
sqlplus / as sysdba @adupdlib.sql so


Section 5: Clone and Configure the Application Tier
5.1 Copy the Application Tier File System from the Source Node to the Target Node
5.2 Configure the Target System Application Tier Nodes

 and prepare apps tier for cloning. Setting up oraInventory before adcfg on apps:
Before mv oraInst.loc in 10.1.2 home and remove old entries from global inventory
This will be recreated if there is only one application running on server. 
[applmgr@uatcloudapp ~]$ cd /u01/uat/fs1/EBSapps/10.1.2/
[applmgr@uatcloudapp 10.1.2]$ mv oraInst.loc oraInst.loc_03082021
[applmgr@uatcloudapp ~]$ find /u01/uat/fs1 -iname oraInst.loc*
/u01/uat/fs1/EBSapps/10.1.2/oraInst.loc_03082021 
[applmgr@uatcloudapp ~]$ cat /etc/oraInst.loc
inventory_loc=/home/applmgr/oraInventory
inst_group=oinstall
[applmgr@uatcloudapp ~]$ 
[applmgr@uatcloudapp ~]$ cd /home/applmgr/oraInventory
[applmgr@uatcloudapp oraInventory]$ ls -ltr
total 32
drwxrwx---. 2 applmgr oinstall   22 Jun 24 07:42 oui
drwxrwx---. 7 applmgr oinstall 4096 Jun 24 08:56 backup
-rw-rw----. 1 applmgr oinstall  303 Jun 24 08:58 oraInstaller.properties
drwxrwx---. 2 applmgr oinstall 4096 Jun 24 09:00 Contents
-rw-rw----. 1 applmgr oinstall   37 Jun 24 09:00 install.platform
drwxrwx---. 2 applmgr oinstall 4096 Aug  3 17:38 ContentsXML
drwxrwx---. 2 applmgr oinstall 8192 Aug  3 17:53 logs
[applmgr@uatcloudapp oraInventory]$ rm -rf *
[applmgr@uatcloudapp oraInventory]$
[applmgr@uatcloudapp oraInventory]$ ls -ltr
total 0
[applmgr@uatcloudapp oraInventory]$



Finally run adcfgclone: <- update sslterminator immediately after dual fs
[root@uatcloudapp u01]# 
[root@uatcloudapp u01]# chown -R applmgr:oinstall /u01/uat (approx. 20 mins)
[applmgr@uatcloudapp ~]$ mkdir -p /u01/uat/fs2
[applmgr@uatcloudapp ~]$ du -sh /u01/uat
65G     /u01/uat
[applmgr@uatcloudapp ~]$ cd /u01/AHCTEST/fs2/EBSapps/comn/clone/bin

export TIMEDPROCESS_TIMEOUT=-1
export CONFIG_JVM_ARG="-Xms1024m -Xmx2048m"
export CONFIG_JVM_ARG="-Xms2048m -Xmx4096m"
echo $CONFIG_JVM_ARG

perl adcfgclone.pl appsTier dualfs
Enter PDB name for DB UATPDB

Remove https login from context file after completing dualfs in OCI
[applmgr@uatcloudapp ~]$ grep sslterminator $CONTEXT_FILE (update ssl setting before running autoconfig replace with below line / add yellow part)
<sslterminator oa_var="s_enable_sslterminator">#</sslterminator>




  
Important Variable of Context file which need update after clone:
 
cp $CONTEXT_FILE ~
grep sslterminator $CONTEXT_FILE  ÃŸ update # to stop redirecting to https in login page
grep deny $EBS_DOMAIN_HOME/config/config.xml  ÃŸ deny
grep site_name $CONTEXT_FILE   (Site name update)
grep s_wfmail_agent $CONTEXT_FILE   (for WFM Profile)
grep networkRetries $ORA_CONFIG_HOME/10.1.2/forms/server/appsweb.cfg (value 30)
grep FORMS_CATCHTERM $ORA_CONFIG_HOME/10.1.2/forms/server/default.env (value 0)
grep s_forms_catchterm $CONTEXT_FILE  (Doc ID 745711.1) (Doc ID 744143.1)
grep s_forms_time $CONTEXT_FILE

For Workflow mailer Account update from itinfocloudp to itinfocloudt
grep s_javamailer_reply_to $CONTEXT_FILE
grep s_javamailer_imap_user $CONTEXT_FILE
grep s_javamailer_outbound_user $CONTEXT_FILE
 
grep s_javamailer_imaphost $CONTEXT_FILE
grep s_javamailer_imapdomainname $CONTEXT_FILE
grep s_smtphost $CONTEXT_FILE
grep s_smtpdomainname $CONTEXT_FILE

Workflow Mailer Profile (Doc ID 2493004.1)
Make Sure if you have load balancer configured for the application, then you have to provide site value as per your primary node and remove "s" if ssl is configured use below variable of context file and build value as shown in example:
grep s_wfmail_agent $CONTEXT_FILE   (for WFM Profile)
grep s_webhost $CONTEXT_FILE
grep s_domainname $CONTEXT_FILE
grep s_webport $CONTEXT_FILE
Ex: http://myhost.mydomain.com:8000
WF: Workflow Mailer Framework Web Agent
Site: URL without s in httpsgrep login $CONTEXT_FILE

You need to run autoconfig again if updated any further in xml file

After cloning before starting the services perform post clone steps to  cancel schedule requests and wf mails change apps pswd reduce FNDLIBR.


Follow here to continue post clone steps in APPS:
Check whether given Application ports RFS/PFS including weblogic server ports are open or not in OCI:
[root@ebstestapp ~]# sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens3
  sources:
  services: dhcpv6-client ssh
  ports: 8002/tcp 7003/tcp 7021/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

[root@ebstestapp ~]#

APPS Tier Cloning is Completed Successfully:


Steps need to be done after ceating new server in OCI:
RPM’s and Preinstall Steps:
# yum -y install oracle-ebs-server-R12-preinstall
# yum -y install openmotif21
# yum -y install xorg-x11-libs-compat

set oracle Central Inventory cat /etc/oraInst.loc
update /etc/hosts add database server entries

user applmgr user to untar the file or will take lot of time execute chow command
tar-xvzf 2021-06-21-fs1.tar.gz -C /u01/uat

Open port for Newly created Apps(Compute) in OCI:  (only on new server installation or new/extra APPS Port)
firewall-cmd --list-all
systemctl status firewalld
sudo firewall-cmd --permanent --add-port=8001/tcp
sudo firewall-cmd --reload
now start the services…..

also open the weblogic console port….
sudo firewall-cmd --permanent --add-port=7002/tcp
sudo firewall-cmd –reload

Install CUPS for printing service on Newly created Apps(Compute) in OCI ….
[root@uatcloudapp ~]# yum -y install cups 
[root@uatcloudapp ~]# rpm -qa cups
cups-1.6.3-51.el7.x86_64
[root@uatcloudapp ~]# 
[root@uatcloudapp ~]# systemctl status cups
● cups.service - CUPS Printing Service
   Loaded: loaded (/usr/lib/systemd/system/cups.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
[root@uatcloudapp ~]# 
[root@uatcloudapp ~]# systemctl stop cups
[root@uatcloudapp ~]#
[root@uatcloudapp ~]# systemctl start cups
[root@uatcloudapp ~]#
 ● cups.service - CUPS Printing Service
   Loaded: loaded (/usr/lib/systemd/system/cups.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-08-10 08:08:03 GMT; 2s ago
Main PID: 8186 (cupsd)
   Memory: 1.5M
   CGroup: /system.slice/cups.service
           └─8186 /usr/sbin/cupsd -f
 
Aug 10 08:08:03 uatcloudapp systemd[1]: Started CUPS Printing Service.
[root@uatcloudapp ~]# lpadmin -p testprinter -E -v socket://192.168.140.20



APPS Tier Cloning is Completed Successfully:

root@ebsprodapp3:~#


Thanks for Reading


Regards,
Mohammed Areefuddin.

No comments:

Post a Comment