################################################################################################################
################################################################################################################
5.3 Adding a New Application Tier Node to an Existing System
Sharing The Application Tier File System in Oracle E-Business Suite Release 12.2 (Doc ID 1375769.1) EBS - Technology Area - Webcast Recording 'E-Business Suite - RAC & Parallel Concurrent Processing (PCP)' [video] (Doc ID 1359612.1)
To check all topics click below link:
###################################################################################################
Pre-checks
before Add EBS Node:
###################################################################################################
select NODE_NAME from fnd_nodes;
select NODE_NAME from ADOP_VALID_NODES; -- prod apps entries
select NODENAME from ad_nodes_config_status; -- prod apps entries
select NODE_NAME from fnd_oam_context_files; --3:DB(template,metadata,xml) 8:apps( 4-RFS, 4-PFS )
verify Context variables parameters:
grep _cluster $CONTEXT_FILE
grep oacore_server $CONTEXT_FILE
grep oa_managed_server $CONTEXT_FILE
grep _server_ports $CONTEXT_FILE
grep s_shared_file_system $CONTEXT_FILE
grep s_appldcp $CONTEXT_FILE
verify Profile Values
Concurrent:TM Transport Type --> QUEUE
Concurrent:PCP Instance Check --> ON
select * from v$restore_point;
create restore point before_addnode guarantee flashback database;
--flashback database to restore point BEFORE_ADDNODE;-- for Restore
[applmgr@ebsprodapp3 ~]$ netstat
-tulpn |grep LISTEN
netstat -tulpn |grep LISTEN |grep /java
netstat -tulpn |grep LISTEN |grep /httpd.worker
netstat -tulpn |grep LISTEN |grep /tnslsnr
netstat -tulpn
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:9003 0.0.0.0:* LISTEN -
tcp 0 0 172.168.10.24:7211 0.0.0.0:* LISTEN 7105/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN -
tcp 0 0 172.168.10.24:7411 0.0.0.0:* LISTEN 7107/java
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:9237 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
tcp 0 0 172.168.10.24:12345 0.0.0.0:* LISTEN 7484/java
tcp 0 0 0.0.0.0:10009 0.0.0.0:* LISTEN 4127/httpd.worker
tcp 0 0 0.0.0.0:1626 0.0.0.0:* LISTEN 5033/tnslsnr
tcp 0 0 172.168.10.24:7611 0.0.0.0:* LISTEN 7103/java
tcp 0 0 0.0.0.0:21821 0.0.0.0:* LISTEN -
tcp 0 0 172.168.10.24:5566 0.0.0.0:* LISTEN 4688/java
tcp 0 0 127.0.0.1:6110 0.0.0.0:* LISTEN 3956/opmn
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 4127/httpd.worker
tcp 0 0 127.0.0.1:32769 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:6210 0.0.0.0:* LISTEN 3956/opmn
tcp 0 0 172.168.10.24:7011 0.0.0.0:* LISTEN 5391/java
tcp 0 0 0.0.0.0:4453 0.0.0.0:* LISTEN 4127/httpd.worker
tcp6 0 0 :::3306 :::* LISTEN -
tcp6 0 0 :::111 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::35007 :::* LISTEN -
tcp6 0 0 :::47651 :::* LISTEN -
tcp6 0 0 :::33060 :::* LISTEN -
[applmgr@ebsprodapp3 ~]$
[root@ebsprodapp3 ~]# firewall-cmd --list-all
ports: 8000/tcp 7001/tcp 7011/tcp
[applmgr@ebsprodapp4 ~]$ nc
-zv ebsprodapp3 7011
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.168.10.24:7011.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[applmgr@ebsprodapp4 ~]$
[applmgr@ebsprodapp4 ~]$ nc
-zv ebsprodapp3 8000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.168.10.24:8000.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[applmgr@ebsprodapp4 ~]$
###################################################################################################
Create
SSH connectivity between EBS Node:
###################################################################################################
[applmgr@ebsprodapp4 ~]$ ssh
ebsprodapp3
Last login: Sat Feb 12 21:00:53 2022 from ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
[applmgr@ebsprodapp3 ~]$
*Steps to setup SSH using txkRunSSHSetup.pl Script on EBS Application Nodes
Important: If you change the password for the relevant operating system account on one or more nodes, you must regenerate the SSH credentials either using the $AD_TOP/patch/115/bin/txkRunSSHSetup.pl script, or your own native solution if you prefer.
The txkRunSSHSetup.pl script has a -help option that shows relevant usage options.
For example, a basic command to enable ssh would be:
$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$
To verify ssh operation:
$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl verifyssh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3 -invalidnodefile=<filename to report ssh verification failures>
To disable ssh:
$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl disablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3 -invalidnodefile=<filename to report ssh verification failures>
Alternate steps to ENABLE SSH Between nodes (on Primary Node and Secondary) when failed ssh in OL8:
perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl enablessh -contextfile=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/appl/admin/PRODPDB_ebsprodapp3.xml -hosts=ebsprodapp3.areefdba.com.sa, ebsprodapp4.areefdba.com.sa
If failed with the above txkRunSSHSetup.pl the use below method for ssh connection
[applmgr@ebsprodapp3 ~]$ rm -rf .ssh
[applmgr@ebsprodapp3 ~]$ ssh-keygen -t rsa
[applmgr@ebsprodapp3 ~]$ ssh-copy-id ebsprodapp4
[applmgr@ebsprodapp3 ~]$ ssh ebsprodapp4
[applmgr@ebsprodapp4 ~]$ rm -rf .ssh
[applmgr@ebsprodapp4 ~]$ ssh-keygen -t rsa
[applmgr@ebsprodapp4 ~]$ ssh-copy-id ebsprodapp3
[applmgr@ebsprodapp4 ~]$ ssh ebsprodapp3
###################################################################################################
Complete
adpreclone and Start Weblogic server in Patch FS:
###################################################################################################
[applmgr@ebsprodapp3 ~]$ ls -ltrh $INST_TOP/admin/log/clone/StageAppsTier_*.log (verify when last ran preclone)
[applmgr@ebsprodapp3 ~]$ adpreclone.pl
appsTier
[applmgr@ebsprodapp3 ~]$ .
/u01/OCI/EBSapps.env patch
E-Business Suite Environment Information
----------------------------------------
RUN File System : /u01/OCI/fs1/EBSapps/appl
PATCH File System : /u01/OCI/fs2/EBSapps/appl
Non-Editioned File System : /u01/OCI/fs_ne
DB Host: prd-scan.prddbsn.ahccloudprdvcn1.oraclevcn.com Service/SID: PRODPDB
Sourcing the PATCH File System ...
[applmgr@ebsprodapp3 ~]$ adadminsrvctl.sh
start forcepatchfs
You are running adadminsrvctl.sh version 120.10.12020000.11
Enter the WebLogic Admin password:
Enter the APPS Schema password:
Starting WLS Admin Server...
Refer /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/admin/log/adadminsrvctl.txt for details
AdminServer logs are located at /u01/OCI/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/admin/log/adadminsrvctl.txt for more information ...
[applmgr@ebsprodapp3 ~]$ adadminsrvctl.sh
status
###################################################################################################
Create a pairs file in new node INST_TOP(APP4) form latest
backup in Primary app node
update section 1 [Services To be Enabled on the Secondary Application
Tier Node]
update section 2 [Instance Specific]
###################################################################################################
[applmgr@ebsprodapp3 admin]$ cp
$INST_TOP/appl/admin/PRODPDB_ebsprodapp3.txt /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/pairsapp4.txt
[applmgr@ebsprodapp4 admin]$ pwd
/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin
[applmgr@ebsprodapp4 admin]$ cat
pairsapp4.txt
# $Header: pairs_file_dualfsclone_ux_txt.tmp 120.0.12020000.2 2015/07/29 19:36:02 jmajumde noship $
# Copyright (c) 2005, 2015 Oracle and/or its affiliates.
# All rights reserved.
# Version 12.0.0
#
# This is a sample pairs file generated by the autoconfig tool based on the
# context values set on the primary application tier node.
# This pairs file can be modified and used when adding additional
# application tier nodes. Some of the parameters set in this pairs file
# assume that you are using a shared file system.
[Base]
s_base=/u01/OCI
s_current_base=/u01/OCI/fs2
s_other_base=/u01/OCI/fs1
s_ne_base=/u01/OCI/fs_ne
[General]
s_applptmp=/usr/tmp
s_appsgroup=oinstall
s_appsuser=applmgr
s_dbuser=oracle
s_dbgroup=oinstall
s_dbdomain=prddbsn.ahccloudprdvcn1.oraclevcn.com
s_at=/u01/OCI/fs2/EBSapps/appl
s_com=/u01/OCI/fs2/EBSapps/comn
s_tools_oh=/u01/OCI/fs2/EBSapps/10.1.2
s_weboh_oh=/u01/OCI/fs2/FMW_Home/webtier
s_fmw_home=/u01/OCI/fs2/FMW_Home
s_dbGlnam=PRODPDB
s_dbSid=PRODPDB
s_dbhost=prd-scan
s_clonestage=/u01/OCI/fs2/EBSapps/comn/clone
s_dbport=1521
s_options_symlinks=Options -FollowSymLinks
s_proxyhost=
s_proxybypassdomain=prdappsn.ahccloudprdvcn1.oraclevcn.com
s_proxyport=80
s_nonproxyhosts=
s_javamailer_imapdomainname=office365.com
s_javamailer_imaphost=outlook
s_javamailer_reply_to=itinfocloudp@areefdba.com.sa
s_javamailer_outbound_user=itinfocloudp@areefdba.com.sa
s_smtphost=smtp
s_smtpdomainname=office365.com
s_file_edition_type=run
s_port_pool=10
patch_s_port_pool=0
s_admhost=ebsprodapp3
s_atName=ebsprodapp3
s_shared_file_system=true
[Web Entry Point Configuration]
s_webentryurlprotocol=https
s_webentryhost=ebsprodapp
s_webentrydomain=areefdba.com.sa
s_active_webport=443
s_endUserMonitoringURL=http://ebsprodapp3.prdappsn.ahccloudprdvcn1.oraclevcn.com:8000/oracle_smp_chronos/oracle_smp_chronos_sdk.gif
s_external_url=https://ebsprodapp.areefdba.com.sa
s_login_page=https://ebsprodapp.areefdba.com.sa/OA_HTML/AppsLogin
[Instance Specific]
# Please provide values for the context variables listed below. On the source
# instance they are instantiated as shown in the comment section below.
# These values should only be used as reference to fill out the instance
# values for the new node.
#s_temp=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/temp
#s_contextname=PRODPDB_ebsprodapp3
#s_hostname=ebsprodapp3
#s_domainname=prdappsn.ahccloudprdvcn1.oraclevcn.com
#s_cphost=ebsprodapp3
#s_webhost=ebsprodapp3
#s_config_home=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3
#s_inst_base=/u01/OCI
#s_display=ebsprodapp3:0.0
#s_forms-c4ws_display=ebsprodapp3:0.0
#s_ohs_instance=EBS_web_OHS
#s_webport=8000
#s_http_listen_parameter=8000
#s_https_listen_parameter=4453
s_temp=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/temp
s_contextname=PRODPDB_ebsprodapp4
s_hostname=ebsprodapp4
s_domainname=prdappsn.ahccloudprdvcn1.oraclevcn.com
s_cphost=ebsprodapp4
s_webhost=ebsprodapp4
s_config_home=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4
s_inst_base=/u01/OCI
s_display=ebsprodapp4:0.0
s_forms-c4ws_display=ebsprodapp4:0.0
s_ohs_instance=EBS_web_OHS1####--------->increase this number as one additional is added
s_webport=8000
s_http_listen_parameter=8000
s_https_listen_parameter=4453
[Services]
# Please provide values for the context variables listed below .
# Enter "enabled" without the quotes to enable the service on the new node .
# Enter "disabled" without the quotes to disable the service on the new node .
# The Root service include the Node Manager .
# The Web Application Services include the Node Manager, Admin Server,
# Managed Servers ( oacore, forms, oafm, formsc4-ws).
# To enable the configuration of Node Manager and the Managed Servers,
# set s_web_applications_status to enabled
# The Web Entry Services include the OPMN and Oracle HTTP Server .
# To enable the configuration of OPMN,OHS , set s_webentry_status and s_apcstatus to enabled.
# The Batch Processing Services include the FNDFS Listener, Concurrent Mgr, ICSM# and JTF FullFillment Server
# To enable the configuration of Concurrent Manager# set s_batch_status to enabled.
# The Other Services group include the Forms Metric Server, Forms Metric Clients# Forms Server for the socket mode configuration and Mobile Web Application
# Server (MWA).
# s_adminserverstatus is set to disabled since this service can only be enabled
# on the primary application tier.
# The services enabled on the primary application tier node are as shown
# below in the comment section.
[Services Enabled on the Primary Application Tier Node]
#s_web_applications_status=enabled
#s_web_entry_status=enabled
#s_apcstatus=enabled
#s_root_status=enabled
#s_batch_status=enabled
#s_other_service_group_status=disabled
#s_adminserverstatus=enabled
#s_web_admin_status=enabled
[Services To be Enabled on the Secondary Application Tier Node]
s_web_applications_status=enabled
s_web_entry_status=enabled
s_apcstatus=enabled
s_root_status=enabled
s_batch_status=enabled
s_other_service_group_status=enabled
s_adminserverstatus=disabled
s_web_admin_status=disabled
# Additional variables can be added below as per your requirement.
[applmgr@ebsprodapp4 admin]$
###################################################################################################
Connection
Filter Rule for RFS & PFS for new node app4
###################################################################################################
[applmgr@ebsprodapp3 ~]$ cd
$EBS_DOMAIN_HOME/config
[applmgr@ebsprodapp3 config]$ vi config.xml
[applmgr@ebsprodapp3 config]$ grep
connection-filter-rule $EBS_DOMAIN_HOME/config/config.xml
<connection-filter-rule>ebsprodapp3.prdappsn.ahccloudprdvcn1.oraclevcn.com * * allow</connection-filter-rule>
<connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>
<connection-filter-rule>192.168.140.200 * 7011 allow http #AC</connection-filter-rule>
<connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>
[applmgr@ebsprodapp3 config]$
[applmgr@ebsprodapp4 config]$ hostname
-f
ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
[applmgr@ebsprodapp3 config]$ perl
$AD_TOP/patch/115/bin/txkWLSConnectionFilterManager.pl
-contextfile=$CONTEXT_FILE -option=add-filter-rule -hostname=ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
Enter WLS Admin Password:
Log: /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/rgf/TXK/ConnectionFilter_Sun_Feb_13_01_15_07_2022/txkWLSConnectionFilterManager_Sun_Feb_13_01_15_07_2022.log
=======================
Validating context file
=======================
Script Name: txkWLSConnectionFilterManager.pl
Started: Sun Feb 13 01:15:13 +03 2022
-----------
Values used
-----------
Context File: /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/appl/admin/PRODPDB_ebsprodapp3.xml
Option: add-filter-rule
OUT Directory: /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/rgf/TXK/ConnectionFilter_Sun_Feb_13_01_15_07_2022
Console Logging: ON
Hostname: ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
========================================
Validating Oracle Weblogic AdminServer status...
========================================
Executing SYSTEM command: perl /u01/OCI/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/appl/admin/PRODPDB_ebsprodapp3.xml -servername=AdminServer -logfile=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/rgf/TXK/ConnectionFilter_Sun_Feb_13_01_15_07_2022/EBSProvisioner.log -promptmsg=hide
EXIT STATUS: 0
Oracle Weblogic AdminServer is RUNNING.
Executing command: /u01/OCI/fs2/EBSapps/comn/util/jdk32/jre/bin/java -classpath /u01/OCI/fs2/EBSapps/comn/java/classes:/u01/OCI/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/OCI/fs2/EBSapps/comn/java/classes/oracle/apps/fnd/jar/wlsconnfilter.jar:/u01/OCI/fs2/FMW_Home/webtier/lib/xmlparserv2.jar oracle.apps.ad.tools.configuration.RegisterNodePreReq add-filter-rule -contextfile /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/appl/admin/PRODPDB_ebsprodapp3.xml -logfile /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/rgf/TXK/ConnectionFilter_Sun_Feb_13_01_15_07_2022/RegisterNodePreReq.log -hostname ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
For details please refer log file /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/logs/appl/rgf/TXK/ConnectionFilter_Sun_Feb_13_01_15_07_2022/RegisterNodePreReq.log
Enter the WebLogic Server Admin Password:Enter the WebLogic Server Admin Password:add-filter-rule option executed successfully.
Ended: Sun Feb 13 01:15:24 +03 2022
[applmgr@ebsprodapp3 config]$
[applmgr@ebsprodapp3 config]$ grep
connection-filter-rule $EBS_DOMAIN_HOME/config/config.xml
<connection-filter-rule>ebsprodapp3.prdappsn.ahccloudprdvcn1.oraclevcn.com * * allow</connection-filter-rule>
<connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>
<connection-filter-rule>192.168.140.200 * 7011 allow http #AC</connection-filter-rule>
<connection-filter-rule>ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
* * allow
#MT</connection-filter-rule>
<connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>
[applmgr@ebsprodapp3 config]$
[applmgr@ebsprodapp3 config]$
add filter rule for patch file system also make sure PFS admin server is up:
[applmgr@ebsprodapp3 ~]$ .
/u01/OCI/EBSapps.env patch
E-Business Suite Environment Information
----------------------------------------
RUN File System : /u01/OCI/fs2/EBSapps/appl
PATCH File System : /u01/OCI/fs1/EBSapps/appl
Non-Editioned File System : /u01/OCI/fs_ne
DB Host: prd-scan.prddbsn.ahccloudprdvcn1.oraclevcn.com Service/SID: PRODPDB
Sourcing the PATCH File System ...
[applmgr@ebsprodapp3 ~]$ adadminsrvctl.sh
status
[applmgr@ebsprodapp3 ~]$ perl
$AD_TOP/patch/115/bin/txkWLSConnectionFilterManager.pl
-contextfile=$CONTEXT_FILE -option=add-filter-rule
-hostname=ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
Enter WLS Admin Password:
Log: /u01/OCI/fs1/inst/apps/PRODPDB_ebsprodapp3/logs/appl/rgf/TXK/ConnectionFilter_Sun_Feb_13_01_16_04_2022/txkWLSConnectionFilterManager_Sun_Feb_13_01_16_04_2022.log
[applmgr@ebsprodapp3 ~]$
###################################################################################################
5.3 Adding a New Application Tier Node to an Existing System
[applmgr@ebsprodapp3 ~]$ grep tcp.invited_nodes $TNS_ADMIN/sqlnet.ora
tcp.invited_nodes = (ebsproddb02.areefdba.com.sa, ebsproddb02-vip.areefdba.com.sa, ebsprodapp3.areefdba.com.sa, ebsproddb01.areefdba.com.sa, ebsproddb01-vip.areefdba.com.sa, ebsprodapp4.areefdba.com.sa)
###################################################################################################
make
a backup copy of apps.conf and mod_wl_ohs before adding a node
###################################################################################################
[applmgr@ebsprodapp3 ~]$ grep
ohs_instance_loc $CONTEXT_FILE
<ohs_instance_loc oa_var="s_ohs_instance_loc">/u01/OCI/fs2/FMW_Home/webtier/instances/EBS_web_OHS</ohs_instance_loc>
[applmgr@ebsprodapp3 ~]$
find $IAS_ORACLE_HOME/instances -type f \( -iname \apps.conf -o
-iname \mod_wl_ohs.conf \)
find $IAS_ORACLE_HOME/instances -iname apps.conf
find $IAS_ORACLE_HOME/instances -iname mod_wl_ohs.conf
cp apps.conf apps.conf_bkp
cp mod_wl_ohs.conf
mod_wl_ohs.conf_bkp
[applmgr@ebsprodapp3 ~]$ grep ebsprodapp4 $IAS_ORACLE_HOME/instances/EBS_web_OHS*/config/OHS/EBS_web/apps.conf
[applmgr@ebsprodapp3 ~]$ grep ebsprodapp4 $IAS_ORACLE_HOME/instances/EBS_web_OHS*/config/OHS/EBS_web/mod_wl_ohs.conf
###################################################################################################
Run
addnode and Configure the Secondary Node app4 with dualfs (approx. 30 Mins) on shared file system.
###################################################################################################
Remove
Env sourcing for adding node (app4)
[applmgr@ebsprodapp4 ~]$ cat
.bash_profile
#. /u01/OCI/EBSapps.env run
[applmgr@ebsprodapp4 ~]$ exit
logout
[root@ebsprodapp4 OCI]# su
- applmgr
Last login: Sun Feb 13 01:17:02 +03 2022 from ebsprodapp3.prdappsn.ahccloudprdvcn1.oraclevcn.com on pts/2
[applmgr@ebsprodapp4 ~]$
[applmgr@ebsprodapp4 ~]$ echo
$APPL_TOP
[applmgr@ebsprodapp4 ~]$
[applmgr@ebsprodapp4 ~]$ cd
/u01/OCI/fs2/EBSapps/comn/clone/bin/
[applmgr@ebsprodapp4 bin]$ ls -ltrh
total 328K
-rwxr-xr-x. 1 applmgr oinstall 98K Dec 13 21:32 adclone.pl
-rwxr-xr-x. 1 applmgr oinstall 5.0K Dec 13 21:32 adchkutl.sh
-rwxr-xr-x. 1 applmgr oinstall 48K Dec 13 21:32 adclonectx.pl
-rwxr-xr-x. 1 applmgr oinstall 126K Dec 13 21:32 adcfgclone.pl
-rwxr-xr-x. 1 applmgr oinstall 11K Dec 13 21:32 adaddnode.pl
[applmgr@ebsprodapp4 bin]$ time perl
adclonectx.pl addnode contextfile=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/appl/admin/PRODPDB_ebsprodapp3.xml
pairsfile=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/pairsapp4.txt dualfs=yes (approx 20 Mins)
Copyright (c) 2011, 2015 Oracle Corporation
Redwood Shores, California, USA
Oracle E-Business Suite Rapid Clone
Version 12.2
adclonectx Version 120.30.12020000.22
Enter the APPS passwd :
Enter the Weblogic AdminServer password :
Executing command: /u01/OCI/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -classpath /u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/ojdbc6.jar:/u01/OCI/fs2/EBSapp s/comn/clone/bin/../jlib/xmlparserv2.jar:/u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/java::/u01/OCI/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/wlfullclient.ja r:/u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/wlclient.jar:/u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/wljmxclient.jar:/u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/ojmisc.jar:/u01/OCI/fs2/EBSapps/comn/clone/bin/ ../jlib/obfuscatepassword.jar:/u01/OCI/fs2/EBSapps/comn/clone/bin/../jlib/emCfg.jar oracle.apps.ad.clone.RCloneSFSAddNode -contextfile /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp3/appl/admin/PRODPDB_ahccloud prodapp3.xml -configfile /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/pairsapp4.txt -machinelistenaddress ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com -logdir /u01/OCI/fs2/EBSapps/comn/ clone/bin/../FMW/logs/sfslog -promptmsg hide
Verifying: Apps schema password
Verifying:
pairsfile /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/pairsapp4.txt
Run file system AdminServer is running
Patch file system AdminServer is running
Run file system domain
Patch file system domain
Executing: run clone context...
Log file located at /u01/OCI/fs2/EBSapps/comn/clone/bin/../FMW/logs/sfslog/run/RCloneSFSAddNode_02130123.log
Target System Base Directory set to /u01/OCI
Target System Current File System Base set to /u01/OCI/fs2
Target System Other File System Base set to /u01/OCI/fs1
Target System Fusion Middleware Home set to /u01/OCI/fs2/FMW_Home
Target System Other File System Fusion Middleware Home set to /u01/OCI/fs1/FMW_Home
Target System Web Oracle Home set to /u01/OCI/fs2/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u01/OCI/fs1/FMW_Home/webtier
Target System Appl TOP set to /u01/OCI/fs2/EBSapps/appl
Target System Other File System Appl TOP set to /u01/OCI/fs1/EBSapps/appl
Target System COMMON TOP set to /u01/OCI/fs2/EBSapps/comn
Target System Other File System COMMON TOP set to /u01/OCI/fs1/EBSapps/comn
Target System Current File System Instance Top set to /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4
Report file located at /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/temp/portpool.lst
The new APPL_TOP context file has been created :
/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/PRODPDB_ebsprodapp4.xml
contextfile=/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/PRODPDB_ebsprodapp4.xml
Configuring: run fs add node...
Executing: patch clone context...
Log file located at /u01/OCI/fs2/EBSapps/comn/clone/bin/../FMW/logs/sfslog/patch/RCloneSFSAddNode_02130130.log
Target System Other File System Instance Top set to /u01/OCI/fs1/inst/apps/PRODPDB_ebsprodapp4
Report file located at /u01/OCI/fs1/inst/apps/PRODPDB_ebsprodapp4/temp/portpool.lst
The new APPL_TOP context file has been created :
/u01/OCI/fs1/inst/apps/PRODPDB_ebsprodapp4/appl/admin/PRODPDB_ebsprodapp4.xml
contextfile=/u01/OCI/fs1/inst/apps/PRODPDB_ebsprodapp4/appl/admin/PRODPDB_ebsprodapp4.xml
Configuring: patch fs add node...
-- Add node operation completed --
Node ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com has been added successfully
[applmgr@ebsprodapp4 bin]$
[applmgr@ebsprodapp4 admin]$
[applmgr@ebsprodapp4 admin]$ echo $CONTEXT_FILE
[applmgr@ebsprodapp4 admin]$ grep
s_shared
/u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/PRODPDB_ebsprodapp4.xml
<shared_file_system oa_var="s_shared_file_system">true</shared_file_system>
[applmgr@ebsprodapp4 admin]$
[applmgr@ebsprodapp4 admin]$ grep appldcp /u01/OCI/fs2/inst/apps/PRODPDB_ebsprodapp4/appl/admin/PRODPDB_ebsprodapp4.xml
[applmgr@ebsprodapp4 ~]$ vi
.bash_profile
[applmgr@ebsprodapp4 ~]$ exit
logout
[root@ebsprodapp4 OCI]# su
- applmgr
Last login: Sun Feb 13 01:21:01 +03 2022 on pts/0
E-Business Suite Environment Information
----------------------------------------
RUN File System : /u01/OCI/fs2/EBSapps/appl
PATCH File System : /u01/OCI/fs1/EBSapps/appl
Non-Editioned File System : /u01/OCI/fs_ne
DB Host: prd-scan.prddbsn.ahccloudprdvcn1.oraclevcn.com Service/SID: PRODPDB
Sourcing the RUN File System ...
[applmgr@ebsprodapp4 ~]$ adautocfg.sh
Enter the APPS user password:
AutoConfig completed successfully.
[applmgr@ebsprodapp4 ~]$
[applmgr@ebsprodapp3 ~]$ adautocfg.sh
AutoConfig completed successfully.
[applmgr@ebsprodapp3 ~]$
[applmgr@ebsprodapp4 ~]$ ps -ef | grep -i applmgr
root 8293 8147 0 01:01 pts/1 00:00:00 sudo su - applmgr
root 8296 8293 0 01:01 pts/1 00:00:00 su - applmgr
applmgr 8297 8296 0 01:01 pts/1 00:00:00 -bash
root 10660 29749 0 01:38 pts/0 00:00:00 su - applmgr
applmgr 10662 10660 0 01:38 pts/0 00:00:00 -bash
applmgr 19947 10662 0 01:41 pts/0 00:00:00 ps -ef
applmgr 19948 10662 0 01:41 pts/0 00:00:00 grep --color=auto -i applmgr
root 21705 21701 0 00:01 ? 00:00:00 /bin/sh -c
[applmgr@ebsprodapp4 ~]$
[applmgr@ebsprodapp4 ~]$ adstrtal.sh
apps/appspswd
[applmgr@ebsprodapp4 ~]$ adstrtal.sh
apps/appspswd -msimode
for shared CM on both nodes with port Application port 1626 (FNDFS)
[applmgr@ebsprodapp3 ~]$ grep FNDFS -A3 $TNS_ADMIN/tnsnames.ora
[applmgr@ebsprodapp3 ~]$ tnsping FNDFS_$HOSTNAME
Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=ebsprodapp3.psau.edu.sa)(PORT=1626)) (CONNECT_DATA= (SID=FNDFS)))
[applmgr@ebsprodapp3 ~]$ nc -zv ebsprodapp4 1626
[applmgr@ebsprodapp4 ~]$ tnsping FNDFS_$HOSTNAME
Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=ebsprodapp4.psau.edu.sa)(PORT=1626)) (CONNECT_DATA= (SID=FNDFS)))
[applmgr@ebsprodapp4 ~]$ nc -zv ebsprodapp3 1626
Add port to allow in Firewall
# firewall-cmd --state
running
# firewall-cmd --list-all
# firewall-cmd --add-port=1626/tcp --perm (default FNDFS port for shared CM)
# firewall-cmd --reload
# firewall-cmd --list-all
###################################################################################################
Sync up the Context File and Update Configuration on All Nodes
###################################################################################################
Login to all the application tier nodes and execute the following scripts to sync up the context file and configuration files for the run file system:
$ . ./EBSapps.env run
[applmgr@ebsprodapp4 ~]$ perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE
[applmgr@ebsprodapp4 ~]$adautocfg.sh
Execute AutoConfig on all application tier nodes
$ adadminsrvctl.sh start forcepatchfs ( on primary node, on patch FS )
Login to all the application tier nodes and execute the following scripts to sync up the context file and configuration files for the patch file system
$ . ./EBSapps.env patch
$ [applmgr@ebsprodapp4 ~]$ perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE
[applmgr@ebsprodapp4 ~]$ adautocfg.sh
Repeat above sync context file steps on node 2
###################################################################################################
Additional steps for parallel Standard CM dedicated for internal node2
Create similar Standard Manager2 for node2 with same given Specialization Rules
Parallel Concurrent Processing details
Primary : ebsprodapp4
Secondary : ebsprodapp3
Cache Size: 30
Work Shift: Standard
Process: 50
Sleep Seconds: 30
[applmgr@ebsprodapp4 ~]$ grep appldcp $CONTEXT_FILE (check on both internal nodes)
<APPLDCP oa_var="s_appldcp">ON</APPLDCP>
[applmgr@ebsprodapp4 ~]$
###################################################################################################
Enable all below port on Application server
1. s_nmport
2. s_ons_localport
3. s_ons_remoteport
4. s_ons_requestport
5. FNDFS port
[applmgr@ebsprodapp3 ~]$ grep -n 's_nmport\|ONS' $CONTEXT_FILE
[applmgr@ebsprodapp3 ~]$ grep FNDFS_$HOTNAME -A3 $TNS_ADMIN/tnsnames.ora
# firewall-cmd --permanent --remove-port={1626/tcp,5630/tcp,6100/tcp,6200/tcp,6500/tcp}
# firewall-cmd --reload
Increase WebLogic Memory for slow perfomance
[applmgr@ebsprodapp3 ~]$ grep nm_jvm_startup_properties $CONTEXT_FILE
-Xms2048m -Xmx3072m
###################################################################################################
Adop config for new node app4
###################################################################################################
[applmgr@ebsprodapp3 ~]$ adop -status if slave node is not added the execute fsclone
[applmgr@ebsprodapp3 ~]$ time adop -validate
free -h
echo 1 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a && printf '\n%s\n' ' Ram-cache and the swap get cleared'
export USER_MEM_ARGS="-Xms10240m -Xmx10240m -XX:MaxPermSize=2048m"
export CONFIG_JVM_ARGS="-Xmx4096m -Xms2048m -XX:MaxPermSize=512m -XX:-UseGCOverheadLimit"
export TIMEDPROCESS_TIMEOUT=-1
[applmgr@ebsprodapp3 ~]$ time adop phase=fs_clone
[applmgr@ebsprodapp4 ~]$ adop -status
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 31
Command: status
Output: /u01/NPROD/fs_ne/EBSapps/log/adop/31/20240822_175707/adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- -----------
ebsprodapp3 master FS_CLONE COMPLETED 2024/08/22 15:09:51 2024/08/22 17:35:49 2:14:58
ebsprodapp4 slave CONFIG_CLONE COMPLETED 2024/08/22 15:09:51 2024/08/22 17:45:21 2:15:30
CONFIG_CLONE COMPLETED 2024/08/22 15:09:51 2024/08/22 17:45:21 2:15:30
File System Synchronization Type: Full
adop exiting with status = 0 (Success)
[applmgr@ebsprodapp4~]$
###################################################################################################
Delete
node command new node app4
###################################################################################################
[applmgr@ebsprodapp4 ~]$ hostname -f
ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
[applmgr@ebsprodapp3 ~]$ perl
$AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-node
-contextfile=$CONTEXT_FILE -hostname=ebsprodapp4.prdappsn.ahccloudprdvcn1.oraclevcn.com
-logfile=del.log
Enter the APPS Schema password:
Enter the WebLogic AdminServer password:
Node deleted successfully.
[applmgr@ebsprodapp3 ~]$
###################################################################################################
Important sql Commands in EBS Multinode Architecture
###################################################################################################
col owner for a10;
col NODE_NAME for a25;
col HOW_MANY_USER_SESSIONS for a35;
select t.owner,t.last_analyzed,t.num_rows,t.blocks from dba_tables t where table_name like 'ICX_SESSIONS';
select min(first_connect), count(*) from icx.icx_sessions;
select a.node_name, 'Number of user sessions : ' || count(distinct b.session_id) How_many_user_sessions
from apps.fnd_nodes a,apps.icx_sessions b
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'),
NULL,limit_time, 0,limit_time,apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate
and a.node_id=b.node_id and counter < limit_connects group by a.node_name;
MIN(FIRST_CONNECT) COUNT(*)
-------------------- ----------
03-Jul-2024 11:23:30 11909
NODE_NAME HOW_MANY_USER_SESSIONS
------------------------- -----------------------------------
EBSPRODAPP3 Number of user sessions : 46
EBSPRODAPP4 Number of user sessions : 20
EBSPRODPUBAPP1 Number of user sessions : 11
