emctl start dbconsole 수행 시 OC4J Configuration issue 해결 방법
============================================================================

PURPOSE
-------

이 문서는 OEM 10g의 dbconsole repository 관련한 문제를 해결하는
방법에 관한 문서로서 emca 를 구동하여 repository를 재생성하고자 할 때
OC4J Configuration issue라는 에러가 발생하는 경우 해결 방법이다.


Problem Description
-------------------
이 문서는 다음과 같은 제품을 사용하는 경우에 해당한다.

Enterprise Manager for RDBMS(Database Control) - Version: 10.1.0.X.X
Enterprise Manager for RDBMS(Database Control) - Version: 10.2.0.X.X

증상
----
1) 운영 중에 OEM Repository가 깨져 있는 경우, web browser에서 접속하면 다음과
같은 에러가 발생한다.

Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://EXRDB1:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control
........................
..................................................................... failed.
------------------------------------------------------------------
Logs are generated in directory /oracle/product/10.2.0/EXRDB1_EXRORA/sysman/log


2) $ORACLE_HOME/hostname_sid/sysman/log 아래의 trace 화일을 보면 다음과 같다.

[ emoms.trc ]

2006-04-26 17:58:52,685 [OmsServiceDriver thread] WARN jdbc.ConnectionCache _getConnection.306 -
Io exception: The Network Adapter could not establish the connection
java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:111)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:145)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:254)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:386)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:493)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:164)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:767)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:324)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:114)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getNewPoolOrXAConnection(OracleConnectionCacheImpl.java:694)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getPooledConnection(OracleConnectionCacheImpl.java:540)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:346)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:324)
at oracle.sysman.util.jdbc.ConnectionCache._getConnection(ConnectionCache.java:296)
at oracle.sysman.util.jdbc.ConnectionCache.getUnwrappedConnection(ConnectionCache.java:423)
at oracle.sysman.emSDK.svc.conn.FGAConnectionCache.getFGAConnection(FGAConnectionCache.java:156)
at oracle.sysman.emSDK.svc.conn.ConnectionService.getPrivateConnection(ConnectionService.java:605)
at oracle.sysman.emSDK.svc.conn.ConnectionService.getRepositoryVersionAndMode(ConnectionService.java:368)
at oracle.sysman.emSDK.svc.conn.ConnectionService.verifyRepositoryEx(ConnectionService.java:395)
at oracle.sysman.emSDK.svc.conn.ConnectionService.verifyRepository(ConnectionService.java:450)
at oracle.sysman.emSDK.svc.conn.ConnectionService.blockUntilValid(ConnectionService.java:475)
at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.getId(OMSHeartbeatRecorder.java:162)
at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.preTask(OMSHeartbeatRecorder.java:386)
at oracle.sysman.emdrep.omsservice.OmsServiceDriver.run(OmsServiceDriver.java:147)

2006-04-26 17:59:31,717 [OmsServiceDriver thread] ERROR conn.ConnectionService verifyRepositoryEx.433 -
Invalid Connection Pool. ERROR = Io exception: The Network Adapter could not establish the connection
2006-04-26 18:04:25,473 [OmsServiceDriver thread] WARN jdbc.ConnectionCache _getConnection.303
- Got a fatal exeption when getting a connection; Error code = 17002; Cleaning up cache and retrying
2006-04-26 18:04:25,512 [OmsServiceDriver thread] WARN jdbc.ConnectionCache _getConnection.306
- Io exception: The Network Adapter could not establish the connection
java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:111)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:145)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:254)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:386)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:493)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:164)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:767)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:324)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:114)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getNewPoolOrXAConnection(OracleConnectionCacheImpl.java:694)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getPooledConnection(OracleConnectionCacheImpl.java:540)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:346)
at oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:324)
at oracle.sysman.util.jdbc.ConnectionCache._getConnection(ConnectionCache.java:296)
at oracle.sysman.util.jdbc.ConnectionCache.getUnwrappedConnection(ConnectionCache.java:423)
at oracle.sysman.emSDK.svc.conn.FGAConnectionCache.getFGAConnection(FGAConnectionCache.java:156)
at oracle.sysman.emSDK.svc.conn.ConnectionService.getPrivateConnection(ConnectionService.java:605)
at oracle.sysman.emSDK.svc.conn.ConnectionService.getRepositoryVersionAndMode(ConnectionService.java:368)
at oracle.sysman.emSDK.svc.conn.ConnectionService.verifyRepositoryEx(ConnectionService.java:395)
at oracle.sysman.emSDK.svc.conn.ConnectionService.verifyRepository(ConnectionService.java:450)
at oracle.sysman.emSDK.svc.conn.ConnectionService.blockUntilValid(ConnectionService.java:475)
at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.getId(OMSHeartbeatRecorder.java:162)
at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.preTask(OMSHeartbeatRecorder.java:386)
at oracle.sysman.emdrep.omsservice.OmsServiceDriver.run(OmsServiceDriver.java:147)
2006-04-26 18:05:03,686 [OmsServiceDriver thread] ERROR conn.ConnectionService verifyRepositoryEx.433 -
Invalid Connection Pool. ERROR = Io exception: The Network Adapter could not establish the connection


3) 이와 같은 에러 발생 이후, repository 재생성을 위해 다음 명령을 수행한다.

emca -config dbcontrol db

EXRDB1:/oracle/product/10.2.0/cfgtoollogs/emca/EXRORA$emca -config dbcontrol db

STARTED EMCA at Apr 27, 2006 11:08:14 AM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Database SID: EXRORA
Listener port number: 1521
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional): jang@korea.com
Outgoing Mail (SMTP) server for notifications (optional): 211.40.197.2
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /oracle/product/10.2.0

Database hostname ................ EXRDB1
Listener port number ................ 1521
Database SID ................ EXRORA
Email address for notifications ............... gsjang@exrkorea.com
Outgoing Mail (SMTP) server for notifications ............... 211.40.197.2

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: y
Apr 27, 2006 11:08:46 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at
/oracle/product/10.2.0/cfgtoollogs/emca/EXRORA/emca_2006-04-27_11-08-14-AM.log.
Apr 27, 2006 11:08:49 AM oracle.sysman.emcp.EMConfig perform
SEVERE: Failed to allocate port(s) in the specified range(s) for the following
process(es): JMS [5540-5559],RMI [5520-5539],Database Control [5500-5519],EM
Agent [3938] | [1830-1849]
Refer to the log file at
/oracle/product/10.2.0/cfgtoollogs/emca/EXRORA/emca_2006-04-27_11-08-14-AM.log
for more details.
Could not complete the configuration. Refer to the log file at
/oracle/product/10.2.0/cfgtoollogs/emca/EXRORA/emca_2006-04-27_11-08-14-AM.log
for more details.

EXRDB1:/oracle/product/10.2.0/network/admin$emctl start dbconsole
OC4J Configuration issue.
/oracle/product/10.2.0/oc4j/j2ee/OC4J_DBConsole_EXRDB1_EXRORA not found.


위 에러 중 주요한 에러 메시지는 다음과 같다.

SEVERE: Failed to allocate port(s) in the specified range(s) for the following
process(es): JMS [5540-5559],RMI [5520-5539],Database Control [5500-5519],EM
Agent [3938] | [1830-1849]

EXRDB1:/oracle/product/10.2.0/network/admin$emctl start dbconsole
OC4J Configuration issue.
/oracle/product/10.2.0/oc4j/j2ee/OC4J_DBConsole_EXRDB1_EXRORA not found.


4) emca_2006-04-27_11-08-14-AM.log 화일을 보면 다음과 같은 에러 로그를 볼 수 있다.

=================================================================================
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager setFlag
CONFIG: Flag '-config' set to true
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager setFlag
CONFIG: Flag 'UPDATE_EMAIL' set to true
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager setFlag
CONFIG: Flag 'dbcontrol' set to true
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager setFlag
CONFIG: Flag 'db' set to true
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager setParam
CONFIG: Setting param: ORACLE_HOME value: /oracle/product/10.2.0
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager getParam
CONFIG: No value was set for the parameter ORACLE_HOSTNAME.
Apr 27, 2006 11:08:14 AM oracle.sysman.emcp.ParamsManager setParam
CONFIG: Setting param: EM_HOME value: /oracle/product/10.2.0
Apr 27, 2006 11:08:18 AM oracle.sysman.emcp.ParamsManager setParam
CONFIG: Setting param: SID value: EXRORA
Apr 27, 2006 11:08:18 AM oracle.sysman.emcp.ParamsManager getParam
CONFIG: No value was set for the parameter ORACLE_HOSTNAME.
Apr 27, 2006 11:08:18 AM oracle.sysman.emcp.util.ClusterUtil getLocalNode
CONFIG: This is expected, as we might be calling it from a non-rac home
Apr 27, 2006 11:08:18 AM oracle.sysman.emcp.util.ClusterUtil getLocalNode
CONFIG: Error retrieving the local node for the cluster
Apr 27, 2006 11:08:18 AM oracle.sysman.emcp.util.ClusterUtil getLocalNode
CONFIG: PRKH-1010 : Unable to communicate with CRS services.
[OCR Error(Native: prsr_initCLSS:[21])]
...........................
CONFIG: Marking JMS port: 5540 from /oracle/product/10.2.0/oc4j/j2ee/OC4J_DBConsole_EXRDB1_EXRDB
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.PortManager markAllUsedPorts
CONFIG: Marking RMI port: 5520 from /oracle/product/10.2.0/oc4j/j2ee/OC4J_DBConsole_EXRDB1_EXRDB
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.PortManager markAllUsedPorts
CONFIG: Marking HTTP port: 1158 from /oracle/product/10.2.0/oc4j/j2ee/OC4J_DBConsole_EXRDB1_EXRDB
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.PortManager markAllUsedPorts
CONFIG: Marking Agent port: 3938 from /oracle/product/10.2.0/EXRDB1_EXRDB/sysman/config/emd.properties
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.PortManager getAgentPort
CONFIG: Cannot parse EMD_URL correctly. No port identified
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.PortManager markAllUsedPorts
CONFIG: Marking Agent port: null from /oracle/product/10.2.0/sysman/config/emd.properties
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.PortManager markAllUsedPorts
CONFIG: Cannnot parse EMD_URL correctly. No port identified
Apr 27, 2006 11:08:48 AM oracle.sysman.emcp.util.FileUtil getProperties
CONFIG: Error reading file /oracle/product/10.2.0/install/staticports.ini
Apr 27, 2006 11:08:49 AM oracle.sysman.emcp.EMConfig perform
SEVERE: Failed to allocate port(s) in the specified range(s) for the following process(es):
JMS [5540-5559],RMI [5520-5539],Database Control [5500-5519],EM Agent [3938] | [1830-1849]
Refer to the log file at /oracle/product/10.2.0/cfgtoollogs/emca/EXRORA/emca_2006-04-27_11-08-14-AM.log
for more details.
Apr 27, 2006 11:08:49 AM oracle.sysman.emcp.EMConfig perform
CONFIG: Stack Trace:
oracle.sysman.emcp.exception.EMConfigException: Failed to allocate port(s) in the specified range(s) for the following process(es):
JMS [5540-5559],RMI [5520-5539],Database Control [5500-5519],EM Agent [3938] | [1830-1849]
at oracle.sysman.emcp.EMDBPreConfig.checkPorts(EMDBPreConfig.java:2266)
at oracle.sysman.emcp.EMDBPreConfig.performConfiguration(EMDBPreConfig.java:670)
at oracle.sysman.emcp.EMDBPreConfig.invoke(EMDBPreConfig.java:233)
at oracle.sysman.emcp.EMDBPreConfig.invoke(EMDBPreConfig.java:160)
at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:141)
at oracle.sysman.emcp.EMConfigAssistant.invokeEMCA(EMConfigAssistant.java:479)
at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:1123)
at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:463)
at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:412)


Workaround
----------
none


Solution Description
--------------------
위에서 port 할당이 불가능했던 이유를 확인하기 위해 다른 process가 사용 중인지 확인한다.
또한 hostname resolution이 가능한지 확인한 후에 Repository 재생성을 시도한다.

해결 방법 :

1. ps -ef | grep java 명령을 사용하여 기존에 떠 있는 process가 있으면 kill한다.

예) /oracle/product/102/jdk/bin/IA64N/java -Xmx512M -XX:MaxPermSize=64m
-XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XdoCloseWi

2. ps -ef | grep emagent 명령을 사용하여 agent process를 내려 놓는다.

예) /oracle/product/102/bin/emagent


3. /etc/hosts file을 확인하여 physical ip address와 hostname이 바로 들어 있는지 확인한다.

4. 현재 사용하는 repository를 지우려면 다음 명령을 사용하여 deconfig한다.

- set ORACLE_HOME and ORACLE_SID
- go to %ORACLE_HOME%bin
- > emca -deconfig dbcontrol db (10g R2)
- > emca -x (10g R1)


5. 다음 Note들을 Metalink에서 참조한다.

Note.330976.1 How to Drop, Create and Recreate DBconsole in Database R2.
Note.330130.1 Overview of the EMCA commands in DBControl 10.2.
Note.300510.1 Starting dbconsole Generates Error 'OC4J Configuration issue'.


6. OEM repository를 재생성하려면 OEM 10g R2 에서 다음 명령을 사용한다.

emca -config dbcontrol db

또는

emca -config dbcontrol db -RMI_PORT 5539 -JMS_PORT 5559 -AGENT_PORT 1849
-DBCONSOLE_HTTP_PORT 1158

이와 같이 PORT 까지 자세히 기술할 수 있다.

7. Repository 생성이 성공했다면 dbconsole을 띄우고 web browser를 통해 접속해 본다.

UNIX> emctl start dbconsole

http://EXRDB1:1158/em/


Reference Documents
-------------------



Applies to:
Enterprise Manager for RDBMS - Version: 10.2.0.1
Information in this document applies to any platform.
Database version (emca) 10.2

Goal
=====
This article will show how to create, drop and recreate DBconsole in Database R2.

Solution
========
To create configuration files and repository run
emca -config dbcontrol db -repos create


To drop configuration files and repository run :
emca -deconfig dbcontrol db -repos drop

To drop and create configuration files and repository run:
emca -config dbcontrol db -repos recreate


If you have any problem you can check the log files under :
ORACLE_HOME/cfgtoollogs/emca/