RAC 实例不能启动 手工启动实例报错 ORA-03113 end-of-file on communication channel

前几天弄好的一个Oracle 10gR2的 RAC. 刚才启动的时候实例无法启动，开始以为是反应慢，就等了一会。 过去了N长时间还没有启动。 觉得不太对劲。

[root@rac1 bin]# ./crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.orcl.db application ONLINE OFFLINE

ora....oltp.cs application ONLINE OFFLINE

ora....cl1.srv application ONLINE OFFLINE

ora....cl2.srv application ONLINE OFFLINE

ora....l1.inst application ONLINE OFFLINE

ora....l2.inst application ONLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

手动用srvctl 命令启动也报错：

[root@rac1 bin]# ./srvctl start instance -d orcl -i orcl1

PRKP-1001 : Error starting instance orcl1 on node rac1

CRS-0215: Could not start resource 'ora.orcl.orcl1.inst'.

因为下午在这个实例上重建了OCR和Voting disk。 所以开始以为是这个过程导致的。 在那用ocrdump命令将OCR dump出来，然后查看。 相关resource的属性没有问题。

想起来以前看到的一个错误，srvctl 不能启动， 但是sqlplus能启动。 参考：

ORA-16032 Can not Start Instance via srvctl but via sqlplus is fine [ID 1062071.1]

http://blog.csdn.net/tianlesoftware/archive/2010/11/19/6020432.aspx

就尝试用sqlplus 启动了：

[oracle@rac1 dbs]$ export ORACLE_SID=orcl1

[oracle@rac1 dbs]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Dec 2 19:18:22 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area 222298112 bytes

Fixed Size 1218628 bytes

Variable Size 125831100 bytes

Database Buffers 92274688 bytes

Redo Buffers 2973696 bytes

Database mounted.

ORA-03113: end-of-file on communication channel

mount 之后直接挂掉，郁闷。 所以去看了一下alert log：

Errors in file /u01/app/oracle/admin/orcl/udump/orcl1_ora_14889.trc:

ORA-16038: log 1 sequence# 2 cannot be archived

ORA-00254: error in archive control string ''

ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.261.736599263'

ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.257.736599275'

ORA-15173: entry 'archivelog' does not exist in directory '/'

USER: terminating instance due to error 16038

看到以上信息，小兴奋了一下。 因为找到原因了。 将数据库启动到mount 状态。 确认下归档目录：

SQL> startup mount;

ORACLE instance started.

Total System Global Area 222298112 bytes

Fixed Size 1218628 bytes

Variable Size 96470972 bytes

Database Buffers 121634816 bytes

Redo Buffers 2973696 bytes

Database mounted.

SQL> show parameter log_

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_1 string LOCATION=+FRA/archivelog/orcl/

...

将RAC的归档放在了ASM上。 所以到ASM 上看了一下，确实没有后面的几个目录。 用asmcmd连上ASM. 在DATA和FRA目录下分别才创建了archivelog/orcl 这个目录：

[oracle@rac1 bdump]$ asmcmd

ASMCMD> ls

DATA/

FRA/

ASMCMD> cd DATA/

ASMCMD> ls

ORCL/

ASMCMD> mkdir archivelog

ASMCMD> cd archivelog

ASMCMD> ls

ASMCMD> mkdir orcl

ASMCMD>cd ../..

ASMCMD> cd FRA

ASMCMD> ls

ORCL/

ASMCMD> mkdir archivelog

ASMCMD> ls

ORCL/

archivelog/

ASMCMD> cd archivelog

ASMCMD> ls

ASMCMD> mkdir orcl

在将mount的实例open:

SQL> alter database open;

Database altered.

成功open. 用srvctl 启动了另一个实例：

[oracle@rac1 bin]$ ./srvctl start instance -d orcl -i orcl1

查看进程：

[oracle@rac1 bin]$ ./crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.orcl.db application ONLINE ONLINE rac1

ora....oltp.cs application ONLINE ONLINE rac1

ora....cl1.srv application ONLINE ONLINE rac1

ora....cl2.srv application ONLINE ONLINE rac2

ora....l1.inst application ONLINE ONLINE rac1

ora....l2.inst application ONLINE ONLINE rac2

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

成功启动。

小结：

官网的那个是因为使用了环境变量导致的。 我的这个是因为归档目录不存在导致的。 归档设置虽然是个小问题，影响确是很大的。

------------------------------------------------------------------------------

Blog： http://blog.csdn.net/tianlesoftware

网上资源： http://tianlesoftware.download.csdn.net

相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx

DBA1 群：62697716(满); DBA2 群：62697977(满)

DBA3 群：62697850 DBA 超级群：63306533;

聊天 群：40132017

--加群需要在备注说明Oracle表空间和数据文件的关系，否则拒绝申请