Oracle Crash案例一则
接到电话报告说一套Oracle库在临近下班的时候莫名crash掉，让我去现场看看。
刚到现场，现场的维护人员说由于扩展了存储，在添加数据文件的时候数据库不知道什么原因自动停掉了，他尝试将数据库起来的时候收到oracle提示文件出错需要恢复。
坐下来看了下是Linux平台的地市库，版本是10204,现场维护人员想让我直接尝试将库拉起来，我觉得有必要先看下系统日志。
从日志看到如下信息：
Wed May 15 17:11:08 2013
Errors in file /oracle/app/oracle/admin/oradbhost/udump/oradbhost_ora_18178.trc:
ORA-27052: 无法刷新文件数据
Linux Error: 30: Read-only file system
Additional information: 1
ORA-19510: 无法设置 2019327 区块的大小给文件 "/data/oracle/oradata/oradbhost/TS091.dbf" (块大小 = 8192)
ORA-27041: 无法打开文件
Linux Error: 30: Read-only file system
Additional information: 4
ORA-19502: 文件 "/data/oracle/oradata/oradbhost/TS091.dbf", 块编号 2019072 写错误 (块大小 = 8192)
ORA-27072: 文件 I/O 错误
Linux Error: 9: Bad file descriptor
Additional information: 4
Additional information: 2019072
Additional information: 192512
Wed May 15 17:11:08 2013
Errors in file /oracle/app/oracle/admin/oradbhost/udump/oradbhost_ora_18178.trc:
ORA-27056: 无法删除文件
Linux Error: 30: Read-only file system
ORA-19510: 无法设置 2019327 区块的大小给文件 "/data/oracle/oradata/oradbhost/TS091.dbf" (块大小 = 8192)
ORA-27041: 无法打开文件
Linux Error: 30: Read-only file system
Additional information: 4
ORA-19502: 文件 "/data/oracle/oradata/oradbhost/TS091.dbf", 块编号 2019072 写错误 (块大小 = 8192)
ORA-27072: 文件 I/O 错误
Linux Error: 9: Bad file descriptor
Additional information: 4
Additional information: 2019072
Additional information: 192512
Wed May 15 17:11:08 2013
ORA-19510 signalled during: ALTER TABLESPACE "TS"
ADD
DATAFILE '/data/oracle/oradata/oradbhost/TS091.dbf' SIZE
20480M --20130515
...
Wed May 15 17:11:17 2013
KCF: write/open error block=0x72933 online=1
file=95 /data/oracle/oradata/oradbhost/TS076.dbf
error=27072 txt: 'Linux Error: 30: Read-only file system
Additional information: 4
Additional information: 469299
Additional information: -1'
Wed May 15 17:11:17 2013
Errors in file /oracle/app/oracle/admin/oradbhost/bdump/oradbhost_dbw1_8355.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 95 (block # 469299)
ORA-01110: data file 95: '/data/oracle/oradata/oradbhost/TS076.dbf'
ORA-27072: File I/O error
Linux Error: 30: Read-only file system
Additional information: 4
Additional information: 469299
Additional information: -1
DBW1: terminating instance due to error 1242
Wed May 15 17:11:17 2013
Errors in file /oracle/app/oracle/admin/oradbhost/bdump/oradbhost_dbw0_8353.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
Instance terminated by DBW1, pid = 8355
在5点11分的时候，维护人员在添加数据文件的时候数据库后台就出现问题了，随后实例中断；而后在23分的时候维护人员尝试将实例启动，但是收到报错，后台日志显示数据库无法完成对数据文件的一致性验证。
这之后现场维护人员还在尝试中，无谓的尝试啊，一直折腾到5点53分，维护人员这才想起来要打电话给我们，浪费了将近半个小时的时间！~。
6点到场，到6点5分基本就诊断了问题的根本原因了，存储出现问题，无法读写：
[root@localhost oradbhost]# uname -a
Linux localhost.localdomain 2.6.18-128.el5PAE #1 SMP Wed Dec 17 12:02:33 EST 2008 i686 i686 i386 GNU/Linux
[root@localhost oradbhost]# df -h
文件系统 容量 已用 可用 已用% 挂载点
/dev/mapper/VolGroup00-LogVol00
1.1T 979G 43G 96% /
/dev/cciss/c0d0p1 99M 12M 83M 13% /boot
tmpfs 8.0G 0 8.0G 0% /dev/shm
/dev/mapper/mpath0p1 3.6T 316G 3.1T 10% /data
/dev/mapper/mpath0p2 3.6T 197M 3.4T 1% /data1
/dev/mapper/mpath0p3 3.6T 197M 3.4T 1% /data2
/dev/mapper/mpath0p4 3.6T 95G 3.3T 3% /data3
[root@localhost oradbhost]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/cciss/c0d0p1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/mapper/mpath0p1 on /data type ext3 (rw)
/dev/mapper/mpath0p2 on /data1 type ext3 (rw)
/dev/mapper/mpath0p3 on /data2 type ext3 (rw)
/dev/mapper/mpath0p4 on /data3 type ext3 (rw)
[root@localhost oradbhost]# touch /data/1
touch: 无法触碰 “/data/1”: 只读文件系统
[root@localhost oradbhost]# touch /data1/1
[root@localhost oradbhost]# touch /data2/1
[root@localhost oradbhost]# touch /data3/1
而出现问题的文件系统正是数据文件所在的位置，至此，将日志文件打包手工，明日再写诊断报告。
-EOF-
