link:http://www.eygle.com/case/sga2.htm案例描述:這是一個(gè)大型生產(chǎn)系統(tǒng)問題出現(xiàn)時(shí)系統(tǒng)累計(jì)大量用戶進(jìn)程用戶請(qǐng)求得不到及時(shí)響應(yīng)，新的進(jìn)程不斷嘗試建立連接 連接數(shù)很快被用完數(shù)據(jù)庫版本:9.2.0.3操作系統(tǒng):Solaris81.檢查alert文件日志中記錄如下錯(cuò)誤信息，說明磁盤異步IO出現(xiàn)問題:WARNING: aiowait timed out 2 timesTue Aug 26 15:33:32 2003WARNING: aiowait timed out 2 timesTue Aug 26 15:33:34 2003WARNING: aiowait timed out 2 timesTue Aug 26 15:33:36 2003WARNING: aiowait timed out 2 timesTue Aug 26 15:33:38 2003WARNING: aiowait timed out 2 timesTue Aug 26 15:33:43 2003WARNING: aiowait timed out 1 timesTue Aug 26 15:33:46 2003WARNING: aiowait timed out 1 timesTue Aug 26 15:33:49 2003WARNING: aiowait timed out 1 timesTue Aug 26 15:33:51 2003WARNING: aiowait timed out 1 timesTue Aug 26 15:33:52 2003WARNING: aiowait timed out 1 timesTue Aug 26 15:33:53 2003WARNING: aiowait timed out 1 times.............我們知道在SUN的某些版本上異步IO存在問題而異步IO缺省是打開的SQL> show parameter disk_aNAME TYPE VALUE------------------------------------ ----------- ------------------------------disk_asynch_io boolean TRUE針對(duì)此問題，我們停用了數(shù)據(jù)庫的異步IO寫入。2.共享內(nèi)存問題alert文件中還記錄了以下錯(cuò)誤信息: Tue Aug 26 21:37:40 2003 WARNING: EINVAL creating segment of size 0x0000000190400000 fix shm parameters in /etc/system or equivalent該信息說明內(nèi)核參數(shù)設(shè)置過小或者和SGA不匹配我們檢查system配置文件$ cat /etc/system.......................set shmsys:shminfo_shmmax=4096000000 set shmsys:shminfo_shmmin=1set shmsys:shminfo_shmmni=200set shmsys:shminfo_shmseg=200set semsys:seminfo_semmap=1024set semsys:seminfo_semmni=2048set semsys:seminfo_semmns=2048set semsys:seminfo_semmnu=2048set semsys:seminfo_semume=200set semsys:seminfo_semmsl=2048我們發(fā)現(xiàn)最大共享內(nèi)存設(shè)置僅有4G3.檢查SGA設(shè)置SQL*Plus: Release 9.2.0.3.0 - ProdUCtion on 星期二 8月 26 21:46:35 2003Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.Connected to:Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit ProductionWith the Partitioning, OLAP and Oracle Data Mining optionsJServer Release 9.2.0.3.0 - ProductionSQL> show sgaTotal System Global Area 6695660272 bytesFixed Size 740080 bytesVariable Size 2399141888 bytesDatabase Buffers 4294967296 bytesRedo Buffers 811008 bytes我們發(fā)現(xiàn)SGA設(shè)置接近7G，這也就是步驟2中錯(cuò)誤提示出現(xiàn)的原因4.交換區(qū)問題我們用top工具檢查系統(tǒng)運(yùn)行狀況# /usr/local/bin/toplast pid: 16899; load averages: 0.82, 0.81, 0.83 21:49:051230 processes:1228 sleeping, 1 running, 1 on cpuCPU states: 50.1% idle, 7.4% user, 8.6% kernel, 33.9% iowait, 0.0% swapMemory: 8192M real, 118M free, 12G swap in use, 11G swap freePID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND15751 oracle 11 44 0 6456M 6408M sleep 0:02 0.49% oracle15725 oracle 11 58 0 6458M 6410M sleep 0:02 0.46% oracle251 root 12 48 0 7096K 1944K sleep 126:00 0.45% picld16540 oracle 11 58 0 6458M 6411M sleep 0:01 0.45% oracle16766 root 1 43 0 3744K 2248K cpu/1 0:01 0.41% top16408 oracle 11 58 0 6457M 6410M sleep 0:01 0.34% oracle15989 oracle 11 58 0 6458M 6409M sleep 0:01 0.34% oracle15919 oracle 11 58 0 6457M 6409M sleep 0:02 0.30% oracle16404 oracle 11 58 0 6457M 6409M sleep 0:00 0.28% oracle16327 oracle 11 55 0 6457M 6410M sleep 0:00 0.27% oracle14870 oracle 11 58 0 6457M 6412M sleep 0:05 0.24% oracle16851 oracle 11 35 0 6457M 6411M sleep 0:00 0.22% oracle16467 oracle 11 58 0 6457M 6409M sleep 0:00 0.21% oracle16163 oracle 11 58 0 6457M 6408M sleep 0:03 0.21% oracle15159 oracle 11 58 0 6457M 6408M sleep 0:05 0.21% oracleMemory: 8192M real, 118M free, 12G swap in use, 11G swap free我們發(fā)現(xiàn)系統(tǒng)僅有8G RAM,物理內(nèi)存僅有118M可用現(xiàn)在SWAP區(qū)使用了12G我們初步作出以下判定:SGA設(shè)置過大(將近7G)導(dǎo)致運(yùn)行時(shí)產(chǎn)生大量交換大量SWAP交換進(jìn)而引發(fā)磁盤問題這也就應(yīng)該是我們第一步看到WARNING: aiowait timed out 1 times的原因大量交換導(dǎo)致數(shù)據(jù)庫性能急劇下降進(jìn)而導(dǎo)致用戶請(qǐng)求得不到快速響應(yīng)，堵塞、累積，直至數(shù)據(jù)庫失去響應(yīng)5.解決方案此問題主要是由于SGA設(shè)置不當(dāng)引起，我們馬上縮小了SGA設(shè)置:SQL> show sgaTotal System Global Area 3591870848 bytesFixed Size 735616 bytesVariable Size 1442840576 bytesDatabase Buffers 2147483648 bytesRedo Buffers 811008 bytes此時(shí)，數(shù)據(jù)庫減少了交換,達(dá)到了穩(wěn)定運(yùn)行,用戶請(qǐng)求可以得到快速響應(yīng)。 問題解決完成.6.系統(tǒng)狀態(tài)調(diào)整后系統(tǒng)運(yùn)行狀況:$ top last pid: 12745; load averages: 0.46, 0.79, 0.65 22:22:49228 processes: 227 sleeping, 1 on cpuCPU states: 92.3% idle, 5.0% user, 1.6% kernel, 1.1% iowait, 0.0% swapMemory: 8192M real, 3817M free, 4015M swap in use, 15G swap free PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 12610 oracle 1 51 0 3511M 22M sleep 0:04 1.96% oracle 12595 oracle 1 48 0 3511M 22M sleep 0:03 0.92% oracle 12630 oracle 1 38 0 3511M 21M sleep 0:01 0.84% oracle 12614 oracle 1 46 0 3511M 22M sleep 0:01 0.64% oracle 12620 oracle 1 58 0 3511M 22M sleep 0:01 0.53% oracle 12709 oracle 1 48 0 3511M 21M sleep 0:00 0.45% oracle 265 root 11 38 0 7032K 1920K sleep 3:16 0.42% picld 12729 oracle 1 0 0 3511M 20M sleep 0:00 0.26% oracle 12741 oracle 1 58 0 2768K 1760K cpu/3 0:00 0.19% top 12745 oracle 1 44 0 3506M 16M sleep 0:00 0.17% oracle 12711 oracle 1 48 0 3506M 16M sleep 0:00 0.11% oracle 12738 oracle 1 43 0 3506M 16M sleep 0:00 0.06% oracle 7606 oracle 1 45 0 17M 6928K sleep 0:07 0.05% tnslsnr 12721 oracle 1 34 0 3506M 16M sleep 0:00 0.05% oracle 12723 oracle 1 53 0 3506M 16M sleep 0:00 0.05% oracle 該系統(tǒng)調(diào)整完以后，一直穩(wěn)定運(yùn)行至今.一點(diǎn)總結(jié):這個(gè)案例和前面我提到的另外一個(gè)極其相似同樣都是SGA設(shè)置不當(dāng)引起的數(shù)據(jù)庫問題本身并不復(fù)雜這一類問題應(yīng)該在數(shù)據(jù)庫規(guī)劃和建設(shè)階段就避免掉.其時(shí)，該問題對(duì)我更像是個(gè)心理測試當(dāng)所有老板都站在你背后的時(shí)候，你能否冷靜快速的找到并解決問題.關(guān)于SUN上的aiowait timed out 有很多總情況及誘因我后面還有相應(yīng)的案例說明 .-Eygle