Some Important Oracle database Memory management queries.
SGA usage by Oracle Instance:
SGA Stat:
Total SGA:
Using V$PROCESS to see individual process memory usage
Using V$PGASTAT for overall PGA statistics
Tips and recommendations on various Oracle products. Solving day to day challenges with easy operational tasks. Providing ideas on scripting, queries and oracle Patching. Find tricks for numerous ORA-error.
Some Important Oracle database Memory management queries.
SGA usage by Oracle Instance:
SGA Stat:
Total SGA:
Using V$PROCESS to see individual process memory usage
Using V$PGASTAT for overall PGA statistics
In a RAC configuration, this is pretty common to corrupt the OLR registries under grid. So sometime we need to restore it to make things work again.
CRS-1013: The OCR location in an ASM disk group is inaccessible.
WARNING: failed to online diskgroup resource ora.ASM.dg (unable to communicate with CRSD/OHASD)
ORA-15368: Oracle Cluster Registry operation failed with error 32.
Running SRDC might show the exact symptoms:
eg: Verifying OLR integrity ...FAILED
Cause: OLR is corrupted
Login as grid user
go to OLR Location:
eg
/opt/app/grid/crsdata/<hostname>/olr
<GI_HOME>/bin/ocrconfig -local -showbackup
This will show you the backups available for the OLR.
To restore:
First check the below process is running or not:
ps -ef| grep ohasd.bin
This should return no process, if ohasd.bin is still up and running, stop it on local node:
# <GI_HOME>/bin/crsctl stop crs -f <========= for GI Cluster
OR
# <GI_HOME>/bin/crsctl stop has <========= for GI Standalone
Once it's down, restore with the following command:
# <GI_HOME>/bin/ocrconfig -local -restore <olr-backup>
(Here consider the most recent available backup file name)
NOTE:
If the command fails, create a dummy OLR, set correct ownership and permission and retry the restoration command:
# cd <OLR location>
# touch <hostname>.olr
# chmod 600 <hostname>.olr
# chown <grid>:<oinstall> <hostname>.olr
Once it's restored, GI can be brought up:
# <GI_HOME>/bin/crsctl start crs <========= for GI Cluster
OR
$ <GI_HOME>/bin/crsctl start has <========= for GI Standalone, this must be done as grid user.
In 12.1 onward, if patches are applied after the OLR is backed up, and later the backup is restored, the patch level will be different and GI won't start with error:
CRS-6706: Oracle Clusterware Release patch level ('1196363452') does not match Software patch level ('600291166'). Oracle Clusterware cannot be started.
To fix the above problem, run as root user:
<GI_HOME>/crs/install/rootcrs.sh -prepatch
<GI_HOME>/crs/install/rootcrs.sh -postpatch
In my case, I was facing another issue. So I had to follow some additional steps. the reason behind is that, my OLR was last backed up 6 months ago, and after that I did CPU patching on the database. so the backup OLR file does not have the information of the new CPU patches and we need to update that information to the current OLR file. to do so,
As root user unlock the GI home:
cd $ORACLE_HOME
$ORACLE_HOME/crs/install/rootcrs.sh -unlock
Apply the local patch to OLR
$ORACLE_HOME/bin/clscfg -localpatch
Lock the GI home
$ORACLE_HOME/crs/install/rootcrs.sh -lock
Start the clusterware:
crsctl start crs -wait
crsctl status resource -t
That it, now the OLR is fixed finally.
Oracle Support: How to backup or restore OLR:(KB137305)
Oracle Support: CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') KB140963
There are certain times when we observer OEM is performing slow or even unable to startup the Admin Server or OMS Server. We can look into few log files below and understand what is exactly causing the issue. Most of the times I have observed Java Heap issue or memory issue.
Sometime it happens due to an old process holding up the session and consuming the memory, the first step is to kill all the processes and remove some of the lok and temp files from the system.
java.lang.OutOfMemoryError: Java heap space.
ps -ef | grep EMGC_ADMINSERVERps -ef | grep EMGC_OMS1ps -ef | grep javaps -ef | grep opmn$kill -9 <PID>
The below temp files can be deleted:
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/tmp/EMGC_ADMINSERVER.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/ldap/ldapfiles/EmbeddedLDAP.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles/EmbeddedLDAP.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/default/_WLS_EMGC_OMS1000000.DAT
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/default/_WLS_EMGC_ADMINSERVER000000.DAT
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/tmp/EMGC_ADMINSERVER.lok
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/*.lck, *.pid, *.state
$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/nodemanager/*.lck, *.pid, *.state
grep "JAVA Memory arguments" <OMS_INST>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
--> This will give the output of the current OMS heap memory.
emctl get property -name OMS_HEAP_MAX
--> This needs Sysman password
Increase it to the desired value, example 4G.
<OMS_HOME>/bin/emctl set property -name OMS_HEAP_MAX -value 4096M
Restart the OMS
emctl stop oms -all
emctl start oms
Hope this will fix the slowness and heap memory error.
Best wishes,
In few cases we need to monitor the progress of an export or import job in oracle. below are the steps can be followed to get the details.
get the session details:
SQL> select * from dba_datapump_jobs where state='EXECUTING';
attach the session:
from Command line:
expdp system/********** attach=SYS_EXPORT_FULL_01
Export > status
Reference:
How To Monitor The Progress Of Datapump Jobs (Doc ID 1471766.1)
Best Wishes!!
Check /etc/oraInst.loc
go to the inventory location:
cd ContentsXML
cat inventory.xml and check which one needs to be detached.
<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?>
<!-- Copyright (c) 1999, 2024, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. --><INVENTORY>
<VERSION_INFO>
<SAVED_WITH>13.9.4.0.0</SAVED_WITH>
<MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraDB19Home1" LOC="/u01/app/oracle/19.3.0/" TYPE="O" IDX="4"/>
<HOME NAME="agent13c1" LOC="/u01/app/em13c/agent_13.4.0.0.0" TYPE="O" IDX="3"/>
<HOME NAME="agent13c2" LOC="/u01/app/em13c/agent_13.5.0.0.0" TYPE="O" IDX="6"/>
</HOME_LIST>
<COMPOSITEHOME_LIST/>
set pagesize 0 feedback off verify off heading off echo off
SELECT decode( nvl( space_used, 0),0, 0, ceil ( ( space_used / space_limit) * 100) ) pct_used FROM v$recovery_file_dest;
In many times, we need to know what are the script is being called while setting up the environment variables, this is very handy when we are getting errors like some env variables are not setup.
If zsh is the login shell:
zsh -xl
Execute the below command to get from which file the environment is setup:
PS4='+$BASH_SOURCE> ' BASH_XTRACEFD=7 bash -xl 7>&2
Best Wishes!!
col START_TIME for a15
col END_TIME for a15
col TIME_TAKEN_DISPLAY for a10
col INPUT_BYTES_DISPLAY for a10
col OUTPUT_BYTES_DISPLAY for a10
col OUTPUT_BYTES_PER_SEC_DISPLAY for a10
col output_device_type for a10
SELECT to_char (start_time,'DD-MON-YY HH24:MI') START_TIME,to_char(end_time,'DD-MON-YY HH24:MI') END_TIME, time_taken_display, status,input_type, output_device_type,input_bytes_display, output_bytes_display,output_bytes_per_sec_display,COMPRESSION_RATIO COMPRESS_RATIO FROM v$rman_backup_job_details WHERE status like 'RUNNING%';
set pagesize 300
set linesize 200
set wrap off
select thread#, sequence#, applied,to_char(first_time,'mm/dd/yy hh24:mi:ss') first,to_char(next_time, 'mm/dd/yy hh24:mi:ss') next,to_char(completion_time, 'mm/dd/yy hh24:mi:ss') completion from v$archived_log where thread# = 1 order by first_time;
control_management_pack_access
SQL> show parameter control_management_pack_access
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_management_pack_access string NONE
SQL> ALTER SYSTEM SET control_management_pack_access="DIAGNOSTIC+TUNING";
System altered.
SQL> show parameter control_management_pack_access
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_management_pack_access string DIAGNOSTIC+TUNING
Note: Before doing that, please make sure you have the corresponding licenses from Oracle.
Best Wishes!!!
SQL> alter database recover managed standby database cancel;
Database altered.
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
DGMGRL> edit database <STBY_DB> set state='APPLY-OFF';
Succeeded.
DGMGRL> edit database <PRIM_DB> set state='LOG-TRANSPORT-OFF';
Succeeded.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; -- Start MRP
DGMGRL> edit database <STBY_DB> set state='APPLY-ON';
Succeeded.
DGMGRL> edit database <PRIM_DB> set state='LOG-TRANSPORT-ON';
Succeeded.
Ref: Steps to restart log transport process in data guard (Doc ID 2819878.1)
In few cases Oracle XDB becomes Invalid and if it needs to be validated again without reinstalling the below work around might work.
XDB.PATH_VIEW and XDB.XDB_PV_TRIG are in INVALID state
To validate XDB, upgrade table XDB.XDB$H_LINK, then recompile the invalid objects.
connect / as sysdba
alter table XDB.XDB$H_LINK upgrade;
alter view XDB.PATH_VIEW compile;
alter trigger XDB.XDB_PV_TRIG compile;
select comp_name,version,Version_Full,status from dba_registry where comp_id = 'XDB';
The XDB should be validated now. If this does not work then you need to reinstall the XDB and befor doing that please consult Oracle, as it might corrupt some dependent XML attributes in other tables.
You can follow XDB Reinstall to reinstall the XDB.
OR,
1. Export any custom tables that has XML type, using Data Pump:
2. Deinstall and reinstall XDB
spool xdb_reinstall.log
set echo on;
connect / as sysdba
shutdown immediate
startup
@?/rdbms/admin/prvtnoqm.plb
@?/rdbms/admin/catproc.sql
@?/rdbms/admin/utlrp.sql
set pagesize 1000
set linesize 150
col comp_name format a36
col version format a12
col status format a8
col owner format a12
col object_name format a35
col object_type format a20
col name format a35
select comp_name, version, status from dba_registry order by status, comp_name;
select owner, object_name, object_type, status
from dba_objects
where status = 'INVALID'
and owner in ('SYS', 'XDB')
order by owner, object_name;
select owner, name, type, text
from dba_errors
where owner in ('SYS', 'XDB')
order by owner, name;
spool off;
3. Drop the tables in step 1 and import them from the dump file.
Best Wishes!!
This is a generic error and there can be multiple reasons to it. to find the actual reason, you need to consider the alert log and additional infrastructure components as well.
In my case, while doing some maintenance, during DB shutdown, the server was crashed and it kept the database in inconsistency. The background processes got cleared but, the running transactions were still in progress. which was preventing the DB to be up. So after drilling down it was some unarchived logfiles causing this error. I did the following to fix the issue.
Note: Please do not perform this in PROD without consulting Oracle support, as this might cause some data loss for the uncommitted transactions.
sqlplus / as sysdba
startup nomount
alter database mount;
alter database clear unarchived logfile group 1;
alter database clear unarchived logfile group 2;
alter database clear unarchived logfile group 3;
Keep adding the logfile groups if you have more groups
shutdown immediate
startup
select name, open_mode from v$database;
Hope this helps.
Best Wishes!!
select OWNER_NAME,JOB_NAME,STATE from DBA_DATAPUMP_JOBS;
make a note of the job name, which is in executing state and then run the below command
$expdp system/********** attach=SYS_EXPORT_FULL_01
OR
$impdp system/********** attach=SYS_IMPORT_FULL_01
This will open the interactive session for the datapump utility and you can execute the below commands:
> status
> parallel=4 (whichever value you have determined to provide)
And now run status again and it will show that 4 workers have been assigned to the job.
Hope this helps. Best Wishes!!
In this article I will cover one of the important issue faced after sustainment patching. It is sometime observed after patching the XDB Component gets Invalid and recompiling does not fix it.
Few known errors like few XDB objects are Invalid under DBA_OBJECTS and in DBA_REGISTRY XDB is showing as Invalid.
As per Oracle, deinstall and reinstall it, but that will impact if there is any data that is using the XML. So, It is aways a good idea to reinstall it.
(this is applicable for Databases version greater than 10g)
In few cases I have seen issues with catqm.sql and not fixing few invalid objects, so if you are ok, you can always run catproc.sql. Please donot run this in prod without proper backup.
This will fix the XDB component and the status should be Valid now.
If reinstalling doesnot fix it, you can completely deinstall and install the component but doing so might have some impact. so it is recommended to reach out to Oracle SR for confirmation. I will be outlining the steps:
Deinstall/Uninstall XDB:
Pease follow the same procedure as earlier installation methods.
Hope this helps.
How to Reinstall XDB on 12c and above? (Doc ID 2403285.1)
Primary Note for Oracle XML Database (XDB) Install / Deinstall (Doc ID 1292089.1)
In this article, I will be converting character set of a 19c oracle database from WE8MSWIN1252 to AL32UTF8
Run the below query and keep record of the existing information.
For my example this is the output:
Now run the below query to see the datatype it is using:
Example output:
Take a full backup of the database. Either through RMAN or Expdp, depending on the db size and your database setup.
For RMAN: follow this link
For EXPDP : follow this link
Once the backup is done, check there are ample archive space available.
If you would like, you can create a restore point for easy revert back of the change.
Check the restore point is created or not.
Invalid object count schema wise:
If there are numerous invalid count, run utlrp.sql to recompile the invalid objects:
Now check for the character set again to validate the change is successful:
Example:
Invalid Count schema wise:
And it completes the characterset changes. for few cases it might not work, in that case you can use Oracle Data Migration Assistant for Unicode (DMU), to learn more about how to use DMU follow the below link.
In case you want to revert back the change, you can revert to the restore point you just created before making the changes.
Note: This is done in Development Instance, before running it in production, please make sure you validated all the data.
Best Wishes!!
First make a note of the existing timezone with the below query.
==================================================
$tzselect
$timedatectl set-timezone <new_time_zone>
==================================================
=============================================================
There are few ways to validate it within the database and at the OS level. The easiest way to check would be running a SQL query which is oracle built-in function and validate the status.
If DataGuard is not configured:
0
Data Guard usage not detected
If DataGuard is configured:
1
Number of standbys: 1, Number of Cascading databases: 0, Number of Terminal
databases: 1, Redo Apply used: TRUE, SQL Apply used: FALSE, Far Sync Instance
used: FALSE, Snapshot Standby used: FALSE, Broker used: TRUE, Protection mode:
MAXIMUM PERFORMANCE, Log transports used: LGWR ASYNC, Fast Sync used: FALSE,
Fast-Start Failover used: FALSE, Real-Time Apply used: TRUE, Compression used:
FALSE, Flashback used: FALSE, Recovery Appliance used: FALSE
DGMGRL> show configurationConfiguration - db1Protection Mode: MaxPerformanceMembers:db1 - Primary databasedb1_stby - Physical standby databaseFast-Start Failover: DisabledConfiguration Status:SUCCESS (status updated 37 seconds ago)
export OPATCH_DEBUG=true
export JAVA_VM_OPTION="$JAVA_VM_OPTION -Doracle.installer.logLevel=FINEST"
RMAN-03002: failure of backup command at 03/24/2025 13:01:37
RMAN-06059: expected archived log not found, loss of archived log compromises recoverability
ORA-19625: error identifying file <datafile>
ORA-17503: ksfdopn:2 Failed to open file <filename>
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
*** 2025-03-24T13:28:11.414105-06:00
*** MODULE NAME:(emagent_SQL_osm_instance) 2025-03-24T13:28:11.414215-06:00
*** ACTION NAME:(FileGroup_Usage) 2025-03-24T13:28:11.414234-06:00
ERROR: submiting READ on /dev/rdsk/<diskname> failed due to kgfknm error
crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
Fatal NI connect error 12547, connecting to:
<JDBC>
VERSION INFORMATION:
TNS for <OS>: Version 19.0.0.0.0 - Production
Oracle Bequeath NT Protocol Adapter for Solaris: Version 19.0.0.0.0 - Production
Version 19.25.0.0.0
Time: 24-MAR-2025 14:46:06
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
For all these error, below permission change will fix the issue.
login as grid user, and go to GRID_HOME/bin location
chmod 6751 oracle
ls -lrt oracle
-rwsr-s--x 1 grid oinstall 563794648 Mar 26 20:13 oracle
But this was is not applicable for our version.
Bug details: Bug 36461761 - ASM numerous trace files generated with 'error: submiting read on /dev/rdsk/nnn 'failed due to kgfknm error' messages (Doc ID 36461761.8)
Some Important Oracle database Memory management queries. SGA usage by Oracle Instance: select round(sum(bytes)/1024/1024,2)||' MB'...