Configure Oracle Data Guard Broker with DGMGRL Fast-Start Failover

Configure Oracle Data Guard broker with DGMGRL for fast-start failover. Master high availability & disaster recovery in your Oracle database.

Overview: Elevating Oracle Data Guard with DGMGRL and Fast-Start Failover for Peoplesoft Environments

In the realm of enterprise applications, few systems are as critical to an organization's operations as Oracle Peoplesoft. Ensuring the continuous availability and rapid recoverability of these mission-critical databases is paramount. Oracle Data Guard stands as the cornerstone technology for achieving high availability (HA) and disaster recovery (DR) for Oracle databases. It maintains one or more synchronized copies of a production database, known as standby databases, ensuring data protection and business continuity.

While Data Guard provides the underlying mechanism for replication, managing a Data Guard configuration manually can be complex and error-prone, especially during a disaster scenario. This is where the Oracle Data Guard Broker (DGMGRL) comes into play. DGMGRL is a command-line interface and API that simplifies the creation, maintenance, and monitoring of Data Guard configurations. It abstracts away much of the manual parameter management and offers a robust framework for automating administrative tasks.

A key feature enhanced by DGMGRL is Fast-Start Failover (FSFO). FSFO enables automatic, rapid failover to a designated standby database without requiring manual intervention, significantly reducing Recovery Time Objective (RTO) in the event of a primary database outage. This capability is particularly vital for Peoplesoft environments, where even minutes of downtime can translate into substantial financial losses and operational disruption. By leveraging DGMGRL with FSFO, organizations can achieve a highly resilient database infrastructure, allowing Peoplesoft applications to seamlessly transition to a standby database with minimal impact on end-users and business processes.

This article will guide you through the detailed process of configuring the Oracle Data Guard Broker with Fast-Start Failover, demonstrating real-world commands and best practices to ensure a robust and automated high-availability solution for your critical Oracle Peoplesoft databases.

Prerequisites: Laying the Foundation for a Robust Data Guard Broker Configuration

Before diving into the DGMGRL configuration and Fast-Start Failover setup, it's essential to ensure that your environment meets several critical prerequisites. A properly configured foundation is key to a stable and reliable Data Guard setup. We assume an Oracle Database 19c Enterprise Edition environment for this guide.

Hardware and Operating System
- Two separate servers: one for the primary database (e.g., host-prod.example.com) and one for the physical standby database (e.g., host-stby.example.com). These should ideally be in different data centers or availability zones for true disaster recovery.
- Identical operating system versions and patches (e.g., Oracle Linux 8).
- Sufficient disk space, CPU, and memory on both servers to handle database workloads.
- Reliable network connectivity between the primary and standby servers, with appropriate firewall rules allowing Oracle Net Services traffic (default port 1521).
Oracle Software and Database Configuration
- **Identical Oracle Database Software:** Both primary and standby servers must have the exact same Oracle Database software version, release, and patch level (e.g., Oracle Database 19.18.0.0.230718).
- **Enterprise Edition:** Data Guard is an Enterprise Edition feature.
- **Database in ARCHIVELOG Mode:** The primary database must be operating in ARCHIVELOG mode.
```
SQL> SELECT LOG_MODE FROM V$DATABASE;
LOG_MODE
------------
ARCHIVELOG
                
```
- **FORCE LOGGING Enabled:** This ensures all database changes are logged and thus protected by Data Guard.
```
SQL> SELECT FORCE_LOGGING FROM V$DATABASE;
FOR
---
YES
                
```
- **STANDBY_FILE_MANAGEMENT=AUTO:** This parameter on the standby database ensures that new datafiles added to the primary database are automatically created on the standby.
```
SQL> SHOW PARAMETER STANDBY_FILE_MANAGEMENT;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_file_management              string      AUTO
                
```
- **Unique Database Names:** Each database in the Data Guard configuration must have a unique DB_UNIQUE_NAME.
  - Primary: psftprod_db
  - Standby: psftstby_db
- **Data Guard Broker Parameter:** The DG_BROKER_START initialization parameter must be set to TRUE in the init.ora or spfile for both databases.
```
SQL> SHOW PARAMETER DG_BROKER_START;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_start                      boolean     TRUE
                
```
- **Listener Configuration:** A listener must be running on both primary and standby servers, configured to listen for the respective database instances. Ensure the LOCAL_LISTENER and REMOTE_LISTENER parameters are correctly set in the init.ora/spfile.
- **TNS Names Entries:** The tnsnames.ora file on both servers must contain entries for both the primary and standby databases, allowing them to communicate via Oracle Net Services.
```
# On host-prod.example.com and host-stby.example.com

PSFTPROD_SVC =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = host-prod.example.com)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = psftprod_svc)
    )
  )

PSFTSTBY_SVC =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = host-stby.example.com)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = psftstby_svc)
    )
  )
                
```
Existing Physical Standby Database

This guide assumes you already have a functional physical standby database configured and synchronizing with the primary. The process of creating a physical standby database (e.g., using RMAN DUPLICATE or backup/restore) is outside the scope of this article but is a fundamental prerequisite. The standby database should be in Mount mode and actively applying redo logs.
```
-- On standby database
SQL> SELECT OPEN_MODE, DATABASE_ROLE, SWITCHOVER_STATUS FROM V$DATABASE;
OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS
-------------------- ---------------- --------------------
MOUNTED              PHYSICAL STANDBY NOT ALLOWED
        
```

With these prerequisites met, your environment is ready for the Data Guard Broker configuration.

Step-by-Step Implementation: Configuring DGMGRL and Fast-Start Failover

Now, let's proceed with the detailed steps to configure the Data Guard Broker and enable Fast-Start Failover. We'll use the following naming conventions:

**Primary Database:** psftprod (DB_UNIQUE_NAME: psftprod_db), Host: host-prod.example.com
**Standby Database:** psftstby (DB_UNIQUE_NAME: psftstby_db), Host: host-stby.example.com
**Observer Host:** host-obs.example.com (a third, independent server)
**Service Names in TNS:** psftprod_svc, psftstby_svc

1. Start DGMGRL and Create the Data Guard Configuration

First, connect to the primary database server and start the DGMGRL command-line interface. Ensure your Oracle environment variables (ORACLE_HOME, ORACLE_SID) are set correctly for the primary database.


[oracle@host-prod ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jan 15 10:00:00 2024
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> CONNECT sys/oracle@psftprod_svc
Connected as SYSDBA.

Now, create the Data Guard configuration, giving it a name (e.g., PSFT_DG_CONFIG). Then, add the primary database to this configuration.


DGMGRL> CREATE CONFIGURATION PSFT_DG_CONFIG AS
> PRIMARY DATABASE IS psftprod_db
> CONNECT IDENTIFIER IS psftprod_svc;
Configuration "PSFT_DG_CONFIG" created with primary database "psftprod_db".

Next, add the physical standby database to the configuration. DGMGRL will automatically discover the standby database and validate its configuration against the primary.


DGMGRL> ADD DATABASE psftstby_db AS
> CONNECT IDENTIFIER IS psftstby_svc
> MAINTAINED AS PHYSICAL;
Database "psftstby_db" added.

At this point, the configuration is created but not yet enabled. You can check its status:


DGMGRL> SHOW CONFIGURATION;

Configuration - PSFT_DG_CONFIG

  Protection Mode: MaxPerformance
  Members:
  psftprod_db - Primary database
    Error: ORA-16607: Data Guard broker is not able to start the DR process
           DGMGRL cannot communicate with the database "psftprod_db".
           DGMGRL cannot communicate with the database "psftstby_db".

  Fast-Start Failover:  Disabled

Configuration status:
ERROR

The error is expected because the configuration is not yet enabled. Let's enable it:


DGMGRL> ENABLE CONFIGURATION PSFT_DG_CONFIG;
Enabled.

After enabling, check the configuration status again. It should now show a healthy state.


DGMGRL> SHOW CONFIGURATION;

Configuration - PSFT_DG_CONFIG

  Protection Mode: MaxPerformance
  Members:
  psftprod_db - Primary database
    psftstby_db - Physical standby database

  Fast-Start Failover:  Disabled

Configuration status:
SUCCESS

You can also check the detailed status of individual databases:


DGMGRL> SHOW DATABASE psftprod_db VSTATUS;
DGMGRL> SHOW DATABASE psftstby_db VSTATUS;

These commands provide detailed information about the database's role, status, and any pending issues.

2. Configure Fast-Start Failover (FSFO)

Fast-Start Failover is the core automation feature we want to enable. It requires setting a protection mode and configuring the FSFO target.

Set Protection Mode

For most Peoplesoft environments, Maximum Performance mode is suitable, balancing data protection with primary database performance. If zero data loss is an absolute requirement, consider Maximum Availability, but be aware of its impact on primary database commit times.


DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXPERFORMANCE;
Succeeded.

Configure FastStartFailoverTarget

This parameter specifies which standby database should become the new primary in case of a failover. In a simple two-database setup, it's the only standby.


DGMGRL> EDIT DATABASE psftprod_db SET PROPERTY 'FastStartFailoverTarget'='psftstby_db';
Property "FastStartFailoverTarget" updated.

Enable Fast-Start Failover

Now, enable FSFO. This command will verify that all conditions for FSFO are met before activation.


DGMGRL> ENABLE FAST_START FAILOVER;
Enabled.

Configure the FSFO Observer

The Observer is a crucial component of FSFO. It's a separate DGMGRL process running on a third, independent server (host-obs.example.com in our example). The Observer continuously monitors the primary and standby databases. If the primary becomes unavailable, the Observer initiates the failover to the designated standby. Without an Observer, FSFO cannot function.

On the observer host (host-obs.example.com), ensure Oracle Client is installed and tnsnames.ora has entries for both primary and standby databases. Set ORACLE_HOME and PATH appropriately, but you don't need to set ORACLE_SID.


[oracle@host-obs ~]$ dgmgrl

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jan 15 10:30:00 2024
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> CONNECT sys/oracle@psftprod_svc
Connected as SYSDBA.

DGMGRL> START OBSERVER;
Observer started.

The Observer will now run in the background. It's good practice to run the Observer in a `nohup` session or as a systemd service to ensure it persists across reboots.


[oracle@host-obs ~]$ nohup dgmgrl sys/oracle@psftprod_svc "START OBSERVER;" &

You can check the FSFO status from any DGMGRL session connected to the primary:


DGMGRL> SHOW FAST_START FAILOVER;

Fast-Start Failover: Enabled
  Threshold:        30 seconds
  Target:           psftstby_db
  Observer:         host-obs.example.com
  Lag Limit:        30 seconds (not in use)
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE

  Configurations:
    psftprod_db - Primary database
    psftstby_db - Physical standby database

Current status for "PSFT_DG_CONFIG":
  Primary database is "psftprod_db"
  Physical standby database "psftstby_db" is performing as expected.
  Fast-Start Failover is enabled and active.

3. Testing Fast-Start Failover

Thorough testing is crucial to validate your FSFO configuration. We'll simulate a primary database failure.

**Simulate Primary Failure:** On the primary host (host-prod.example.com), abruptly shut down the database instance.


[oracle@host-prod ~]$ sqlplus / as sysdba
SQL> SHUTDOWN ABORT;
ORACLE instance shut down.

**Observe Automatic Failover:** The Observer, detecting the primary database's unavailability, will initiate a failover. This process usually takes a few seconds, plus the configured FSFO threshold (default 30 seconds).


-- On observer host (host-obs.example.com) - you might see output like:
ORA-16659: observer is unable to connect to database "psftprod_db"
Initiating Fast-Start Failover to database "psftstby_db"...
Failover succeeded. New primary is "psftstby_db".

**Verify New Primary:** Connect to DGMGRL (from any host, connecting to the new primary's service name) and verify the configuration.


[oracle@anyhost ~]$ dgmgrl
DGMGRL> CONNECT sys/oracle@psftstby_svc
Connected as SYSDBA.

DGMGRL> SHOW CONFIGURATION;

Configuration - PSFT_DG_CONFIG

  Protection Mode: MaxPerformance
  Members:
  psftstby_db - Primary database
    psftprod_db - Physical standby database (disabled)

  Fast-Start Failover: Enabled

Configuration status:
SUCCESS (Fast-Start Failover is enabled)

Notice that psftstby_db is now the primary, and the old primary (psftprod_db) is disabled and marked as a standby. It's disabled because it needs to be reinstated before it can function as a standby again.

**Reinstate Old Primary (Optional but Recommended):** To bring the old primary back into the Data Guard configuration as a standby, you need to reinstate it. First, ensure the old primary database instance is started in MOUNT mode.


-- On host-prod.example.com
[oracle@host-prod ~]$ sqlplus / as sysdba
SQL> STARTUP MOUNT;
ORACLE instance started.
Database mounted.

Then, from DGMGRL connected to the new primary:


DGMGRL> REINSTATE DATABASE psftprod_db;
Reinstating database "psftprod_db", please wait...
Operation requires shutdown of instance "psftprod" on database "psftprod_db".
Shutting down instance "psftprod"...
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Database "psftprod_db" reinstated.

The DGMGRL reinstate command will automatically shut down, re-create the control file, and start the old primary as a physical standby. It might show ORA-01034 if the instance was already down, which is normal. After reinstatement, verify the configuration again:


DGMGRL> SHOW CONFIGURATION;

Configuration - PSFT_DG_CONFIG

  Protection Mode: MaxPerformance
  Members:
  psftstby_db - Primary database
    psftprod_db - Physical standby database

  Fast-Start Failover: Enabled

Configuration status:
SUCCESS (Fast-Start Failover is enabled)

**Re-enable FSFO Observer:** If the Observer was stopped during the failover or restart, ensure it's running on the observer host.

4. Other Useful DGMGRL Commands

While FSFO automates failover, DGMGRL offers other commands for manual intervention and configuration management:

EDIT DATABASE <db_unique_name> SET PROPERTY 'LogShipping'='OFF'; (To stop redo apply temporarily)
SWITCHOVER TO <standby_db_unique_name>; (Planned role transition)
FAILOVER TO <standby_db_unique_name>; (Manual, unplanned failover without FSFO)
DISABLE DATABASE <db_unique_name>;
ENABLE DATABASE <db_unique_name>;
REMOVE DATABASE <db_unique_name>;
REMOVE CONFIGURATION <config_name>;

Security Considerations: Protecting Your Data Guard Environment

While Data Guard enhances availability, security must not be overlooked. A robust Data Guard implementation includes several security layers:

**Network Security:**
- Implement firewalls between your primary, standby, and observer hosts. Restrict Oracle Net Services (default port 1521) to only allowed IP addresses.
- Use Virtual Private Networks (VPNs) or Oracle Net encryption (e.g., TLS/SSL) for redo transport across insecure networks.
**Database User Privileges:**
- The SYS user (or a user with SYSDBA privilege) is typically used for DGMGRL connections. Ensure strong, unique passwords are used for SYS on both primary and standby databases.
- Limit direct OS access to the Oracle software owner (e.g., oracle user) to authorized personnel only.
**Oracle Wallet:** For enhanced security and to avoid hardcoding passwords in scripts or configuration files, consider using Oracle Wallet to store database connection credentials for DGMGRL connections and the Observer.
**Operating System Security:**
- Harden the operating systems of all servers involved (primary, standby, observer) following best practices (e.g., disable unnecessary services, regular patching).
- Ensure proper file system permissions for Oracle software and database files.
**Audit Logging:** Enable comprehensive audit logging for database activities, especially those related to Data Guard operations and user connections, to track changes and detect anomalies.

Best Practices: Maximizing the Effectiveness of Your Data Guard FSFO Setup

Implementing Data Guard with FSFO is a significant step towards high availability. Adhering to these best practices will ensure its long-term reliability and efficiency:

**Dedicated Observer Host:** Always deploy the FSFO Observer on a separate, independent host from both the primary and standby databases. This prevents the Observer from being impacted by the same failure that takes down the primary or standby. For ultimate resilience, consider deploying multiple observers (though only one is active at a time).
**Continuous Monitoring:** Implement robust monitoring for all components:
- **Database Status:** Monitor `V$DATABASE`, `V$DATAGUARD_STATUS`, and `V$ARCHIVED_LOG` for redo apply health and log gaps.
- **DGMGRL Configuration Status:** Regularly check `DGMGRL> SHOW CONFIGURATION VERBOSE;` and `DGMGRL> SHOW DATABASE ALL VERBOSE;` for any alerts or errors.
- **Observer Process:** Monitor the Observer process on its host to ensure it's always running.
- **Network Connectivity:** Monitor network latency and availability between primary, standby, and observer.
- Oracle Enterprise Manager (OEM) provides excellent integrated monitoring for Data Guard.
**Regular Failover Testing:** Conduct periodic, planned failover tests (at least once a year, or after major changes) to validate the entire process, including application connectivity to the new primary. This identifies potential issues before a real disaster strikes.
**Patching Strategy:** Ensure a consistent patching strategy across all Data Guard members. Apply recommended Oracle patches (e.g., Release Updates, Data Guard-specific patches) uniformly to avoid compatibility issues.
**Parameter Tuning:**
- FastStartFailoverThreshold: Adjust this based on your RTO requirements. A lower threshold means faster failover but increases the risk of false positives. Default is 30 seconds.
- FastStartFailoverPFL (Potential Failures Limit): This property can be set to allow FSFO to occur even if a specific number of data files or redo logs are offline on the primary, improving availability in specific scenarios.
**Application Reconnection Strategy:** Ensure your Peoplesoft application servers are configured to gracefully reconnect to the new primary database after a failover. This typically involves using a Transparent Application Failover (TAF) setup in `tnsnames.ora` or connection string, or using a service name that can be redirected by a load balancer.
**Documentation:** Maintain up-to-date documentation of your Data Guard configuration, failover procedures, and contact information for responsible personnel.

FAQ: Common Questions About Data Guard Broker and Fast-Start Failover

Q1: What happens if the Fast-Start Failover Observer itself fails?

If the Observer process fails or the Observer host becomes unreachable, Fast-Start Failover is automatically disabled. DGMGRL will issue an alert indicating that the Observer is no longer monitoring the configuration. In this scenario, automatic failover cannot occur. You would need to restart the Observer on the same or a different host. If a primary failure occurs while the Observer is down, a manual failover using DGMGRL's FAILOVER command would be necessary.

Q2: Can Fast-Start Failover be used with a Logical Standby database?

No, Fast-Start Failover is exclusively designed for use with Physical Standby databases. Logical Standby databases, which apply SQL statements rather than redo logs, have a different architecture and are not supported for FSFO. If you require zero data loss and automated failover, a physical standby with FSFO is the recommended approach.

Q3: How do I revert a failover, or switch back to my original primary database after a failover?

After a Fast-Start Failover, the original primary database is typically disabled and needs to be reinstated into the Data Guard configuration as a physical standby. Once reinstated and synchronized, you can perform a planned switchover operation using DGMGRL to make the original primary the primary database again. The steps are:

Ensure the old primary (now standby) is mounted and synchronized.
From DGMGRL connected to the current primary: REINSTATE DATABASE <original_primary_db_unique_name>;
Once reinstated and healthy: SWITCHOVER TO <original_primary_db_unique_name>;
Remember to restart the FSFO Observer if it was stopped during this process.

Conclusion: Empowering Peoplesoft with Automated High Availability

The configuration of Oracle Data Guard Broker with Fast-Start Failover is a cornerstone for achieving robust high availability and disaster recovery for critical Oracle Peoplesoft environments. By leveraging DGMGRL, database administrators can significantly simplify the management of complex Data Guard configurations, moving away from error-prone manual interventions towards a streamlined, automated approach.

Fast-Start Failover, in particular, transforms the recovery landscape by enabling automatic failover within seconds of a primary database outage, dramatically reducing RTO and minimizing the impact on business operations. For applications as vital as Peoplesoft, this translates directly into enhanced business continuity, reduced operational risk, and greater confidence in the underlying infrastructure.

While the initial setup requires careful planning and execution of prerequisites, the long-term benefits of an automated, self-healing Data Guard environment far outweigh the effort. Coupled with diligent monitoring, regular testing, and adherence to best practices, your Peoplesoft databases will be fortified against unforeseen disruptions, ensuring uninterrupted service delivery and data integrity. Embracing DGMGRL and FSFO is not just a technical implementation; it's a strategic investment in the resilience and reliability of your enterprise's most valuable assets.

Configure Oracle Data Guard Broker with DGMGRL Fast-Start Failover