Back to all use cases

RemediationTelecommunications

The cluster was up for seven weeks with half its redundancy gone

Regional Telecom Operator

Recovering a Greenplum cluster that had run on a single copy of its data for seven weeks after a host failure, then rebuilding the mirroring layout so one host loss no longer halves performance.

Overview

Greenplum keeps a mirror of every primary segment on another host and promotes it when the primary fails. The promotion is silent: queries keep working, so a failover that nobody recovers can sit unnoticed until the next host event.

Challenge

A segment host in a twelve-host Greenplum 6 cluster failed. The fault tolerance service promoted the mirrors on its partner host, which then ran two sets of primary segments. Monitoring only checked that the coordinator accepted connections, so nothing alerted. Queries that touch every segment ran noticeably longer for seven weeks, and the team put it down to data growth. During routine operating system patching the partner host was rebooted, and with no in-sync copy of those segments the cluster stopped serving queries.

Environment

Greenplum 6 on twelve on-premises Linux segment hosts with group mirroring, a coordinator and a standby coordinator, feeding customer usage reporting and billing reconciliation.

Approach

AceMQ restored service first, then fixed the conditions that let the failure stay hidden. After the repaired host was back, we ran full segment recovery from the surviving copies, throttled so it fitted the overnight load window, and rebalanced roles. We then redesigned the mirror layout and put the cluster state under alerting.

Solution

  1. Read gp_segment_configuration and gpstate to confirm which segment pairs ran on a single copy and which mirrors were out of sync
  2. Brought the cluster back with the partner host's surviving data, then ran full gprecoverseg recovery of the failed host's segments in batches sized to the load window
  3. Rebalanced primary and mirror roles with gprecoverseg -r once every pair was in sync
  4. Moved from group mirroring to spread mirroring, so one host failure spreads its extra work across several hosts
  5. Added alerts on segments down, mirrors out of sync and change tracking state, with a runbook for the first failover
  6. Scheduled a gpbackup restore test and a rehearsal of standby coordinator activation

Outcome

Every segment pair was back in sync within two nights, and the nightly load returned to its earlier duration after the rebalance. A failed segment now pages the on-call DBA within minutes, and the team has a rehearsed recovery runbook.

Technologies

GreenplumPostgreSQLLinux
  • Assessment

    Greenplum MPP Cluster and Distribution Assessment

    Enterprise Technology OrganizationsCross-Industry

    A review of Greenplum segment count, distribution keys, storage selection and vacuum scheduling for petabyte-scale analytics, aimed at the design decisions that cause skew.

    GreenplumPostgreSQL
    Read case study: Greenplum MPP Cluster and Distribution Assessment
  • Consulting

    PostgreSQL High Availability and Failover Design

    Telecommunications OperatorTelecommunications

    Designing an automated failover architecture with quorum-based leader election, synchronous replication policy, and tested recovery procedures.

    PostgreSQLPatroniKubernetesetcdHAProxy
    Read case study: PostgreSQL High Availability and Failover Design
  • Support

    Greenplum ETL Window Concurrency and Spill Support

    Global Consumer Goods ManufacturerIndustrial

    Diagnosing queries that queued and never started during the nightly ETL window, and spill files that filled segment disks, then separating workloads with resource queues and memory limits.

    GreenplumVMware vSphereLinux
    Read case study: Greenplum ETL Window Concurrency and Spill Support
  • Consulting

    Greenplum 6 to 7 Upgrade Planning and Rehearsal

    Regional Health Benefits AdministratorHealthcare

    Planning and rehearsing an upgrade from Greenplum 6 to Greenplum 7 on AWS, including catalog checks, a compatibility review of SQL and extensions, and a tested rollback.

    GreenplumAWSPostgreSQL
    Read case study: Greenplum 6 to 7 Upgrade Planning and Rehearsal

Related Reading

Facing a Greenplum Production Issue?

AceMQ's senior Greenplum engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.