GDSC Datamart Ingestion Issue

Incident Report for IBM Security

Resolved

The ingestion service has returned to stable operation, and the platform is processing data normally. Ingestion has caught up to current data, and all core services are functioning as expected.
Our teams will continue to monitor the system, but no further operational impact is anticipated related to this incident.
Thank you for your patience.
Posted May 12, 2026 - 00:01 UTC

Update

Ingestion has caught up to current data, and the platform is operating in a stable state. Backlog queues have been processed successfully, and new incoming data is being handled normally as it arrives.
Downstream processing, including risk generation, has resumed and data is flowing as expected. We will continue to monitor closely to ensure sustained stability as regular ingestion cycles continue.
Posted May 05, 2026 - 23:44 UTC

Update

Ingestion continues to make steady progress. The new ingest queue is decreasing, and retry volume is now very low and stable. Queued data is actively being processed, and the platform remains stable under the current operating conditions.
We will continue to monitor closely as the remaining backlog drains and ingestion continues toward full normalization.
Posted May 05, 2026 - 15:10 UTC

Monitoring

Our Engineering and SRE teams worked through the weekend and made some adjustments to the ingestion service process today. The platform is actively processing queued data now and ingestion throughput is improving. We expect the ingestion of data to catch up by tomorrow morning.
Our teams remain closely engaged and are monitoring progress to ensure continued recovery. Further updates will be provided about the progress to completely restoring all services.
Posted May 04, 2026 - 21:06 UTC

Identified

We continue to experience material degradation in data ingestion and downstream event generation. While some progress has been made, ingestion throughput remains inconsistent and backlog processing is ongoing. As a result, risk events and outliers may be delayed or incomplete.
Posted May 04, 2026 - 14:32 UTC

Monitoring

A mitigation has been successfully implemented and we are monitoring the results.
The data warehouse engine rollback has completed, and data ingestion throughput has recovered as expected. The processing backlog is decreasing rapidly, and downstream components dependent on ingestion are beginning to normalize.
We will continue to closely monitor ingestion performance and data freshness to ensure sustained stability before marking the incident as fully resolved.
Posted Apr 30, 2026 - 23:50 UTC

Investigating

Status: Mitigating
We are continuing to address degraded data ingestion performance, which is causing delays in downstream components that rely on ingested data. The platform remains available; however, data freshness is impacted.
The root cause has not yet been confirmed. In coordination with Cloud provider Support, current findings suggest database lock contention in Data warehouse DB as a contributing factor.
We are actively tuning ingestion concurrency, scaling database capacity, and working with Cloud provider to restore normal processing rates.
Further updates will be provided as mitigation progresses.
Posted Apr 30, 2026 - 16:25 UTC

Update

A fix has been applied and issue is resolved. We are monitoring the service.
Posted Apr 29, 2026 - 12:45 UTC

Update

We are continuing to investigate an issue affecting the datamart ingestion service. Our engineering team is actively working to identify the root cause and restore normal operations. During this time, some data may be delayed or incomplete. We will provide further updates as more information becomes available.
Posted Apr 27, 2026 - 22:21 UTC

Identified

The root cause has been identified, and we are currently working on implementing a potential fix for the issue.
Posted Apr 27, 2026 - 18:51 UTC

Investigating

The datamart ingestion pipeline is experiencing partial failures, causing some datasets not to be ingested successfully. We are actively investigating the issue to determine the root cause and restore normal operation.
Posted Apr 27, 2026 - 15:58 UTC
This incident affected: IBM Guardium Data Security Center (US) (Data Management – Ingestion (Datamart)) and IBM Guardium Data Security Center (EU) (Data Management – Ingestion (Datamart)).