Modernizing FEMA's business intelligence

Challenge

The Federal Emergency Management Agency (FEMA) and National Flood Insurance Program (NFIP) faced a series of challenges - data quality, manual testing and data validation testing, and a legacy data architecture that was difficult to adapt rapidly, including specific issues in the areas of:

• Data visualization software instability and scalability
• Business intelligence and data visualization software support deficiencies and high costs
• Lack of self-service reporting capabilities, and a backlog of data requests
• Long processing times and limited real-time analysis capabilities because of inefficient extract, transform, and load (ETL) processes

Agency leadership recognized that to move to the proactive decision-making and response environment they needed, they would have to transform their environment to provide efficient data processing, reporting, and advanced analytics. Agency leaders turned to Guidehouse to work with them to modernize their business intelligence reporting and analytics tools.

Approach

Guidehouse used a phased approach to meet FEMA’s mission needs and achieve their efficiency and cost reduction goals..

Assessment and planning—Our experts conducted an in-depth analysis of the current system, worked with agency leaders and experts to define future-state capabilities, and facilitated the FEMA’s assessment of third-party vendors to identify system alternatives. As a result of the assessment, FEMA concluded that a Google-centric solution for reporting and analytics would best meet their needs, including ease of transition and adoption.

Tool selection and integration—Our experts worked with FEMA to identify the tools that would best align with the agency’s specific needs, including Looker Core, BigQuery, Cloud Storage, Datastream, Dataform, Cloud Composer, and Dataplex. The existing system architecture was then successfully migrated to Google Cloud.

Architecture development—Following our recommendations, and with Google’s support, FEMA adopted a Bronze (raw) – Silver (enriched) – Gold (curated) medallion data architecture, improving data quality for reporting and analytics while also enhancing data-staging, integration, and business-level aggregation.

Implementation and support—We developed structured query language workflows for data transformations and deployed them using continuous integration and continuous delivery (CI/CD) tools. To ensure data quality, these workflows were integrated with source control and further enhanced with a suite of automated integration tests. Our team provided hands-on support and collaboration with Google Professional Services for seamless integration and optimization.

Training and adoption—We delivered both self-paced and instructor-led training to ensure that knowledge gaps were closed during the journey to adopt the new business intelligence solution. Our team established a scalable data management strategy to enable better data cataloguing, lineage, security, and sharing capabilities.

Impact

These business intelligence and data analytics modernization efforts resulted in significant improvements and greater efficiency for agency analysts. 

• Batch ETL processing times have improved by up to 150%, reducing turnaround times for large recurring jobs. 
• Batch processing that previously took up to 24 hours with legacy tools were completed within two to six hours on BigQuery.
• Complex data modelling that previously took more than six hours to run can now be completed in under a minute.
• ETL streaming, now enabled through the new Google data tools, enables the agency to perform analysis in 15 minutes, down from 6-12 hours previously.

The implementation of Google Looker Core for data visualization has addressed challenges of software instability and allowed remediated past performance problems. Through report consolidation, Guidehouse was able to deliver operational performance and financial data in seconds, reducing complexity and improving the user experience. 

Since implementation, the agency’s simplified processes and improved integration enabled the team to implement self-service reporting, data catalogues with access-controlled metadata, automated data lineage tracking, and enhanced metadata management.

Adopting Google technology helped FEMA save hundreds of thousands of dollars in redundant license costs and standardized integration of environments, leading to better overall reliability and performance. The investments in the modernized data architecture and data management have positioned the program to adopt AI-based Google solutions with greater confidence. With enhanced forecasting capabilities, geospatial analysis, and fraud detection, the agency can be more proactive, provide better customer experience for policyholders, and respond more nimbly when needs arise.