Escaping PoC Purgatory - Lessons Learned Scaling Databricks Solutions with the Government of Canada

Our team recently hosted a dynamic session on overcoming the challenges of scaling Databricks solutions within the Government of Canada. This session addressed common data challenges observed in public sector projects, provided an overview of Databricks, shared insights from past projects, and introduced our Databricks Launchpad framework.

The Challenge: PoC Purgatory

"85% of GenAI projects end up staying in the draft or PoC (Proof-of-Concept) stage" - Databricks

PoC purgatory occurs when a small scale proof-of-concept (PoC) demonstrates its value but fails to transition to full-scale production. This stagnation often arises due to misalignment with organizational priorities, technical bottlenecks, or scalability issues. 

"Over78% of the IT executives from the Global 2,000 companies surveyed said thatless than half of the PoCs they participate in result in productiondeployments" - CIO Innovation Index 

These PoCs can include AI and data projects such as chatbots, dashboards, analytics, or data consolidation across various sources. Overcoming these barriers requires thoughtful planning and execution from the outset. 

Key Strategies to Escape PoC Purgatory:

1. Plan for production from day one - Define production goals early in the planning process.
2. Prioritize and tackle high-risk items early - Identify and address potential blockers in the initial stages.
3. Streamline communication and stakeholder engagement - Maintain clear and consistent communication with all stakeholders. 
4. Emphasize data quality with rigorous testing - Build data quality checks into development workflows.
5. Address scalability and governance from the outset - Develop frameworks for governance and scalability early.
6. Harness automation and modern deployment platforms - Use automation tools to reduce manual intervention and accelerate delivery.

Through past projects and lessons learned, we developed Databricks Launchpad—a toolkit designed to help teams adopt best practices early in the PoC process and successfully transition from development to production.

Projects and lessons learned

To overcome common challenges moving from PoC to production, we developed the Databricks Launchpad framework after working on four significant projects across government and private enterprises. Here are the key lessons learned:

1. Federal Science Datahub (FSDH)

The FSDH is an open-source data science portal designed to provide scientists with access to data tools and storage. By automating approval processes and enabling cross-department collaboration, the FSDH eliminated data silos and scaled successfully to production. 

Key takeaways:  

• Early automation and deployment enable scalability.
• Cost tracking on a per-project basis helped predict production costs.
• Documentation and user guidance were critical to project success.
• Stakeholder engagement during the PoC phase gathered valuable feedback.

The FSDH portal houses hundreds experiments, requiring scalability in the solution, which early automation and deployment effectively enabled. Scientists relied on cost tracking within experimentation to predict production costs and project future budgets. To ensure the project gained traction departmentally, seamless user guidance and thorough documentation were crucial. Finally, valuable feedback gathered during the PoC phase ensured the project met user needs during production, contributing to its overall success.

2. NRCan Enterprise Datahub

The Enterprise Datahub centralized Databricks and analytical tools for NRCan, addressing the lack of a unified data repository and lack of reporting. The project provided a modern platform with automated pipelines and tools like Tableau and Power BI.

Key takeaways:

• Early governance models and data quality reduced manual efforts.
• Delayed deployment processes hindered early progress.
• Security requirements should have been considered earlier. 

Having governance models and ensuring data quality prior to production was essential to avoid unnecessary manual inputs and wasted resources. However, delayed deployment processes impacted early progress, as planning for deployment was not streamlined early enough, resulting in setbacks when moving to production. Additionally, the project required significant security measures that were not addressed during experimentation, further contributing to delays during the transition to production.

3. NSERC Data Consolidation

NSERC's data consolidation project tackled decentralized architecture and legacy tools, which hindered analytics and reporting. The project moved to production, delivering dashboards, a Databricks platform with modern ETL workflows, and automated pipelines. 

Key takeaways:  

• Lack of early stakeholder engagement slowed the transition to production
• Unified analytics and modern solutions enabled seamless analytics across multiple data sources.

The learning curve for Databricks and analytics required early hands-on involvement from stakeholders to avoid delays in transitioning to production. Additionally, implementing unified analytics and modern solutions necessitated thorough onboarding and training to ensure deep understanding and effective use across multiple platforms.

4. U.S. Legal Firm Document Processing with AI

This project automated document processing for a legal firm handling thousands of documents weekly. The AI-powered solution integrated with Databricks, reducing manual effort and streamlining workflows. 

Key takeaways:

• AI-driven data extraction is highly effective but can be costly.
• Deployment and early governance simplified production integration. 

AI was highly effective in extracting data from documents, providing valuable and specific insights. However, this capability required robust cost management structures to ensure expenses remained within the firm’s budget. Additionally, as the project grew larger during production, early deployment and governance efforts proved essential for simplifying integration and maintaining efficiency throughout the process.

These four projects helped shape Apption's Databricks Launchpad as they brought to light many easily avoidable obstacles when moving from PoC to production. 

Introducing Databricks Launchpad 

What is Databricks Launchpad?

Databricks Launchpad is a data proof-of-concept (PoC) template and framework for accelerating Databricks projects. 

The Launchpad streamlines development, enforces governance, optimizes costs, and ensures production readiness. Data teams and organizations looking to scale from experimentation to production can use the launchpad framework to help avoid common obstacles.

Why Databricks Launchpad? 

Launchpad helps you focus on your critical path. It was developed to help with common pain points encountered in scaling projects:

• Governance gaps and cost management
• Data quality challenges left unresolved for production
• Scaling issues from PoC to production environments

By using the Launchpad, organizations can achieve 20-50% savings in project timelines (based on saving 1-3 months of engineering hours). 

An overview of Databricks Launchpad

Values & Benefits

1. Speed and efficiency - Templates and automation streamline deployment.
2. Scalability and security - Role-based access, hidden secrets, and encryption ensure production readiness.
3. Cost optimization - Reduce unnecessary expenses with optimized compute and storage.
4. Training and onboarding - Simplify the onboarding of new resources. 

Databricks Launchpad was created to accelerate data and AI initiatives and help your PoCs transition to production effectively.

Watch the full webinar here: https://apption.com/discover/webinar-escaping-poc-purgatory