Building MLOps Bridges: Our Journey in Uplifting Agencies

A Practical Guide to MLOps Adoption Across Government Agencies

In publishing our MLOps playbook, we have spoken to over 20 government agencies to understand their challenges with Machine Learning Operations (MLOps). What we realised is that agencies were at various stages of their MLOps journey. Some were just starting from scratch and eager to know more, while others were grappling with legacy issues or technical debt. As such, we hope to share some insights and tips to help agencies chart their path forward.

Below is an outline of what we will cover:

1. Foundations: Even small changes can lead to substantial long-term benefits. We will discuss a few basic principles that data science teams should follow, which remain relevant whether or not MLOps is implemented.
2. Engagement Strategy & Archetype: We outline our engagement strategy for understanding agencies’ archetypes and adopting MLOps. For each archetype, we propose suitable approaches and strategies for adopting and advancing MLOps.
3. Blueprint: We will show you an example of what a high-level setup might look like. Just a heads up — things are usually more complex in reality but we want to give you “an idea”.
4. Recommendations & Ending Remarks: A compilation of observations and alternative approaches to adopting MLOps, such as engaging a vendor for setup. While not comprehensive, we hope it sparks valuable discussions for your team.

Foundations

The approach to MLOps adoption is much like building bridges. You have to check what building materials you have on hand, and have a clear picture of where you want to end up. This “bridge-building” metaphor describes how we connect an agency’s current state with its future aspirations. Ultimately, bridges are built so that one can get to the other side fast. In MLOps, we aim to shorten the time from zero to experimentation, and experimentation to deployment. We have identified three core MLOps foundations:

Culture — A bedrock for MLOps

MLOps need a supportive culture to thrive. Successful transformations require:

• Standardised processes and delivery: Set up data science workflows and environments to enable Data Scientists and Machine Learning(ML) Engineers to perform their work effectively. This is particularly important when integrating with existing products and/or vendor platforms.
• Proper team processes to balance rigour and speed: Establish processes for code reviews & peer reviews of scientific methodology.

Workflow & Tools — Well-engineered workflows that connect and support MLOps component

MLOps need well-designed workflows and processes. Successful transformations require:

• Experiment Tracking, using tools like MLFlow.
• Version Control, using tools like Git.
• Storage of model and data artifacts.
• Monitoring of ML Systems and models to ensure consistent performance and to detect any degradation over time.

Product Mindset — Making sure what you develop is what the users want

ML systems must serve real users. This includes continuous improvement, ensuring required uptime and providing the right user experience. Successful transformations require:

• Understanding your users and ensuring your machine learning system remains relevant to their needs.
• Continuously refining and adding new features.
• Implementing automation and pipeline (think CI/CD!) to streamline your workflow and increase efficiency.

Engagement Approach

We have developed an engagement flow to help understand agencies’ current state and where they want to get to.

Every engagement begins with a thorough assessment:

• What’s happening right now (use cases and business value)?
• What’s getting in the way (pain points and worries)?
• What is the current landscape (data infrastructure and team capabilities)?

This assessment helps us categorise agencies into three distinct types. The type of “bridge” needed depends on the agency’s landscape. An agency could be a new

This assessment helps us categorise agencies into three distinct types.

The New Settler (Stage 0 MLOps) — A new settler exploring uncharted territories

Agencies beginning their journey typically conduct data science work largely manually and using notebooks. Like building a footbridge, we recommend starting small:

• Choose manageable and contained projects.
• Identify and document pain points. Use these experiences to build a compelling case for enhanced capabilities like deployment, automation, or monitoring.

Pro-tip: For agencies with already existing numerous use cases, identify common pain points across projects. If a solid use case is not available, create mock scenario or simulation. Whether using mock or real use cases, adopting MLOps should demonstrate benefits, particularly when it leads to discussions about challenges and opportunities for automation and scaling. For those requiring an MLOps environment for experimentation, MAESTROis available via the intranet.

The City (Stage 1 MLOps) — Already maintaining an existing infrastructure

Agencies at this stage typically have existing data infrastructure with proper data pipeline but have not yet explored CICD or create ML pipeline. As with expanding or build new bridges to manage increased traffic, we aim to:

• Identify technical debt early and introduce enhanced workflows (including tracking, version control, model registry & monitoring).
• Implement pilot projects that scale up current models that demonstrate quick wins.
• Build automation around existing processes

Pro-tip: Agencies with existing data pipelines are well-positioned to implement ML workflows and pipelines. You can either request create a space (e.g., a compartment in GCC) and start creating a proof of concept. If that is not possible, then setting up data integration and creating a proof of concept in MAESTRO is also a viable option. Reach out to us if you have any difficulties.

The Metropolis (Stage 2 & 3 MLOps) — Established and already operating a set of sophisticated bridges

Agencies at this stage typically are already adopting MLOps, and may be seeking to further enhance their practices. This includes consolidating use cases in a regular basis and refining monitoring, testing and maintenance protocols.

The ultimate goal is to reduce workload and improve oversight. Agencies still at bare minimum Stage 2 MLOps should seek to improve their processes by implementing automated testing to ensure your pipeline operation, and continuously improving monitoring systems to track model and system performance. The scope of improvements varies according to each agency’s needs. Further details are available in our capability map levels.

The Blueprint: Target Architecture

Agencies often enquire about the ideal “end state”. We’ve put together a basic architecture framework that agencies can work backwards from or adapt based on their priorities. Below is a quick overview of all the layers you may expect for bare minimum Stage 2 MLOps.

Data Layer

• Data Warehouse / Data Lake
• Feature stores
• Vector DB

CI/CD (Build & Deploy) Execution Layer

• Gitlab, with docker runners
• AWS codepipeline
• Azure DevOps

MLOps & LLMOps Management Layer

• MLOps Orchestration Layer (e.g., SageMaker, Azure ML or Google Vertex AI)
• Model registry
• Experiment Tracking (e.g., MLflow)
• Unit, Data & Model Testing

AI Service Layer

• API services for foundational Model like OpenAI GPT-5
• Amazon Bedrock

Monitoring & Operations Layer

• System Observability
• Model monitoring (e.g., data drift, concept drift)

More Tips & Ending Remarks

MLOps integrates DevOps, AI Science and AI Engineering, and is frequently considered a complex field. And we recognise this challenge. We offer some recommendations for facilitating an effective MLOps journey.

Break down complex tasks into smaller, achievable “quick wins”

• Our Stage 1 MLOps is crafted with simplicity in mind, specifically a basic service end point. Even if everything else is manual, we have improved model accessibility for end-users. Something simple in your case might be different requirements entirely.
• By doing something simple, we can identify blockers, improve processes and most importantly build momentum and confidence.

Internal capability building is essential, whether or not external expertise or vendors are involved

• While external MLOps expertise or vendor can accelerate adoption, they often require clear articulation of MLOps needs, and internal capabilities are essential to provide that guidance.
• Developing a Proof of Concept (POC) is an effective way to refine requirements and gain clarity. The POC can then be scaled through external expertise or vendors.

Take action and address security concerns, which are manageable

• Avoid both underestimation, which can lead to security becoming a later roadblock, and overestimation, which can result in inaction.

Leverage on AI Coding Assistants

• Utilising AI coding assistants can significantly expedite MLOps development. Ideal for development and prototyping.

Call for help

• For consultations, visit us at Lorong AI on Thursdays. Government agencies can book a slot at go.gov.sg/lai-maestro-att.

Thank you for reading this far. If you want to know more, check out the following blogs and playbooks.

• MLOps playbook
• MLOps Transformation: Moving from Stage 0 to Stage 3 (Part I)

If you like this post, give it a like or leave us a note at aipractice@tech.gov.sg to tell us what you would like to hear more about. Our next article will be on “Beyond Accuracy: The importance of MLOps evaluation and monitoring”.

Acknowledgements

• This is joint work with my MLOps teammates from AI Practice, Richard Yan and Raymond.
• Special thanks to Daniel Yuen from the MAESTRO team for his advice and big support from MAESTRO.
• Special thanks to Mehul Shah from AI Practice for his engagement tips & writing guidance.