Building ML Platforms That Scale

The Challenge

The program had tried twice to build a systems integration laboratory, and both attempts failed. When I joined in October 2023, I had four months to figure out what they couldn't—get the lab open by February 2024 or the entire evaluation program would remain blocked.

But here's the thing. "Opening" a lab doesn't mean it works. February came and the lab opened, but data lines were still being installed, the simulation cab was coming together, and nobody had actually tested whether all of this—the workload monitoring, the Unreal Engine environment, the multi-modal data streams, the live analytics—worked as a system. We had tested pieces, but we hadn't tested the whole.

So we created the Data Validation Pilot. One week, 20 participants, first real end-to-end test of everything. If it worked, we had a fully functional laboratory. If it didn't, we would be starting over. Beyond the technical validation, this was the first time personnel from data science, research, simulation, and software engineering were coming together as an actual team.

My Role & Approach

I was the single-threaded owner for the Data Validation Pilot—meaning if it went sideways, that was on me. I made the final go/no-go call to run the event, approved the datasets as ready for analysis, and presented results to leadership. If something failed its performance thresholds, I could block it from moving forward or send it back for rework.

The hardest part wasn't the technical work—it was coordination. Five internal teams, all with their own roadmaps: game development, software orchestration, data engineering, data analysis, human factors. Plus an external partner providing dashboard tooling. Getting everyone aligned on what "done" looked like took more time than building anything.

And then there was the GVSC situation. Three organizations—DCS (my company), Army Research Laboratories (ARL), and Ground Vehicle Systems Center (GVSC)—had different opinions on how simulation data should flow through the system. DCS and ARL needed it parsed into separate streams. GVSC's architecture sent it as one big chunk. After weeks of back-and-forth across leadership and engineering teams, I got GVSC to change their approach. That unblocked the Data Validation Pilot and became the standard data model all three organizations use now.

On the technical side, I defined what "good enough" meant: millisecond latency for physiological data (seconds would break our correlation analysis), perfect synchronization across all data streams (if physio data didn't line up with simulation events, the dashboard was useless), and less than 30% data loss (measured by comparing actual vs expected row counts). Those weren't arbitrary—they were the thresholds where the system stopped being useful.

I also spent a lot of time thinking about what could go wrong. Top three risks: simulation software crashing mid-event, data pipelines failing to log correctly, or time synchronization breaking across streams. Any of those would've invalidated the results. So we tested aggressively in two-week sprints and monitored everything live during the actual event.

During the week itself, I ran command and control—managing seven concurrent data pipelines across about 40 Kafka topics, watching for issues in real time, and making calls on the fly when things didn't go as planned.

Technical Solution

Results & Impact

• Successfully launched laboratory in February 2024 after two prior failed attempts (4-month delivery from project start), but launch was just the beginning—lab capability remained unproven until validation event
• Executed week-long Data Validation Pilot (July 2024) as first comprehensive end-to-end test of entire laboratory: Made final go/no-go call, ran live day-of execution coordinating newly-formed cross-functional team (data science, research, simulation, software engineering working together for first time), approved all datasets as analysis-ready
• Validation event success officially put the laboratory "on the map" as fully-functional: Demonstrated real-time multi-modal analytics (physiological monitoring, Unreal Engine 5 simulation, eye-tracking, UI events) working in concert with millisecond-latency synchronization—proving lab was truly "all systems go"
• Authored comprehensive 44-page laboratory overview post-validation: Documented mission statement, accomplished capabilities, complete software and data architecture, Unreal Engine plugins, and program structure—providing definitive snapshot of where program stood and establishing technical foundation for all future work
• Solidified new program structure across functional areas: Validation event execution week brought together personnel from data science, research, simulation, and software engineering as cohesive team for first time, establishing collaborative model still used today
• Platform supported first formal technology evaluation event (Dec 2024): "We were officially in business"—laboratory transitioned from experimental to operational evaluation facility
• Platform enabled 3 total evaluation events post-validation, supporting ~60 users; 4 teams now use platform for daily work (game dev, ML/analysis, data eng, human factors)
• Trained 2 engineers in advanced analytics capabilities: One data scientist (3-month intensive with daily meetings, now independent contributor making significant contributions) and one engineer in signal processing & data pipeline development
• Resolved critical 3-org architectural conflict: After weeks of negotiation, convinced GVSC to modify Kafka architecture to parse simulation data into separate topics, establishing common data model now mandated across DCS, ARL, and GVSC
• Primary technical author on winning $70.7M government contract (written January 2024 during laboratory launch): Advanced analytics components represented ~25% of proposal value (~$18M), earning equity incentive from CEO for technical vision that distinguished winning proposal
• Recognized for delivery impact: Promoted from Systems Engineer to Technical Program Manager (Feb 2024) after successfully launching laboratory that had failed twice, then proving capability through comprehensive validation event

Tech Stack

The Challenge

We received a computer vision prototype from Army Research Laboratories that wasn't in great shape—outdated model, unmaintained code, documentation that didn't exist. We could have tried to salvage it, but it would have taken longer than starting fresh, so we built a real-time ML inference platform from scratch that could handle 18 concurrent camera streams at 30-40 FPS, keep latency under 70ms, and hit 75%+ accuracy across 13 different target types.

Here's where it got interesting. Six months into development, with RGB cameras fully optimized, the customer mentioned almost in passing, "Oh, and we'll need IR capability too. Soldiers will definitely use that." Infrared. Completely different data stream. Two months until our major evaluation event.

That one hurt, not because it was technically impossible—we could do it—but because it shouldn't have been a surprise. I should have asked better questions earlier. We delivered RGB + IR in that two-month window, but it taught me something I haven't forgotten. Proactive customer communication isn't optional. You have to check in constantly, surface assumptions early, and not wait for them to remember the critical requirement they forgot to mention.

My Role & Approach

I owned this platform end-to-end—model selection, architecture decisions, delivery schedule. If we missed performance targets or the timeline slipped, that was on me. After getting the project charter approved and assembling the cross-functional team, I ran it from initial prototype through production MVP and still own the roadmap today.

First decision was defining what actually constitutes "good enough" for this system. I defined hard gates at 30-40 FPS processing, sub-70ms latency, and 75%+ accuracy, and those weren't arbitrary. Early versions were detecting tree branches and shadows as drones constantly, which made soldiers want to turn the system off. If the false positive rate is high enough that users don't trust it, the accuracy number doesn't matter—it's unusable.

We evaluated YOLO model variants (small, medium, large) and selected medium for the best balance between speed and accuracy. The Army Research Laboratories prototype was rejected—it would have taken longer to fix than building from scratch.

Big architecture call was migrating from Python to C++, because Python wasn't going to meet real-time performance requirements. Then we hit GPU bottlenecks, so I pushed for a client-server refactor to split frame extraction from inference and distribute the load across dual RTX 4090 systems over 100 Gigabit Ethernet. That got us from 3 streams to 18 with about 2× the throughput.

The IR situation was a learning moment. Six months in, the customer mentioned needing infrared support with two months to delivery, so I reset expectations with leadership—this is happening, but here's what it costs—and we delivered. Both RGB and IR working in that window. The lesson stuck. I now implement continuous stakeholder alignment checkpoints, because waiting for requirements to surface late is a recipe for disaster.

Technical Solution

Results & Impact

• Successfully delivered MVP supporting 18 concurrent camera streams (6× scale from initial 3-stream prototype) at 30-40 FPS with 50-70ms latency
• Achieved ~2× throughput improvement through client-server refactor and FP32→FP16 optimization, enabling low-latency, high-volume data streaming across dual-GPU systems
• Improved video streaming accuracy from 30% (baseline) to 75%+ across 13 target classes—critical distinction from static image accuracy, demonstrating real-world operational viability
• Delivered RGB + IR capability in 2-month sprint after mid-project scope discovery, meeting hard deadline for major evaluation event despite 6 months of RGB-only development
• Platform now serves as core dependency for all third-party model evaluations and downstream software integration efforts with external partners
• Continue to own roadmap and scaling strategy: Currently leading A/B testing framework for evaluating ruggedized computer vision models from industry partners against baseline architecture
• Key learning: Proactive customer communication prevents scaling surprises—now implement continuous stakeholder alignment checkpoints across all platform programs

Tech Stack

The Challenge

Build a voice AI assistant for military vehicles that soldiers would actually trust. Not a demo, not a prototype—a production system with 99.99% uptime requirements and formal SLAs, because when you're operating hands-free across four concurrent radio channels in a safety-critical environment, "mostly working" isn't good enough.

The technical bar was high: sub-300ms command latency (anything over 500ms feels sluggish and breaks user trust), 75%+ accuracy for military-specific speech, handling four audio streams simultaneously. But honestly? The harder part was navigating a vendor relationship that went sideways.

We partnered with an industry contractor through Army Research Laboratories. Six months in, they delivered something we didn't ask for—a complete model plus architecture instead of just model integration, which is what we'd agreed on. Integrating their approach would've taken more time and introduced more risk than starting over. I had to make the call: walk away from six months of work with this partner. It wasn't easy. But here's what I learned: advanced technology doesn't kill projects. Poor communication has a 100% failure rate.

My Role & Approach

I owned this platform from concept through user acceptance testing—currently in Phase 5 of a 6-phase roadmap—with final call on roadmap sequencing, scope changes, and whether we move from one phase to the next. If quality thresholds weren't met, I could block deployment and send it back for rework, and if we blew the schedule or budget in a safety-critical system, that was on me.

First thing I did was define what "good enough" actually means: 75%+ accuracy (measured by word error rate and command execution), sub-300ms latency, and 99.99% uptime. Those became the gates between phases—you don't move forward unless you hit them.

I set up a 6-phase roadmap—transcription, then command execution, then full AI assistant—with formal milestones for leadership and 2-week sprints underneath for the team. Built the architecture to be modular so we could swap models in and out as better ones became available, because this needed to be a platform service that other systems depend on, not a standalone tool.

Then came the vendor decision. Six months into a partnership with an industry contractor, they delivered something that didn't match what we'd agreed on—we'd asked for model integration (just plug their model into our architecture), but they gave us model + architecture. Integrating it would've taken longer than rebuilding from scratch and introduced risks we didn't want, so I recommended terminating the contract. Leadership agreed, despite the sunk time and despite pressure from Army Research Laboratories to make it work.

That one taught me something: you can have great technology and still fail if communication breaks down. Now I frontload alignment conversations, define deliverables in detail, check in constantly, and surface mismatches early rather than six months in.

We validated the platform with 21 users over a week—3-hour sessions, 4 concurrent radio channels, real operational conditions—and it worked. Currently the platform is on a funded roadmap, mandated for reuse, and serving as the reference architecture for next-generation capabilities.

Technical Solution

Results & Impact

• Successfully progressed platform through Phase 5 of 6 (User Acceptance Testing), achieving all formal SLA requirements with 21 users over week-long operational validation
• Met critical performance gates: <300ms command latency, 75%+ military speech accuracy (transcription + command execution), 99.99% uptime in safety-critical environment
• Made decisive vendor termination call that saved program integrity: After 6 months of development, terminated misaligned industry partner when delivered solution (model + architecture) didn't match requirements (model integration only), despite ARL pressure to continue
• Platform now on funded roadmap beyond initial project and mandated for reuse as reference architecture for next-generation AI assistant capabilities
• Currently working with industry partners to define expanded multi-agentic workflows, with architecture serving as baseline for future development efforts
• Established platform as mission-critical service at center of vehicle SoS architecture, with other subsystems depending on it for core functionality
• Key learning: Advanced technology doesn't kill projects, but poor communication has a 100% failure rate—applied this lesson to strengthen vendor alignment processes across all subsequent platforms

Tech Stack

The Challenge

When I joined in March 2023, the workload measurement program was struggling. External partners had been working on this for 5 years, collecting cardiac data with commercial biometric devices, but no quantitative workload metrics were making it through to inform design decisions. The result was a significant investment with limited return: multiple evaluation cycles producing data that couldn't be analyzed effectively, hardware reliability issues preventing consistent collection, and no clear path to the objective performance metrics the program needed.

I was brought in to figure out what was going wrong. Within three months of joining—still fairly new to the organization—I dug into the code, talked to stakeholders across teams, and worked to understand the technical gaps. What I found was that the challenge wasn't just one thing: the analytical approaches being used weren't well-suited to time-series physiological data, and the commercial hardware had fundamental reliability issues with connectivity, event synchronization, and data quality that made downstream analysis nearly impossible.

My Role & Approach

I was brought in to diagnose what wasn't working and recommend a path forward. What made this challenging—and ultimately rewarding—was that leadership trusted my technical assessment even though I was relatively new to the organization. They didn't take these decisions lightly, but they valued my perspective as a subject matter expert and acted on my recommendations. That trust shaped how this work unfolded.

Recommendation 1: Transition to Internal Capability (June-October 2023)

By June 2023, I'd identified that the external partnerships weren't working—not for lack of effort, but because the analytical requirements didn't align well with their expertise. I built cardiac preprocessing pipelines from scratch to demonstrate what internal capability could look like, then recommended transitioning those contracts and building expertise in-house instead. Leadership agreed. By October 2023, both partnerships had concluded, and I was asked to hire and train an internal data scientist to establish this as a permanent capability. This work, combined with my concurrent role in launching the systems integration laboratory, led to my promotion to Technical Program Manager in February 2024.

Recommendation 2: Walk Away from Vendor's $1M Demand (April-Mid 2024)

We'd partnered with a vendor who provided a neural network model for converting raw ECG signals to workload scores. My team built the complete architecture around it—real-time processing, platform integration, the full stack. Then the vendor raised IP concerns and asked for $1M to continue. Meanwhile, the hardware side still wasn't reliable. I evaluated what we'd built, what alternatives existed, and whether the investment made sense. My recommendation: walk away. Leadership trusted that assessment, and we moved on.

Recommendation 3: Pivot to Neural-Based Measurement (Mid-Late 2024)

With my neuroscience background, I was asked to evaluate an alternative approach using brain-computer interface technology. I met with the technology provider, reviewed their capabilities, and assessed how it would integrate with our existing architecture. The solution was compelling—it addressed the hardware reliability issues we'd been fighting and represented where the field was heading. I recommended pivoting the entire program from cardiac-based to neural-based measurement. Leadership approved. That decision aligned us with industry-leading technology and gave us the real-time, synchronized workload analysis capability we'd been working toward.

What I contributed across this work:

• Technical diagnosis through code review, stakeholder conversations, and hands-on analysis to understand root causes
• Built proof-of-concept preprocessing pipelines that demonstrated internal capability and informed strategic decisions
• Hired and trained internal data scientist, establishing the analytical expertise that replaced external dependencies
• Designed and built real-time processing architecture, including reverse-engineering existing models for platform compatibility
• Evaluated alternative technologies and assessed integration feasibility to inform strategic recommendations
• Provided technical guidance to leadership on vendor relationships and technology direction
• Led integration of new measurement approach into laboratory architecture for operational use

Technical Solution

Results & Impact

• Turned around a struggling program in months: Within three months of joining, identified what wasn't working and built proof-of-concept pipelines that demonstrated a path forward. My recommendations led to transitioning external partnerships and establishing internal capability by October 2023.
• Built lasting analytical capability: Hired and trained an internal data scientist who's now a fully independent contributor. What started as a gap-filling hire became a permanent addition to the team's analytical expertise.
• Navigated a complex vendor situation: When a vendor relationship became untenable financially, I evaluated our options and recommended walking away from a $1M demand. Leadership trusted that call, and we found a better path forward.
• Led a strategic technology pivot: Used my neuroscience background to evaluate an alternative measurement approach. The recommendation to pivot from cardiac to neural-based monitoring was significant—it meant changing direction on years of work—but it solved our hardware problems and aligned us with where the field is heading.
• Delivered operational capability from years of investment: The program now has real-time, time-synced workload analysis running in production. That capability has drawn interest from academic partners and industry, and it's become a foundation for how we evaluate new technologies.
• Support four active research partnerships: I define technical scope and collaboration strategy for government-funded partnerships with academic and industry organizations. The platform we built has become the basis for cross-organization technology assessment.
• Enabled new evaluation approaches: The workload platform opened doors to partnership opportunities we didn't have before—objective assessment of emerging technologies that need this kind of measurement capability.
• Created technical de-risking path: Built the cardiac preprocessing pipeline as a fallback before committing to the neural pivot. That work proved our internal capability and gave leadership confidence in the recommendation.
• Established repeatable methodology: Platform has supported four major evaluation events and become the foundation for physiological monitoring across programs. What was a capability gap is now a core strength.

Tech Stack