Senior Software Engineer - Orchestration

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a Senior Software Engineer on the Orchestration pod within Engine Productivity, you'll design and run the platform that executes large-scale end-to-end and integration tests, running the real, shipping client on real hardware, across Roblox's data centers, cloud, and our own device labs, so our engineering teams can ship the engine, clients, Studio, and more with speed and confidence. Every Roblox engine, client, and Studio change, along with the experiences built on top of them, should ship with confidence, and the Orchestration team is the layer that makes that possible. We build large-scale distributed services that turn thousands of test suites into a reliable, push-button pipeline: fanning work out across fleets of machines and real devices, moving artifacts to where they're needed, managing single- and multi-client test state, and giving test owners and maintainers a system to validate their own runs. It looks a lot like building a specialized cloud platform, with capacity-aware scheduling, isolation and sandboxing, and smart retry and backoff, plus the classic distributed systems problems (fairness, efficiency, failure handling, and reliability) at Roblox scale. You Will: Design and build the distributed services at the heart of our test platform: schedulers, control-plane components, and APIs that fan end-to-end and integration test work across fleets of machines and real devices. Run the real, retail-configuration client on physical hardware at scale, coordinating single- and multi-client test state, join flows, and gameplay scenarios. Build capacity-aware scheduling, isolation/sandboxing, and smart retry and backoff to run tests reliably and efficiently. Build systems that track the health of our hardware and cloud fleet and automatically detect and recover from failures with minimal human intervention. Develop the pieces that move artifacts, manage client state, and let test owners validate their own runs, including the launcher that drives the game client. Help evolve our legacy automation harnesses and point solutions into a cohesive Universal Test Platform, defining the core APIs other Client teams build on. Take projects from design through production, and support the many engineering teams (engine, clients, Studio, and beyond) that grow to depend on the platform. You Have: Strong experience building distributed systems or backend services such as schedulers, control planes, orchestration, or similar large-scale infrastructure. Proficiency in Go and/or Python (we work mostly in Go and Python, with some Luau for test and integration tooling). A solid grasp of distributed systems fundamentals: fairness, efficiency, failure handling, and reliability. Experience building systems that stay healthy on their own: health monitoring, fault detection, and automated recovery. Familiarity with running workloads on physical test hardware (phones, computers, consoles) is a strong plus; comfort getting hands-on with real devices. Experience with end-to-end or integration testing of real applications, not just unit tests or standard CI pipelines, is a plus. A passion for correctness and doing things "the right way." For roles that are based at our headquarters…