Engenheiro de Dados Databricks/DBT/Streaming/ML/AI

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Engenheiro de Dados Databricks/DBT/Streaming/ML/AI based in Brazil.** We are looking for a Senior Data Engineer to join a strategic initiative focused on enabling data capabilities for Artificial Intelligence solutions. This role combines data architecture, engineering excellence, and hands-on implementation to build scalable foundations for AI and machine learning applications. The professional will help define technical standards, create reliable data pipelines, and enable the consumption of structured and unstructured data by advanced AI systems. You will work with modern data platforms, cloud environments, and large-scale processing technologies while collaborating with multidisciplinary teams. The position requires strong technical ownership, curiosity, and the ability to validate new approaches through experimentation and practical solutions. This is an opportunity to contribute to innovative data architectures that support the next generation of AI-driven products and services. ### Accountabilities: The role is responsible for designing, evolving, and enabling scalable data solutions that support AI initiatives, acting as a technical reference for data architecture and engineering practices. Key responsibilities include: - Define architectural directions, standards, and best practices for data platforms focused on AI and machine learning use cases. - Design and implement mechanisms for ingesting, processing, and making unstructured data available, including documents, PDFs, audio, images, and videos. - Prepare and structure data for consumption by AI/ML and generative AI models, including processes such as feature engineering, chunking, and embeddings. - Build, optimize, and maintain scalable ETL/ELT pipelines and data workflows. - Automate, improve, and recommend enhancements to existing data processes and architectures. - Conduct proof-of-concepts and technical experiments to validate solutions before large-scale adoption. - Integrate data platforms with external systems, APIs, SaaS applications, and streaming solutions. - Support multidisciplinary teams by providing technical guidance and enabling data-driven innovation. - Apply best practices for data governance, reliability, scalability, and performance optimization. ## Requirements: We are looking for a professional with strong experience in data engineering, modern data platforms, and AI-enabled data solutions. The ideal candidate combines architectural vision with hands-on technical execution and collaboration skills. - Solid experience with Databricks and Apache Spark for large-scale distributed data processing. - Advanced knowledge of Python and SQL applied to data engineering (Scala experience is a plus). - Experience processing unstructured data such as documents, PDFs, audio, images, or videos for AI consumption. - Experience defining data architectures and technical strategies beyond pure pipeline execution. - Knowledge of data preparation for AI/ML and generative AI solutions, including embeddings, chunking, and model consumption workflows. - Experience building and maintaining scalable ETL/ELT pipelines. - Experience with DBT for data modeling and transformation processes. - Knowledge of Delta Lake, Lakehouse architectures, data versioning, and governance. - Experience integrating data with external systems through APIs, SaaS platforms, or streaming technologies. - Experience with cloud computing environments, preferably GCP. - Experience with Git, CI/CD practices, and agile development environments. - Ability to work independently, propose solutions, test hypotheses, and continuously improve processes. **Nice-to-have skills:** - Experience with MLflow and Databricks ML. - Hands-on experience with RAG pipelines, vectorization, embeddings, and vector databases. - Knowledge of generative AI integrations and machine learning A…