Lead Data Engineer, Data and Systems
4 days ago
🏢 In-office - San Francisco
Description
• Design, build, and maintain efficient and scalable ETL pipelines using tools like dbt, Fivetran, and orchestration frameworks such as Airflow. • Develop and implement robust schema designs and data models that support efficient querying and data integration across the organization. • Manage and optimize our data warehouse and lakehouse environments on platforms like Snowflake, ensuring data is accessible, reliable, and performant. • Implement data validation, cleansing, and anomaly detection processes to ensure the integrity and quality of our data. • Collaborate with Data Science and Analytics teams to support the deployment of ML models in production, including training, inference, and evaluation processes. • Implement monitoring and observability solutions to maintain the reliability and performance of data pipelines and models in production. • Leverage Docker, Kubernetes, and workflow management systems to ensure scalable and automated data processing workflows. • Ensure compliance with data governance standards and regulations, including GDPR and CCPA, through proper data lineage tracking, metadata management, and secure data handling practices.
Requirements
• MSc/PhD in Computer Science, or a related field • 5+ years of experience in data engineering, with a strong focus on building and managing data pipelines, data warehouses, and big data platforms. • Expert in Python and SQL, and experience with data technologies (Kafka, Spark, Hive/Iceberg, Postgres, Redis) and cloud infrastructure (GCP, AWS) is required. • Proficiency in orchestration and workflow management tools like Airflow, with experience in container orchestration such as Kubernetes being a plus. • Strong understanding of data quality management practices, data lineage, metadata management, and compliance with regulations like GDPR and CCPA. • Proven ability to work closely with Data Science and Analytics, and other cross-functional teams to deliver data solutions aligned with business needs. • Ability to lead data projects independently, make strategic infrastructure decisions, and stay updated with the latest data engineering technologies and practices. • Ability to work in a fast paced, dynamic environment where adaptability is imperative.
Benefits
• Medical, Dental, and Vision • Commuter Benefits • Life & STD Insurance • Company match on 401 (k) • Flexible Time Off (FTO) • Equity
Apply Now
