Data Engineer - Community
This job is no longer accepting applications.
About the Role
Data is central to Twitch Community team’s decision-making process, and building the infrastructure to support this is critical to enable data-driven decision making in our product operations. As a Data Engineer at Twitch, you will be responsible for leveling up the capabilities of stakeholders across your team and cross-functional teams, enabling them to make better decisions using trusted data.
As part of the Community team at Twitch, you will be on the ground floor with the product data team, defining the way we collect and operationalize data, building coherent Logical Data Models that drive physical design and influencing future data roadmaps and strategy. In a typical week or month, your responsibilities may range from optimizing operational data storage to processing semi-structured data streams to building self-service business intelligence infrastructure for analysts. Whether you specialize in one functional area or work across all of them, your end product is always usable datasets that provide business value. Your work will pave the way for high-quality, high-velocity decision-making that will lead to safer, more rewarding community interactions across the platform
The ideal candidate is proficient in a broad range of data design approaches, has experience working with cross-functional product development teams, and has a passion for shaping the future of community-driven entertainment.
- Design, build and maintain a set of trusted data assets for a product or a group of products.
- Act as our team’s thought leader for defining data telemetry, storage and ETL processes.
- Partner with the Central Data Platform & Analytics teams to standardize data storage, decrease redundancies and evangelize finalized data assets.
- Partner with Analytics, Product and Engineering teams to understand data needs.
- Write software code and data solutions that are high quality and comprehensible.
- Have rigor around data architecture best practices:
- Create coherent logical data models that drive physical design.
- Balance customer requirements with technology requirements.
- Be proficient in a broad range of data design approaches.
- Be judicious about introducing dependencies.
- Create flexible data solutions without over-engineering.
- Understand how to be efficient with resource usage (e.g., system hardware, data storage, query optimization, AWS infrastructure etc.)
- Have knowledge of engineering and operational excellence best practices. Be able to make enhancements that improve data processes (e.g., data auditing solutions, management of manually maintained tables, automating, ad-hoc or manual operation steps).
- 3+ years of industry experience as a data engineer or in a related role, preferably in the consumer internet or gaming space, or working with a high-velocity, high-growth product / business.
- 3+ years experience in custom ETL design, implementation and maintenance.
- Proficient in SQL -- comfortable working with complex joins, window functions and writing SQL for aggregations.
- Experience working with Amazon Webservices, S3, EMR, Redshift etc.
- Experience building aggregates, optimizing data workstreams and maintaining data pipelines
- Comfort working independently, prioritizing projects, and managing stakeholder expectations across teams.
- Strong written and verbal communication skills.
- Eager to shape the development of a growing team and contribute to the design of novel products that shape the community experience for millions of viewers and creators.
- Obsessed with data quality and a strong belief in test driven development
- Strong familiarity with Twitch, our creators, and our community.
- Masters degree (preferred, but not required).
- Fluency in statistical analysis and programming using Python, R, or similar tools.
- Prior experience building end-to-end pipelines for supporting experimentation with machine-learning systems (e.g. recommendations, spam & fraud detection, notifications).
- Experience with a data orchestration framework such as Airflow, AWS Step etc.
- Experience with big data processing tech such as Spark, Hadoop etc.
Your application has been successfully submitted.
We are Twitch: a global community of millions who come together each day to create their own entertainment.