Data Engineer
Per Google, data engineers “design, build, operationalize, secure, and monitor data processing systems with a particular emphasis on security and compliance; scalability and efficiency; reliability and fidelity; and flexibility and portability.”
Responsibilities
Section titled “Responsibilities”Data Infrastructure
Section titled “Data Infrastructure”- Creates and maintains data pipelines
- Creates and maintains frameworks for data provenance
- Integrates external data stores, data lakes, data analysis frameworks and application database(s)
Database Architecture
Section titled “Database Architecture”- Architects and maintains the database(s) used for data analysis
- Contributes to the design and maintenance of the application database(s)
- Wearing multiple hats as necessary
Top 5 Competencies
Section titled “Top 5 Competencies”This role combines Manager of One competencies with:
- Work effectively with data - Core responsibility: build data pipelines, ensure provenance, integrate data sources
- Design and deliver technical solutions - Architect databases and data systems for analysis and applications
- Design maintainable systems - Build scalable data pipelines that the team can work with
- Master the technical stack - Polyglot expertise with databases, warehouses, lakes, ETL tools
- Establish and follow best practices - Ensure data quality, security, compliance, and governance
Additional Domain Expertise
Section titled “Additional Domain Expertise”Technical Polyglot
Section titled “Technical Polyglot”- Understanding of the relative strengths of various databases, data warehouses/data lakes, management tools, etc. acquired through experience.
- Data engineering programming languages (R, Python, SQL, etc.)
Database and Data Warehousing Administration
Section titled “Database and Data Warehousing Administration”- Set-up and maintenance of secure cloud environment
- Excellent communication skills; can bridge the gap between engineering lingo and human language
- Good schema design experience and understanding
- Strong command of:
- Business intelligence, query, and reporting tools
- Database design for read-only access
- Data warehousing design issues such as star schema
- Data warehousing technologies
- Data transformation and conversion
- Data quality issues
- Data formats for loading and unloading of data
- Data lakes