Data Engineer

Data Engineer

Department: IT Development/Support

Employment Type: Full Time

Location: Remote - United States

Reporting To: Tom Cotton

Compensation: $85,000 - $100,000 / year

Description

The Data Engineer is responsible for designing, developing, testing, implementing, and supporting data stores and ETL processes to support organizational operations and business intelligence. This role implements the technical architecture and strategy for data ingestion, transformation, and storage, including data provisioning and integration. The Data Engineer is responsible for implementing and validating data governance standards and best practices.

Key Responsibilities

• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that support current and future data needs.
• Security and Compliance: Ensure data security and compliance with relevant data privacy and accessibility regulations.
• Data Pipeline Development: Design, develop, and maintain scalable data pipelines using Microsoft Fabric, including Azure Data Factory, PySpark, and Spark SQL.
• Data Integration: Integrate data from various sources into a unified data platform, ensuring data quality and consistency.
• Data Transformation: Implement ETL processes to transform raw data into meaningful insights.
• Data Storage Management: Manage data storage solutions, including Microsoft OneLake and Azure Data Lake.
• Performance Optimization: Optimize data processing workflows for efficiency and scalability.
• Documentation: Maintain comprehensive documentation of data engineering processes and workflows.
• Continuous Improvement: Stay updated with industry trends and propose new technology options for data platforms and tools.

Job Qualifications

Education:

• Bachelor’s degree in IT discipline, or a related field, or equivalent work experience.

Skills:

• Proficiency in Microsoft Fabric tools, including Azure Data Factory, PySpark, and Spark SQL
• Experience with data storage solutions such as Microsoft OneLake and Azure Data Lake
• Strong programming skills in Python and SQL
• Familiarity with data lakehouse and warehouse architectures, components, and modeling
• SQL / T-SQL
• Microsoft SQL Server / Azure SQL / Azure SQL Managed Instance
• Microsoft SQL Server Integration Services and Microsoft SQL Server Analysis Services
• Relational database architecture and components (views, tables, stored procedures, functions, etc.)

Experience:

• 3 - 5 years of experience in data engineering or a related role.

Desired Training:

• Microsoft Certified: Fabric Data Engineer Associate
• Microsoft Certified: Azure Data Engineer Associate

Desired Skills:

• Azure Data Lake Storage Gen 2 / Azure Blob Storage / Azure Files
• Reporting tools (Power …