Senior Data Engineer

Job Description 

ICF is a mission-driven company filled with people who care deeply about improving the lives of others and making the world a better place. Our core values include Embracing Difference; we seek candidates who are passionate about building a culture that encourages, embraces, and hires dimensions of difference.  

Our Health Engineering Systems (HES) team works side by side with customers to articulate a vision for success, and then make it happen. We know success doesn't happen by accident. It takes the right team of people, working together on the right solutions for the customer. We are looking for a seasoned Senior Data Engineer who will be a key driver to make this happen. 

Responsibilities:  

Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow 

Develop and deploy data processing workflows on the Databricks platform 

Develop API services to facilitate data access and integration 

Create interactive data visualizations and reports using AWS QuickSight 

Builds required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies 

Monitor and optimize the performance of data infrastructure and processes 

Develop data quality and validation jobs 

Assembles large, complex sets of data that meet non-functional and functional business requirements 

Write unit and integration tests for all data processing code 

Work with DevOps engineers on CI, CD, and IaC 

Read specs and translate them into code and design documents 

Perform code reviews and develop processes for improving code quality 

Improve data availability and timeliness by implementing more frequent refreshes, tiered data storage, and optimizations of existing datasets 

Maintain security and privacy for data at rest and while in transit 

Other duties as assigned 

Minimum Qualifications:  

Bachelor's degree in computer science, engineering or related field 

7+ years of hands-on software development experience 

4+ years of data pipeline experience using Python, Java and cloud technologies 

Candidate must be able to obtain and maintain a Public Trust clearance 

Candidate must reside in the US, be authorized to work in the US, and work must be performed in the US 

Must have lived in the US 3 full years out of the last 5 years 

Preferred Qualifications:  

Experienced in Spark and Hive for big data processing 

Experience building job workflows with the Databricks platform 

Strong understanding of AWS products including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, EMR, …