Data Engineer (Databricks)

Company Overview:

Blue Orange Digital is a cloud-based data transformation and predictive analytics development firm with offices in NYC and Washington, DC. From startups to Fortune 500 companies, we help organizations make sense of their business challenges by applying modern data analytics techniques, visualizations, and AI/ML. Founded by engineers, we love passionate technologists and data analysts. Our startup DNA means everyone on the team makes a direct contribution to the growth of the company.

Position Overview:

Blue Orange is looking for an experienced Data Engineer with hands-on Databricks experience to join our talented multi-disciplinary team. The ideal candidate will have a passion for Databricks and data engineering, as well as modern data infrastructure practices and patterns. The candidate should be well-versed in SQL, Python, Delta Lake, Delta Live Tables, and Azure.

Candidates should have a strong understanding of modern data technologies, know how to drive the extraction of business requirements for data transformations, assess data quality, and possess excellent communication skills. This candidate will work directly with our clients to design, build, scale, and maintain production data validation systems and platforms.

Note: Please submit your resume in English, as all application materials must be in English for review and consideration.

Responsibilities:

• Provide advanced Databricks DLT (Delta Live Tables) performance improvement services, as well as Databricks infrastructure design, deployment, and operational services to our clients.
• Offer expertise and data engineering support with Python, Spark, SQL notebooks, and jobs to our clients.
• Work with the team and stakeholders to define data source, accuracy, and validation requirements.
• Build and maintain data ingestion pipelines, data models, orchestrations, transformations, and validation tests.
• Work with source data systems to extract and prepare data for analytics and testing.
• Collaborate with technical and business teams to evolve the data architecture.
• Work within an Agile environment to consistently deliver value for our clients.

Requirements:

• Databricks DLT (Delta Live Tables) experience.
• 2+ years of core Databricks experience.
• 4+ years of experience in a data engineering role, with expertise in ETL, data warehousing, data lakes, lakehouses, pipelines, modeling, data quality validation, and performance tuning.
• Expert experience with data ingestion, modeling, and conformance/compliance validation.
• Proficiency in SQL, Python, Spark, and data validation.
• Experience with AWS, GCP, or Azure.
• Ability to interact with others using sound judgment and a steady professional demeanor in a fast-paced environment.
• BA or BS degree in a technical or quantitative field …