Databricks Data Engineer

PM Consulting · Philippines · posted today ago

engineering python devops data machine-learning

Listing supplied by Himalayas. 101 Careers did not originate this post and applications are handled by the employer.

About this role

Role Overview
A leading global technology organization is looking for a Databricks Data Engineer to design, build, and optimize modern data platforms in a cloud environment. This role focuses on developing scalable data pipelines, ensuring data quality, and enabling advanced analytics and AI use cases.
The ideal candidate brings strong experience in big data processing, cloud technologies, and the Databricks ecosystem, with a focus on building efficient, secure, and production-ready data solutions.

Key Responsibilities
Data Engineering & Pipeline Development

Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and processing.
Build high-performance data workflows using the Databricks ecosystem, including Delta Lake and structured streaming capabilities.
Implement reusable data transformation and validation logic for consistency across pipelines.
Ensure data quality, integrity, and reliability across all data processing stages.

Cloud & Platform Engineering

Develop and deploy data solutions on at least one major cloud platform:
Amazon Web Services
Google Cloud Platform
Microsoft Azure
Configure secure data environments, including access controls, auditing, and data governance.
Enable cross-platform data access and sharing using modern data architecture patterns.

Advanced Data Processing & Optimization

Build efficient data transformation pipelines using PySpark and Python.
Optimize performance of data jobs through tuning, resource management, and cost optimization techniques.
Implement data layering strategies (e.g., Bronze/Silver/Gold architecture) for structured data processing.

Data Integration & Analytics Enablement

Integrate data platforms with BI and analytics tools such as:
Microsoft Power BI
Tableau
Looker
Develop dashboards and reporting solutions to monitor KPIs, pipeline health, and operational metrics.
Support data sharing initiatives and enable collaboration across teams and domains.

AI/ML & Data Innovation

Prepare datasets for advanced analytics and AI use cases, including support for machine learning workflows.
Work with modern tools for experimentation, feature management, and model lifecycle support.
Explore new technologies and approaches to enhance data engineering capabilities.

DevOps & Deployment

Package and deploy data pipelines using CI/CD practices and version control systems.
Collaborate with engineering teams to ensure reliable and repeatable deployments.

Qualifications and Experience

At least 2 years of experience in data engineering or related roles.
Strong hands-on experience with Python and PySpark for data processing.
Proven experience building data pipelines in cloud-based environments.
Solid understanding of data modeling, data warehousing, and ETL/ELT concepts.
Experience working with large-scale, distributed data systems.

Technical Skills

Expertise in:
Apache Spark / PySpark
Delta Lake and modern data lakehouse architecture
SQL and data transformation logic
Familiarity with:
Data orchestration tools (e.g., Apache Airflow)
Big data technologies (e.g., Hadoop, Kafka, Hive)
ETL tools such as Informatica, Talend, Matillion, or Fivetran
Data build tools (e.g., dbt)

Preferred Qualifications

Databricks or cloud certifications
Exposure to machine learning concepts and frameworks
Experience with DevOps practices and CI/CD pipelines
Knowledge of real-time streaming and serverless architectures

Skills and Competencies

Strong analytical and problem-solving skills
Attention to detail and commitment to data quality
Ability to work independently and in collaborative environments
Effective communication skills for both technical and non-technical stakeholders
Adaptability in a fast-paced, technology-driven environment