Senior Data Engineer (JB-2370)

As a Data Platform Engineer, you will join our data platform team, working alongside data engineers, backend engineers, and analysts and B team
In this crucial role, you will take part in building our next-generation data platform that will advance the company as a data-driven company. You will collaborate with different departments to expose them to data sources that shed light on every aspect of our product, including user behavior, new feature adoption, operational, financial, marketing KPIs, and more.
You will work with cutting-edge data technologies and address the business needs of other department members based on an in-depth
 understanding of our business.
Responsibilities
End-to-end development of company data infrastructure
Build and develop high-performance, near real-time ETL/ELT processes incorporating Airflow, AWS, Kubernetes, Databricks, Spark, and Kafka
Build data architecture to enable clients to produce and consume data using SQL or programmatic.
Drive the collection of new data and refine existing data sources
Develop, implement, and maintain change control and testing processes
Design and build monitoring tools
Implement data pipelines and data architecture that are scalable, fault-tolerant, and support high throughput and low latency, while considering security aspects at all times

Requirements

  • 5+ years of industry experience in a similar role - a must
  • 5+ years of Python experience with the ability to understand other languages - a must
  • Experience working with and deploying distributed data technologies - Kafka, Airflow, Spark, Presto, AWS Glue, DBT (or similar), etc. - a must
  • Relevant certification in big data architecture in one or more of the public clouds (AWS Solution Architect / Big Data | GCP Data Engineer) - an advantage
  • Build Microservices with high scale and Zero downtime
  • Deep understanding and experience with data lakes & data warehouses in the cloud (S3, Redshift, Cloud storage, Athena)
  • Proven experience working with SQL and NoSQL databases
  • Ability to build and deliver working software through iterative, agile processes
  • Experience working in a collaborative CI/CD software development environment, including Git, peer code review, and easily maintained, scalable, and documented code
  • Experience with ETL processes, data integration, and data warehousing