Data Engineer (India)

Summary

Andersen is hiring a Data Engineer in India for a project with a Georgian digital bank. The role involves data infrastructure modernization and support for analytics in a mobile-first environment.

The customer is a rapidly growing digital financial platform that operates entirely through mobile applications, without any physical branches. The company has been providing convenient and secure financial services, including quick loans, money transfers, utility payments, and card management, all without bureaucracy or hidden fees.

The project is modernizing data infrastructure by migrating legacy pipelines to a new platform, implementing an on-premises LakeHouse architecture, applying PII anonymization/tokenization, and developing analytics applications for users and automated systems.

Responsibilities

• Designing and optimizing ETL/ELT pipelines using SQL and Python.
• Migrating legacy pipelines to a modern on-prem Lakehouse architecture.
• Implementing data lake metadata and governance.
• Ensuring PII anonymization/tokenization with support for GDPR compliance and Banking KYC workflows.
• Developing real-time/batch processing solutions, with deployments on Kubernetes.
• Deploying and managing cloud and on-premise data infrastructure, with awareness of serverless paradigms.
• Building analytics applications and semantic layers, optimizing for SQL ACID and NoSQL BASE use cases.
• Managing data storage solutions including NoSQL databases, with understanding of their internal data models, use cases, and trade-offs.
• Integrating specialized databases and vector databases for storing embeddings or tokens.
• Maintaining and automating data orchestration workflows using Apache Airflow, leveraging DAG factories, XComs, and smart sensor patterns.
• Developing and exposing Python-based data services and maintaining reusable modules/packages following functional programming principles.

Requirements

• Experience in data engineering or a related field for 3.5+ years.
• Experience with on-premises infrastructure and systems tuning.
• Strong experience with SQL and Python for data processing.
• Hands-on experience with Apache Spark (batch & streaming), Flink, and Kafka.
• Experience with Spark Streaming, Spark ML, and Spark on Kubernetes.
• Experience working with cloud environments (AWS, Azure) and object storage solutions such as MinIO.
• Understanding of RDBMS and SQL engines like PostgreSQL, MS SQL, ClickHouse, Greenplum, Trino, and StarRocks.
• Experience with NoSQL databases (e.g., Neo4j, MongoDB, DynamoDB, Cosmos DB), their internal data structures, advantages, and use cases.
• Familiarity with SQL ACID vs. NoSQL BASE trade-offs.
• Experience working with HDFS, Delta Lake, and Apache Iceberg.
• Understanding of partition pruning, partition evolution, and differences/limitations between Delta Lake, Iceberg, Apache Hudi, and Apache Paimon.
• Familiarity with data deletion strategies in lakehouses (hard delete vs. soft delete).
• Experience with HIVE Metastore, Unity Catalog, and Trino.
• Knowledge of Apache Airflow for workflow orchestration.
• Experience with containerization using Docker, and orchestration using Kubernetes, including running Flink and Spark on Kubernetes.
• Experience working with Timeseries databases (e.g., TimescaleDB) and Vector databases.
• Awareness of semantic layers (e.g., Cube.js) and OpenLineage.
• Familiarity with Python web frameworks such as Django.
• Understanding of Python internals.
• Understanding of GDPR and Banking KYC compliance within data pipelines.
• Level of English – from Intermediate+.

Desired skills

• Experience with Elasticsearch for search and analytics.
• Exposure to DBT for analytics engineering workflows.
• Experience working with Scala in data engineering projects.
• Exposure to cloud serverless services (e.g., AWS Lambda, DynamoDB, Azure Cosmos DB).

Reasons to join us

• Andersen cooperates with such businesses as Siemens, Johnson & Johnson, AstraZeneca, BNP Paribas, Allianz, Ryanair, TUI, Verivox, etc..
• For the past four years, our company has been growing annually by 60%-100%, and we constantly involve top-notch specialists in our team.
• We have been strengthening our expertise since 2007. During this time, we have formed excellent teams with streamlined processes, where you can learn something new from your colleagues every day and enjoy your work.
• The company has mentoring and adaptation systems for each new employee. Our transparent performance review and assessment systems will allow you to determine your development path and plan your growth.
• The most important thing that we value in our employees is a commitment to continuous learning. The company supports them in this and gives them access to the best educational platforms, seminars, and practices. In addition, for over 17 years, Andersen has assembled a huge knowledge base and established a robust resource management institution.
• Our employees are provided with a benefit package and an extensive set of bonuses.
• We at Andersen have many different ways to grow. You can improve as a specialist or a manager, and all your activities will be decently rewarded.
• Work format: Onsite job from the office located in Bangalore.

Join us!