Data Engineer

Summary

The IT company Andersen invites an experienced Data Engineer to work on a project for one of our customers in Georgia.

The customer is the first fully digital neobank in Georgia, existing solely on mobile phones. Since 2015, the company has been providing convenient and secure financial services, including quick loans, money transfers, utility payments, and card management, all without bureaucracy or hidden fees.

The project is modernizing data infrastructure by migrating legacy pipelines to a new platform, implementing an on-premises LakeHouse architecture, applying PII anonymization/tokenization, and developing analytics applications for users and automated systems.

Responsibilities

• Designing and optimizing ETL/ELT pipelines using SQL and Python.
• Migrating legacy pipelines to a modern on-prem Lakehouse architecture.
• Implementing data lake metadata and governance.
• Ensuring PII anonymization/tokenization with support for GDPR compliance and Banking KYC workflows.
• Developing real-time/batch processing solutions, with deployments on Kubernetes.
• Deploying and managing cloud and on-premise data infrastructure, with awareness of serverless paradigms.
• Building analytics applications and semantic layers, optimizing for SQL ACID and NoSQL BASE use cases.
• Managing data storage solutions including NoSQL databases, with understanding of their internal data models, use cases, and trade-offs.
• Integrating specialized databases and vector databases for storing embeddings or tokens.
• Maintaining and automating data orchestration workflows using Apache Airflow, leveraging DAG factories, XComs, and smart sensor patterns.
• Developing and exposing Python-based data services and maintaining reusable modules/packages following functional programming principles.

Requirements

• Experience in data engineering or a related field for 3.5+ years.
• Experience with on-premises infrastructure and systems tuning.
• Strong experience with SQL and Python for data processing.
• Hands-on experience with Apache Spark (batch & streaming), Flink, and Kafka.
• Experience with Spark Streaming, Spark ML, and Spark on Kubernetes.
• Experience working with cloud environments (AWS, Azure) and object storage solutions such as MinIO.
• Understanding of RDBMS and SQL engines like PostgreSQL, MS SQL, ClickHouse, Greenplum, Trino, and StarRocks.
• Experience with NoSQL databases (e.g., Neo4j, MongoDB, DynamoDB, Cosmos DB), their internal data structures, advantages, and use cases.
• Familiarity with SQL ACID vs. NoSQL BASE trade-offs.
• Experience working with HDFS, Delta Lake, and Apache Iceberg.
• Understanding of partition pruning, partition evolution, and differences/limitations between Delta Lake, Iceberg, Apache Hudi, and Apache Paimon.
• Familiarity with data deletion strategies in lakehouses (hard delete vs. soft delete).
• Experience with HIVE Metastore, Unity Catalog, and Trino.
• Knowledge of Apache Airflow for workflow orchestration.
• Experience with containerization using Docker, and orchestration using Kubernetes, including running Flink and Spark on Kubernetes.
• Experience working with Timeseries databases (e.g., TimescaleDB) and Vector databases.
• Awareness of semantic layers (e.g., Cube.js) and OpenLineage.
• Familiarity with Python web frameworks such as Django.
• Understanding of Python internals.
• Understanding of GDPR and Banking KYC compliance within data pipelines.
• Level of English – from Intermediate+.

Desired skills

• Experience with Elasticsearch for search and analytics.
• Exposure to DBT for analytics engineering workflows.
• Experience working with Scala in data engineering projects.
• Exposure to cloud serverless services (e.g., AWS Lambda, DynamoDB, Azure Cosmos DB).

Reasons to join us

• Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
• The opportunity to change the project and/or develop expertise in an interesting business domain.
• Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
• Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
• The opportunity to earn up to an additional 1,000 EUR per month, depending on the level of expertise, which will be included in the annual bonus, by participating in the company's activities.
• Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
• Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
• Certification compensation (AWS, PMP, etc).
• Referral program.
• English courses.
• Private health insurance and compensation for sports activities.

Join us!