Streaming Data Engineer

About Data Reply As part of the Reply group, Data Reply offers a wide range of services that support customers in becoming data driven. We operate in various industries and business areas and work intensively with our customers so that they can achieve meaningful results through the effective use of data. Data Reply offers many years of experience in transformation projects to achieve “data-driven companies”. We focus on the development of data platforms, machine learning solutions and streaming applications - automated, efficient, and scalable - without making any compromises in IT security. Benefits Motivizer Benefits Platform to choose and manage all your benefits in one place. You receive a budget (550 PLN monthly). You can choose medical care package, meal tickets, sports cards (we have Multisport and on preferential terms, we have membership cards to one of the most popular Gyms), cinema tickets, shop vouchers, discounts and many more. Language Courses – you'll have access to a multi-language learning platform enabling you to practice you language skills and learn new ones! Regular and systematic further training opportunities - both internally and from external providers. We support your ongoing learning and development. Cooperation within an internal community is our everyday reality. We have networking events, coding challenges, and company parties for different occasions.
Requirements:
Educational Background: Bachelor’s or master’s degree in CS, Engineering, IT, or a related field Streaming Technologies: Knowledge of the Kafka ecosystem (Kafka clusters, connectors and clients, Kafka Streams) and other streaming platforms (e.g. Flink) Programming Language: Strong programming skills in languages such as Python, Java or Kotlin Cloud: Experience with public cloud providers AWS/Azure Database Technologies: Knowledge of SQL and NoSQL databases. Expertise in Big Data Technologies: Familiarity with big data frameworks and tools like Spark, formats such as Apache Iceberg Containerization: Experience in Docker, familiarity with Kubernetes is nice to have Data Modeling and Warehousing: Familiarity with data modeling and warehousing techniques Strong Analytical Skills: Ability to analyze complex data structures and derive insights to provide strategic guidance Excellent Communication: Strong interpersonal and communication skills to effectively collaborate with team members and clients. Very good knowledge of English and Polish Problem Solving: Strong problem-solving skills and the ability to propose creative, efficient solutions to complex problems Nice to have experience with Databricks, DBT Availability to work in a hybrid mode with at least 2 visits to the office per month
Must have: Python, SQL, Data warehouse, AWS, Azure, Java, Kafka
Nice to have: Databricks
Daily tasks: - Streaming Data Processing: Design, implement, test and maintain highly scalable real-time stateless and stateful data transformations, ensuring low latency and data quality
– Data System Design: Design and implement robust, scalable data processing systems: this involves selecting appropriate storage technologies, designing schemas, and planning integration strategies
– Data Integration and ETL Development: Develop and maintain pipelines for data transformation, integration, and ETL processes. Ensure data quality and accessibility
– Performance Optimization: Monitor, tune, and optimize data applications and database performance. Address any issues that may affect data processing speeds or analytics capabilities
– Consulting and Strategy: Provide expert advice and consultancy services to clients on data strategies, architecture choices, and technological advancements
– Analytics and Business Intelligence Support: Assist in developing analytics platforms and business intelligence solutions, ensuring that data can be effectively transformed into actionable insights
– Client Interaction: Work closely with clients to understand their business needs and technical requirements. Translate these requirements into effective data engineering solutions