Senior Data Engineer-IRD, Karachi

By Interactive Research and Development (IRD) Global, Singapore   Qualification:

• Bachelor’s degree in Computer Science, Data Science, Data Engineering, Software Engineering, or other related fields. 

Experience:

• At least 5 years of professional experience in data engineering or software development with focus on large scale data solutions.
• Hands-on experience with Apache NiFi or Apache Spark for data ingestion and processing. Strong experience in MySQL Server and MongoDB database administration and optimization.
• In-depth understanding of Linux-based environments and shell scripting. Strong knowledge of data modeling principles for both OLTP and OLAP systems. Experience in building and optimizing distributed datastores for high availability and performance.
• Familiarity with microservice architectures and their integration with data platforms. Expertise in query optimization techniques to enhance system performance.
  

Duties & Responsibilities:

• Design and implement scalable data ingestion and processing workflows using Apache NiFi and Apache Spark.
• Optimize and maintain analytical databases such as Vertica and ClickHouse for high-performance querying on large datasets. Administer and tune MySQL Server and MongoDB to ensure reliability, availability, and optimal performance.
• Develop robust data models for both OLTP and OLAP systems to support analytics and operational use cases.
• Build and manage distributed data architectures integrated with microservices, ensuring seamless and secure data access.
• Apply advanced query optimization and data lifecycle management practices to enhance overall system efficiency.
• Managing the complexity of designing and maintaining scalable data pipelines and distributed systems that can reliably process and serve 100 M+ row datasets across diverse platforms.
• Ensuring optimal performance and availability of analytical databases (e.g., Vertica, ClickHouse) while balancing cost, speed, and storage efficiency in high-demand environments.
• Integrating data workflows within microservice-based architectures, while maintaining data consistency, security, and low-latency access across teams and systems.