Skip to content

Interview questions

Data engineer interview questions

Common questions for Data engineer roles. Read, copy and practise.

Free with every account.

Practice questions from open collections and written with AI. Not reported from interviews.

Technical

Coding

  1. How would you use Python to efficiently parse and process large JSON files for a data pipeline?

  2. Write a Python function to deduplicate records in a large CSV file without loading it fully into memory.

System design

  1. Describe a system design for an end-to-end Spark batch processing pipeline that processes terabytes of data nightly.

  2. Can you design a scalable data architecture using a cloud lakehouse to support real-time analytics and batch processing?

  3. How would you design a scalable data platform to handle petabytes of data ingestion and querying with low latency?

  4. Describe how you would architect a data platform that supports real-time analytics and batch processing simultaneously.

Other technical questions

  1. Name types of join methods RDBMS (e.g. postres) supports.

  2. What is 'nested loops join'?

  3. What is 'hash join'?

  4. When hash join may be selected by optimizer?

  5. What is window function?

  6. When would you use left join and when left outer join?

  7. What is table size limitation for postgres?

  8. What is Apache Hive Metastore?

Show 87 more other technical questions
  1. What is MapReduce, how it works?

  2. Assuming that your have developed a pyspark program to process a data batch, how spark executes it?

  3. What is default gateway?

  4. Imagine you upload a file to s3 bucket and receive 503 http error. How would you troubleshoot and what possible solutions would you suggest?

  5. A network resource is unavailable, how would you troubleshoot this issue?

  6. What is Quality of Service?

  7. Explain how kafka works?

  8. What is one-liner? Give example in python.

  9. What is the difference between generator and iterator in python?

  10. Why GIL is used in python?

  11. How do you use EXPLAIN or EXPLAIN ANALYZE to identify bottlenecks in SQL queries?

  12. What indexing strategies would you apply to optimize a slow query on a large transactional table?

  13. Describe the difference between correlated and non-correlated subqueries and their impact on performance.

  14. How would you optimize a query that joins multiple large tables with millions of rows each?

  15. What are window functions in SQL, and can you provide an example where using them improves query efficiency?

  16. Explain the difference between clustered and non-clustered indexes and when to use each for query optimization.

  17. How do you handle query optimization in a database with frequent insert and update operations affecting index maintenance?

  18. Can you describe how partitioning tables might improve query performance and give an example scenario?

  19. Explain the trade-offs between using SELECT * versus selecting specific columns in your queries.

  20. How do you optimize queries involving aggregation and grouping on large datasets?

  21. What strategies do you use to rewrite inefficient SQL queries for better performance?

  22. Describe how you would use query hints or optimizer directives to improve query execution plans.

  23. Can you explain how Spark executes a batch processing job from submission to completion?

  24. How do you optimize Spark batch jobs to handle large datasets efficiently?

  25. Describe how Spark handles data partitioning in batch processing and why it matters.

  26. How do you manage schema evolution in Spark batch pipelines when the source data schema changes?

  27. Which file formats do you prefer for batch processing with Spark and why?

  28. Can you discuss the trade-offs between using Spark SQL and DataFrame APIs versus RDDs for batch jobs?

  29. How would you handle job failures and ensure data correctness during Spark batch processing?

  30. Explain the role and configuration of the Spark shuffle process in batch jobs.

  31. Can you explain how Kafka achieves high throughput and fault tolerance in its architecture?

  32. How do you design a Kafka topic and partition strategy to handle a high volume of sensor data streaming?

  33. Describe how you would implement exactly-once processing semantics in a Kafka Streams application.

  34. What are the main differences between Kafka's consumer groups and how do they help with scalability?

  35. How do you monitor the health and performance of a Kafka cluster in a production environment?

  36. Can you describe a scenario where using Kafka Streams is more appropriate than using Apache Spark Streaming?

  37. What strategies do you use to handle schema evolution in Kafka messages without breaking downstream consumers?

  38. Explain how Kafka's log compaction feature works and give a use case where it is beneficial.

  39. How do you ensure data consistency when integrating Kafka with external databases or data lakes?

  40. Can you explain the differences between ETL and ELT pipelines and when you would choose one over the other?

  41. How do you design an ETL pipeline to ensure data quality and integrity throughout the process?

  42. What strategies do you use to optimize the performance of ETL jobs processing large datasets?

  43. Describe how you would handle incremental data loading in an ELT pipeline to minimize processing time.

  44. How do you monitor and alert failures or delays in an ETL or ELT pipeline?

  45. Can you walk me through the process of transforming raw data in an ELT pipeline using SQL?

  46. What are common challenges you have faced when migrating ETL pipelines to modern cloud data platforms?

  47. Explain how you manage schema evolution in source data to ensure ETL pipelines continue working without disruption.

  48. In your past projects, how did you balance the trade-off between data freshness and processing cost in pipeline scheduling?

  49. Can you explain how Airflow's DAGs are structured and how task dependencies are managed within them?

  50. How do you handle retries and failure notifications in Airflow workflows to ensure reliability?

  51. Describe how you would use Airflow sensors and triggers to orchestrate workflows based on external events or data availability.

  52. In your experience, what are the best practices for managing and scaling the Airflow scheduler in a production environment?

  53. How can you optimize Airflow DAG performance when dealing with large numbers of tasks and complex scheduling intervals?

  54. Explain how you would implement dynamic task generation in Airflow to cater to variable dataset sizes or schemas.

  55. What are the limitations or challenges of using Airflow for orchestration in a real-time streaming data pipeline?

  56. How do you monitor Airflow workflows and handle alerting to ensure timely resolution of issues?

  57. Can you compare Airflow to other orchestration tools you've worked with, focusing on scheduling capabilities and flexibility?

  58. How do you ensure idempotency and data consistency when rerunning Airflow tasks after failure recovery?

  59. How do you define data quality in a data engineering context, and what key metrics do you monitor to ensure it?

  60. Describe strategies you have used to detect and handle corrupted or missing data in large data pipelines.

  61. Can you explain how you implement data validation tests during ETL processes to prevent bad data from entering the warehouse?

  62. What tools or frameworks do you prefer for data quality testing and why?

  63. How would you design an observability system to monitor data pipeline health and quickly detect anomalies?

  64. Explain how you would use checksums or hashes to validate data integrity across distributed systems.

  65. Walk me through how you would test a new data pipeline end-to-end before deploying it to production.

  66. How do you balance thorough data testing with pipeline performance and latency requirements?

  67. Explain how you would manage memory usage in a Python script that handles streams of data in real time.

  68. Can you describe the differences between Python’s list, tuple, and generator in data processing scenarios?

  69. What Python libraries would you use to connect and query a relational database, and why?

  70. How would you design a Python-based data ingestion system to consume data from multiple APIs?

  71. Describe your approach to error handling and logging in Python scripts used for ETL processes.

  72. What strategies would you implement in Python to optimize the performance of a data transformation job?

  73. Explain how Python's multiprocessing and threading differ and when you would use each in data engineering.

  74. Have you used Python to automate data quality checks? Please give an example of your implementation.

  75. How do you approach designing data storage layers to optimize for both cost efficiency and high performance?

  76. Can you explain the trade-offs between a centralized data warehouse and a data lake in a data platform design?

  77. How would you structure the data pipeline architecture to ensure fault tolerance and data recovery capabilities?

  78. Explain how metadata management fits into designing an enterprise-level data platform.

  79. What monitoring and alerting mechanisms would you implement to maintain platform reliability and performance?

  80. What are the differences between CSV, JSON, Parquet, and Avro file formats in terms of storage efficiency and query performance?

  81. How does columnar file format like Parquet improve performance for big data processing compared to row-based formats?

  82. Can you explain how data partitioning works in distributed storage systems and why it is important for query optimization?

  83. Explain the concept of bucketed tables and how bucketing complements or differs from partitioning in distributed file storage.

  84. What are the implications of storing data in a data lake versus a traditional data warehouse in terms of file formats and storage methods?

  85. How does predicate pushdown work with different file formats, and why does it matter for query performance?

  86. When designing a data pipeline, how do you optimize the choice of file format and partition strategy to minimize downstream processing costs?

  87. Can you walk through an example where improper partitioning led to performance degradation and how you resolved it?

Domain knowledge

  1. What's the difference between transactional and analytics databases?

  2. What's the difference between star and snowflake schema?

  3. Can you explain the difference between star schema and snowflake schema in data warehousing and when to use each model?

  4. How do you approach designing a data warehouse schema to optimize for query performance and scalability?

  5. Describe the role of fact tables and dimension tables in a data warehouse and how they relate.

  6. When modeling data for a financial transaction system, what considerations impact your data warehouse design?

  7. How do you model and manage time-series data in a data warehouse setup?

  8. What are some techniques to ensure data integrity and consistency in a data warehouse environment?

Show 19 more domain knowledge questions
  1. How would you design a data warehouse architecture to support both batch and real-time data ingestion?

  2. Explain the concept of normalization and denormalization in data modeling and their impact on a data warehouse.

  3. Can you discuss the trade-offs between columnar and row-based storage formats for data warehousing?

  4. In a marketing analytics context, how would you use Kafka streams to provide real-time customer segmentation insights?

  5. Can you explain the differences between cloud data warehouses and lakehouses, and when you would choose one over the other?

  6. How do cloud data platforms ensure data security and compliance when handling sensitive information in a lakehouse architecture?

  7. Describe your experience with implementing data ingestion pipelines into a cloud lakehouse environment, including the tools you used.

  8. How do you optimize query performance in a cloud lakehouse platform when dealing with large-scale, unstructured data?

  9. Explain how ACID transactions work in a cloud lakehouse platform and why they're important for data integrity.

  10. What strategies do you use to control data partitioning and clustering in cloud lakehouses for efficient data retrieval?

  11. In your opinion, what are the main advantages of using lakehouses over traditional data lakes in cloud environments?

  12. How do you monitor and maintain data quality and lineage in a cloud lakehouse ecosystem?

  13. What strategies would you use to ensure data quality and consistency across multiple data sources in your platform?

  14. What considerations would you include for data security and compliance when designing a cloud-based data platform?

  15. In your experience, how does domain knowledge in business operations influence your data platform design decisions?

  16. What are the trade-offs between partitioning data by date versus by customer ID in a data lake?

  17. How would you decide when to use compressed file formats like Snappy or Gzip for storage in a data pipeline?

  18. Describe how you would organize data storage for a multi-tenant application where each tenant’s data must be isolated but efficiently queryable.

  19. What impact does the choice of file format have on schema evolution and backward compatibility in a data warehouse?

Behavioral

Motivation

  1. What draws you to working with batch processing technologies like Spark in data engineering?

  2. What motivates you to work with streaming technologies like Kafka and real-time data processing?

  3. What motivates you to specialize in building and maintaining ETL and ELT data pipelines?

  4. What motivates you to specialize in Python for data engineering tasks?

Other behavioral questions

  1. Describe a challenging data modeling problem you solved and what approach you took to address it.

  2. How do you stay updated on best practices and emerging trends in data modeling and warehousing?

  3. Tell me about a challenging Spark batch processing problem you solved and the approach you took.

  4. Describe a challenging problem you faced when building a real-time data pipeline with Kafka and how you resolved it.

  5. Tell me about a time you had to troubleshoot a pipeline failure under tight deadlines—what was your approach?

  6. Describe a situation when you had to troubleshoot scheduling conflicts or missed runs in Airflow and how you resolved it.

  7. Describe a time you identified a critical data quality issue in production and how you resolved it.

  8. How do you stay current with new Python tools and libraries relevant to data engineering?

Show 3 more other behavioral questions
  1. Describe a project where you migrated on-premise data infrastructure to a cloud-based lakehouse platform. What challenges did you face?

  2. Describe a challenge you faced when designing a data platform and how you resolved it.

  3. How do you collaborate with data scientists and analysts while designing a data platform to meet their needs?

All interview questions

How this data is collected

By company

Shaped by a company.

More Data engineer questions, shaped by what each of these companies publicly builds.

Next

When the interview comes.

  • Transcribe

    Live transcription of a call, readable on any phone as it happens.

    Learn more
  • Interview copilot

    Suggested answers during your interview, from your resume and the role.

    Learn more