Peer Reviews in Data Science: Beyond Code Reviews

Data scientists, through their programming and modeling tasks, strive to build accurate and efficient models that inform decisions based on vast amounts of data. However, the process of reviewing these models and the code that supports them goes beyond traditional code reviews. This article explores how peer reviews are conducted among data scientists, focusing on the unique aspects of their workflow and the importance of reviewing modelling of reality, statistical assumptions, and code robustness and efficiency.

Understanding the Role of Data Scientists

Data scientists program models, but their primary task is not coding in the traditional sense. Instead, they select the most relevant data points to inform and enhance the performance of their models. The process of refining these models involves showing the intended use of the data, explaining why certain choices were made, and demonstrating the positive impact on predetermined performance benchmarks. This requires a deep understanding of both the technical and domain-specific aspects of the project.

The Limitations of Traditional Code Reviews

While code reviews are essential for ensuring the quality and efficiency of the code, they may not always address the core concerns of data science peer reviews. The decisions made by the team should be discussed in terms of the modelling of reality, the statistical assumptions underlying the models, and the robustness and efficiency of the code. These are critical elements that traditional code reviews might overlook.

Utilizing BI Database Professionals for Code Reviews

One approach to addressing these concerns is to involve Business Intelligence (BI) database professionals in code reviews. These individuals, who have programming skills and a strong understanding of data, can provide valuable feedback on the code quality without having strong opinions on the data science aspects. BI professionals are well-suited for this task due to their technical expertise and practical experience with data.

Conducting Effective Peer Reviews

Peer reviews, when conducting for data scientists, focus more on checking the assumptions rather than just the technical aspects of the code. For example, discussions may revolve around the business rules applied to the matching of data, the criteria used to define a customer, or the rationale behind the chosen data points. These elements are crucial for ensuring that the model aligns with the business objectives and the broader context of the project.

In my experience, these type of reviews are far more useful than just code reviews. While developing coding skills is important, it is not the primary objective for data scientists. Instead, the focus should be on refining their understanding of the business context and the assumptions behind the modeling process.

Varying Approaches in Training and Guidance

The approach to training and guidance can vary, depending on the background of the data scientist. For those coming from a market research background, their coding skills may be weaker, and they may require more active training. In such cases, it is beneficial for them to regularly show their code to more experienced members of the team or BI database professionals until they adhere to the team's guidelines.

Effective peer reviews in data science should focus on the broader aspects of decision-making, quality assurance, and alignment with business goals. While code reviews are important, they should be complemented by reviews that address the statistical and domain-specific assumptions that drive the success of the data models.