Dataset Development Lifecycle

Dataset Development Lifecycle

Those working in data analysis and machine learning have all probably needed to collect and create data at some point. Here’s a paper from Google that provides a structured framework for data collection inspired by software development concepts in a 5-step cyclical process. The stages are requirements analysis, design, implementation, testing, and maintenance.

Requirements analysis: In this stage, we determine the required data by deliberating about the intentions of the project, consulting with the stakeholders, and analyzing use cases.

Design: This is where we find out whether data requirements can be met, and if so what is the most optimal way to do it by conducting research about the subject matter and consulting the experts of the field.

Implementation: Design decisions are transformed into technologies such as software systems, annotator guidelines, and labeling platforms.

Testing: Data is evaluated and decisions about whether or not to use it are made.

Maintenance: Once collected, a dataset requires a large set of affordances, including tools, policies, and designated owners.

One noteworthy aspect of Google’s approach is its emphasis on producing artifacts at each stage. This means that a document must be prepared at each stage based on provided templates, which is considered the output of that stage. According to the paper, there are critical document types for accountable dataset development. Each one is directly analogous to documentation types produced by the Software Development Lifecycle.

The paper mentioned three of Nissenbaum’s barriers to accountability and specific data concerns and also prepared a proposal to mitigate these barriers which can be found in the figure below.