We also need reproducible results and well-annotated data. In life sciences, there are few to no incentives for making things reproducible. Reproducibility has the potential to expose lots of scams, perhaps even during review stage.
The typical scenario right now is that you get access to raw data, often through an application, that can take months and requires having an ethics committee to oversee your work. So, unless you are at an academic institution, you can't get access to any raw data that contains personal information. Raw data will be often poorly annotated and you will need to spend weeks to figure things out, and sometimes significant computing resources to preprocess it. Then, good luck running some badly coded scripts to get the results reported in the article.
Studies should provide preprocessed data that has been anonymized and can be freely shared, one-click reproducible results from preprocessed data, well-annotated raw data, and simpler procedures to get access to raw data. For example, a tarball with code, preprocessed data, and a makefile to turn preprocessed data into results reported in the article. Bonus points if dependencies are reproducible, with e.g. a Nix flake.
> Then, good luck running some badly coded scripts to get the results reported in the article.
I think even this part is unlikely? My dad was a statistician involved in research and he didn't provide copies of his SAS programs/scripts to authors.
I believe the papers would list what tests he performed but there'd be no easy to way to reproduce the exact script for tests (and certainly no easy way to reproduce exact charts/graphs/maps).
Supercomputing 24 has a reproducibility program involving Chameleon Cloud as infrastructure, and authors are expected to get their code running there for the purpose. It's a massive undertaking but it's looking promising.
My friend does research and while he can provide you with the code, he can't share the census data it executes on. Even the code has to be executed in a census computer in a airgaped lab
The typical scenario right now is that you get access to raw data, often through an application, that can take months and requires having an ethics committee to oversee your work. So, unless you are at an academic institution, you can't get access to any raw data that contains personal information. Raw data will be often poorly annotated and you will need to spend weeks to figure things out, and sometimes significant computing resources to preprocess it. Then, good luck running some badly coded scripts to get the results reported in the article.
Studies should provide preprocessed data that has been anonymized and can be freely shared, one-click reproducible results from preprocessed data, well-annotated raw data, and simpler procedures to get access to raw data. For example, a tarball with code, preprocessed data, and a makefile to turn preprocessed data into results reported in the article. Bonus points if dependencies are reproducible, with e.g. a Nix flake.