Review of a poster

Presenting a new AI tool

The SAFE poster is a unique in several ways. First, it was presented during an open lab event at the Lectoraat Betekenisvol Digitaal Innoveren of the Hogeschool Utrecht, The Netherlands. This lab works on a number of AI projects so most of those attending the event would have already been familiar with this work. The specific project referenced here is ASReview (

https://asreview.nl/

), an AI tool for screening a large number of scientific abstracts. A researcher passes the titles and abstracts to the AI, which finds, with user feedback, the most relevant. The problem addressed in the poster has to do with the screening process for papers passed to the AI. Some of these will be relevant to the researcher, others will not. How can the AI most effectively find the most relevant ones? The poster describes a method, that has since been incorporated into the software.

Given the way the poster was displayed, some of our usual evaluation criteria will be less valid. However, I will first review the poster as if it were viewed without the author present, and then return to comment on how it was actually used as part of an open lab event.

While the title is fine for an open lab, no one but the author will understand the meaning of SAFE. One has to read most of the poster to know that this acronym stands for Screen Random Set - Active Learning - Find More Using Deep Learning - Evaluate Quality. Even after viewing the central graphic one has to think carefully to understand the meaning of this acronym. It is certainly clever, but it does not contribute much to the understanding of the procedure. The title would be easier to understand if the first phrase were removed. A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses provides a clear summary of the main result of this research.

The layout of the poster is excellent. Both the overall layout and the use of cute and informative graphics make it easier for the reader to follow the research. The horizontal rows clearly indicate the direction of information flow. The viewer is encouraged to read from left to right within each row, and from top to bottom between rows. The vertical layout of the poster also encourages this reading frame. The top row below the title introduces the research problem and provides a high-level summary of the results. If this poster had been presented at a national meeting, I would have liked more background on the state of the art before introducing the SAFE procedure. However, for an open lab where most visitors will be familiar with the software, the current description is sufficient.

Some of the background information about the problem is provided in the second row, where three cartoons explain some of the problems with existing procedures. Each problem is set off in its own bubble, which clearly indicates their independence. However, even these cartoons assume some knowledge of ASReview.

The graphical explanation of SAFE in the third row makes excellent use of informative drawings to explain the process. Both the main drawings and the stacks of papers clearly illustrate the process. Human decisions and human–computer interactions are clearly illustrated by the drawings. Unfortunately, the meaning of some of the smaller text is not intuitively obvious. A deep understanding of ASReview is necessary to understand tags like “Native Bayes TF-IDF” and “CNN Sbert.” These abbreviations will have meaning to the lab members, but it is unlikely that anyone visiting the lab will understand them without further explanations.

The final row before the references contains a written explanation of the SAFE process and clarifies the meaning of the drawings that appear immediately above. This is an excellent addition, as it removes any uncertainty that might’ve been created by the drawings. The graph on the right is a nice indication of the duration of the screening process, however without an indication of the absolute duration of at least one example it’s unclear if the SAFE process produces results in a reasonable amount of time. If the time to complete the screening process is measured in weeks, then SAFE is probably useless. If it’s measured in seconds, then it is amazingly efficient. The truth is probably somewhere in between, but it’s impossible to tell from this graph.

I have mentioned the drawings in icons several times. Let me return to these as it is rare to find such clear and informative drawings used to illustrate complex concepts. The drawings are simple but effective. Their simplicity does not detract from the usefulness. This should encourage others, like me, who have little artistic ability, to experiment with the use of simple drawings to illustrate concepts.

So far, I have been reviewing the poster as if the author was not present to answer questions. Of course, this was not the case. The author was present during the open lab and could answer questions about her poster. This mitigates almost all of my criticisms of the poster. The poster is an excellent backdrop for a personal presentation. There is enough information to allow the author to explain her work and explain its significance, without overwhelming the viewer. Even the seemingly inexplicable image tags become comprehensible with the authors explanations.

This is an excellent poster for its setting. With the addition of an author’s presentation, it clearly presents the problem and the solution. Both the layout of the poster and the drawings attract the viewer’s attention and will increase interest in this research.

Excellent job!

Boetje, J., and R. van de Schoot (2023). The SAFE procedure: a Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses. Figshare https://doi.org/10.6084/m9.figshare.23522733.v1