The aim of the International Conference on Agents and Artificial Intelligence is to bring together researchers, engineers and practitioners interested in the theory and applications in the field of Agents and Artificial Intelligence. The conference schedule consists of different types of sessions such as technical sessions, poster sessions, keynote lectures, tutorials and panels.
In February, our Data Scientist Victor Margallo presented his paper “Understanding Summaries: Modelling Evaluation in Extractive Summarisation Techniques” at the ICAART 2022.
The conference offered new insights and research on recent and upcoming trends such as knowledge graph modelling or robotic agents. The spectrum was very diverse, covering most topics tackled in Data Science and showing how broad the field can get. There was also a good balance between theoretical and applied papers, creating a middle ground for a successful interchange between academia and companies. All in all, the participation in ICAART was a great experience to meet researchers from different areas, get the latest work on interesting topics and present some of the research we do in PublicSonar!
Understanding Summaries: Modelling Evaluation in Extractive Summarisation Techniques
In the task of providing extracted summaries, the assessment of quality evaluation has been traditionally tackled with n-gram, word sequences, and word pairs overlapping metrics with human annotated summaries for theoretical benchmarking. This approach does not provide an end solution for extractive summarising algorithms as output summaries are not evaluated for new texts. Our solution proposes the expansion of a graph extraction method together with an understanding layer before delivering the final summary. With this technique we strive to achieve a categorisation of acceptable output summaries. Our understanding layer judges correct summaries with 91% accuracy and is in line with experts’ labelling providing a strong inter-rater reliability (0.73 Kappa statistic).
NLP, Extractive Summarisation, Evaluation, Summary Quality Modelling.