publications

Bibtex file with the publications listed below.

2022

1. Relational graph convolutional networks: a closer look Thanapalasingam, Thiviyan, Berkel, Lucas, Bloem, Peter, and Groth, Paul PeerJ Computer Science 2022 [Link] [DOI:10.7717/peerj-cs.1073]
2. Question Answering with Additive Restrictive Training (QuAART): Question Answering for the Rapid Development of New Knowledge Extraction Pipelines Harper, Corey A., Daniel, Ron, and Groth, Paul In Knowledge Engineering and Knowledge Management (EKAW) 2022 [Abs] [Link] [DOI:10.1007/978-3-031-17105-5_4]

   Abstract Numerous studies have explored the use of language models and question answering techniques for knowledge extraction. In most cases, these models are trained on data specific to the new task at hand. We hypothesize that using models trained only on generic question answering data (e.g. SQuAD) is a good starting point for domain specific entity extraction. We test this hypothesis, and explore whether the addition of small amounts of training data can help lift model performance. We pay special attention to the use of null answers and unanswerable questions to optimize performance. To our knowledge, no studies have been done to evaluate the effectiveness of this technique. We do so for an end-to-end entity mention detection and entity typing task on HAnDS and FIGER, two common evaluation datasets for fine grained entity recognition. We focus on fine-grained entity recognition because it is challenging scenario, and because the long tail of types in this task highlights the need for entity extraction systems that can deal with new domains and types. To our knowledge, we are the first system beyond those presented in the original FIGER and HAnDS papers to tackle the task in an end-to-end fashion. Using an extremely small sample from the distantly-supervised HAnDS training data – 0.0015%, or less than 500 passages randomly chosen out of 31 million – we produce a CoNNL F1 score of 73.72 for entity detection on FIGER. Our end-to-end detection and typing evaluation produces macro and micro F1s of 45.11 and 54.75, based on the FIGER evaluation metrics. This work provides a foundation for the rapid development of new knowledge extraction pipelines.

3. Serenade - Low-Latency Session-Based Recommendation in e-Commerce at Scale Kersbergen, Barrie, Sprangers, Olivier, and Schelter, Sebastian In Proceedings of the 2022 International Conference on Management of Data 2022 [Abs] [Link] [DOI:10.1145/3514221.3517901]

   Session-based recommendation predicts the next item with which a user will interact, given a sequence of her past interactions with other items. This machine learning problem targets a core scenario in e-commerce platforms, which aim to recommend interesting items to buy to users browsing the site. Session-based recommenders are difficult to scale due to their exponentially large input space of potential sessions. This impedes offline precomputation of the recommendations, and implies the necessity to maintain state during the online computation of next-item recommendations.We propose VMIS-kNN, an adaptation of a state-of-the-art nearest neighbor approach to session-based recommendation, which leverages a prebuilt index to compute next-item recommendations with low latency in scenarios with hundreds of millions of clicks to search through. Based on this approach, we design and implement the scalable session-based recommender system Serenade, which is in production usage at bol.com, a large European e-commerce platform.We evaluate the predictive performance of VMIS-kNN, and show that Serenade can answer a thousand recommendation requests per second with a 90th percentile latency of less than seven milliseconds in scenarios with millions of items to recommend. Furthermore, we present results from a three week long online A/B test with up to 600 requests per second for 6.5 million distinct items on more than 45 million user sessions from our e-commerce platform. To the best of our knowledge, we provide the first empirical evidence that the superior predictive performance of nearest neighbor approaches to session-based recommendation in offline evaluations translates to superior performance in a real world e-commerce setting.

4. Towards Data-Centric What-If Analysis for Native Machine Learning Pipelines Grafberger, Stefan, Groth, Paul, and Schelter, Sebastian In Proceedings of the Sixth Workshop on Data Management for End-To-End Machine Learning 2022 [Abs] [Link] [DOI:10.1145/3533028.3533303]

   An important task of data scientists is to understand the sensitivity of their models to changes in the data that the models are trained and tested upon. Currently, conducting such data-centric what-if analyses requires significant and costly manual development and testing with the corresponding chance for the introduction of bugs. We discuss the problem of data-centric what-if analysis over whole ML pipelines (including data preparation and feature encoding), propose optimisations that reuse trained models and intermediate data to reduce the runtime of such analysis, and finally conduct preliminary experiments on three complex example pipelines, where our approach reduces the runtime by a factor of up to six.

5. Responsible Data Management Stoyanovich, Julia, Abiteboul, Serge, Howe, Bill, Jagadish, H. V., and Schelter, Sebastian Communications of the ACM 2022 [Abs] [Link] [DOI:10.1145/3488717]

   Perspectives on the role and responsibility of the data-management research community in designing, developing, using, and overseeing automated decision systems.

6. Methods Included Crusoe, Michael R., Abeln, Sanne, Iosup, Alexandru, Amstutz, Peter, Chilton, John, Tijanić, Nebojša, Ménager, Hervé, Soiland-Reyes, Stian, Gavrilović, Bogdan, Goble, Carole, and The CWL Community, Communications of the ACM 2022 [Abs] [Link] [DOI:10.1145/3486897]

   Standardizing computational reuse and portability with the Common Workflow Language.

7. CITRIS: Causal Identifiability from Temporal Intervened Sequences Lippe, Phillip, Magliacane, Sara, Löwe, Sindy, Asano, Yuki M., Cohen, Taco, and Gavves, Efstratios In Proceedings of the 39th International Conference on Machine Learning, ICML 2022 [arXiv]
8. SlotGAN: Detecting Mentions in Text via Adversarial Distant Learning Daza, Daniel, Cochez, Michael, and Groth, Paul In Proceedings of the Sixth Workshop on Structured Prediction for NLP 2022 [Abs] [Link] [DOI:10.18653/v1/2022.spnlp-1.4]

   We present SlotGAN, a framework for training a mention detection model that only requires unlabeled text and a gazetteer. It consists of a generator trained to extract spans from an input sentence, and a discriminator trained to determine whether a span comes from the generator, or from the gazetteer.We evaluate the method on English newswire data and compare it against supervised, weakly-supervised, and unsupervised methods. We find that the performance of the method is lower than these baselines, because it tends to generate more and longer spans, and in some cases it relies only on capitalization. In other cases, it generates spans that are valid but differ from the benchmark. When evaluated with metrics based on overlap, we find that SlotGAN performs within 95% of the precision of a supervised method, and 84% of its recall. Our results suggest that the model can generate spans that overlap well, but an additional filtering mechanism is required.

9. Making Canonical Workflow Building Blocks Interoperable across Workflow Languages Soiland-Reyes, Stian, Bayarri, Genís, Andrio, Pau, Long, Robin, Lowe, Douglas, Niewielska, Ania, Hospital, Adam, and Groth, Paul Data Intelligence 2022 [Abs] [Link] [DOI:10.1162/dint_a_00135]

   We introduce the concept of Canonical Workflow Building Blocks (CWBB), a methodology of describing and wrapping computational tools, in order for them to be utilised in a reproducible manner from multiple workflow languages and execution platforms. The concept is implemented and demonstrated with the BioExcel Building Blocks library (BioBB), a collection of tool wrappers in the field of computational biomolecular simulation. Interoperability across different workflow languages is showcased through a protein Molecular Dynamics setup transversal workflow, built using this library and run with 5 different Workflow Manager Systems (WfMS). We argue such practice is a necessary requirement for FAIR Computational Workflows and an element of Canonical Workflow Frameworks for Research (CWFR) in order to improve widespread adoption and reuse of computational methods across workflow language barriers.

10. Letter from the Special Issue Editor Schelter, Sebastian IEEE Data Engineering Bulletin (Special issue on Directions Towards GDPR-Compliant Data Systems and Applications) 2022 [Link]
11. Defining a Knowledge Graph Development Process Through a Systematic Review Tamašauskaitundefined, Gytundefined, and Groth, Paul ACM Transactios on Software Engineering and Methodology 2022 [Abs] [Link] [DOI:10.1145/3522586]

    Knowledge graphs are widely used in industry and studied within the academic community. However, the models applied in the development of knowledge graphs vary. Analysing and providing a synthesis of the commonly used approaches to knowledge graph development would provide researchers and practitioners a better understanding of the overall process and methods involved. Hence, this paper aims to define the overall process of knowledge graph development and its key constituent steps. For this purpose, a systematic review and a conceptual analysis of the literature was conducted. The resulting process was compared to case studies to evaluate its applicability. The proposed process suggests a unified approach and provides guidance for both researchers and practitioners when constructing and managing knowledge graphs.

12. Data distribution debugging in machine learning pipelines Grafberger, Stefan, Groth, Paul, Stoyanovich, Julia, and Schelter, Sebastian The VLDB Journal 2022 [Link] [DOI:10.1007/s00778-021-00726-w]
13. Structure-based knowledge acquisition from electronic lab notebooks for research data provenance documentation Schröder, Max, Staehlke, Susanne, Groth, Paul, Nebe, J. Barbara, Spors, Sascha, and Krüger, Frank Journal of Biomedical Semantics 2022 [Link] [DOI:10.1186/s13326-021-00257-x]
14. Packaging research artefacts with RO-Crate Soiland-Reyes, Stian, Sefton, Peter, Crosas, Mercè, Castro, Leyla Jael, Coppens, Frederik, Fernández, José M., Garijo, Daniel, Grüning, Björn, La Rosa, Marco, Leo, Simone, and al., Data Science 2022 [Link] [DOI:10.3233/DS-210053]
15. The Semantic Web - 19th International Conference, ESWC 2022, Hersonissos, Crete, Greece, May 29 - June 2, 2022, Proceedings 2022 [Link] [DOI:10.1007/978-3-031-06981-9]
16. GitSchemas: A Dataset for Automating Relational Data Preparation Tasks Döhmen, Till, Hulsebos, Madelon, Beecks, Christian, and Schelter, Sebastian In 2022 IEEE 38th International Conference on Data Engineering Workshops (ICDEW) 2022 [Link] [DOI:10.1109/ICDEW55742.2022.00016]
17. Towards improving Wikidata reuse with emerging patterns Carriero, Valentina Anita, Groth, Paul, and Presutti, Valentina In Proceedings of the 3rd Wikidata Workshop 2022 co-located with the 21st International Semantic Web Conference (ISWC2022), Virtual Event, Hanghzou, China, October 2022 2022 [Link]
18. Proceedings of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching co-located with the 20th International Semantic Web Conference (ISWC 2021), Virtual conference, October 27, 2021 Jiménez-Ruiz, Ernesto, Efthymiou, Vasilis, Chen, Jiaoyan, Cutrona, Vincenzo, Hassanzadeh, Oktie, Sequeda, Juan, Srinivas, Kavitha, Abdelmageed, Nora, Hulsebos, Madelon, Oliveira, Daniela, and Pesquita, Catia 2022 [Link]
19. AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning Huang, Biwei, Feng, Fan, Lu, Chaochao, Magliacane, Sara, and Zhang, Kun In International Conference on Learning Representations 2022 [Link] [ Spotlight Presentation ]
20. Making Table Understanding Work in Practice Hulsebos, Madelon, Gathani, Sneha, Gale, James, Dillig, Isil, Groth, Paul, and Demiralp, In 12th Conference on Innovative Data Systems Research, CIDR 2022, Chaminade, CA, USA, January 9-12, 2022 2022 [Link]
21. Screening Native Machine Learning Pipelines with ArgusEyes Schelter, Sebastian, Grafberger, Stefan, Guha, Shubha, Sprangers, Olivier, Karlas, Bojan, and Zhang, Ce In 12th Conference on Innovative Data Systems Research, CIDR 2022, Chaminade, CA, USA, January 9-12, 2022 2022 [Link]

2021

1. The non-linear impact of data handling on network diffusion models Nevin, James, Lees, Michael, and Groth, Paul Patterns 2021 [Link] [DOI:10.1016/j.patter.2021.100397]
2. GraphPOPE: Retaining Structural Graph Information Using Position-aware Node Embeddings Boef, Jeroen Den, Cornelisse, Joran, and Groth, Paul In Proceedings of the Workshop on Deep Learning for Knowledge Graphs (DL4KG 2021) 2021 [Link]
3. Quality Assessment of Knowledge Graph Hierarchies using KG-BERT Szarkowska, Kinga, Moore, Veronique, Vandenbussche, Pierre-Yves, and Groth, Paul In Proceedings of the Workshop on Deep Learning for Knowledge Graphs (DL4KG 2021) 2021 [Link]
4. Perspectives on automated composition of workflows in the life sciences Lamprecht, Anna-Lena, Palmblad, Magnus, Ison, Jon, Schwämmle, Veit, Manir, Mohammad Sadnan Al, Altintas, Ilkay, Baker, Christopher J. O., Amor, Ammar Ben Hadj, Capella-Gutierrez, Salvador, Charonyktakis, Paulos, Crusoe, Michael R., Gil, Yolanda, Goble, Carole, Griffin, Timothy J., Groth, Paul, Ienasescu, Hans, Jagtap, Pratik, Kalaš, Matúš, Kasalica, Vedran, Khanteymoori, Alireza, Kuhn, Tobias, Mei, Hailiang, Ménager, Hervé, Möller, Steffen, Richardson, Robin A., Robert, Vincent, Soiland-Reyes, Stian, Stevens, Robert, Szaniszlo, Szoke, Verberne, Suzan, Verhoeven, Aswin, and Wolstencroft, Katherine F1000Research 2021 [Link] [DOI:10.12688/f1000research.54159.1]
5. SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts Harper, Corey, Cox, Jessica, Kohler, Curt, Scerri, Antony, Daniel Jr., Ron, and Groth, Paul In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) 2021 [Abs] [Link] [DOI:10.18653/v1/2021.semeval-1.38] [ SemEval 2021 Best Task Paper ]

   We describe MeasEval, a SemEval task of extracting counts, measurements, and related context from scientific documents, which is of significant importance to the creation of Knowledge Graphs that distill information from the scientific literature. This is a new task in 2021, for which over 75 submissions from 25 participants were received. We expect the data developed for this task and the findings reported to be valuable to the scientific knowledge extraction, metrology, and automated knowledge base construction communities.

6. Further with Knowledge Graphs: Proceedings of the 17th International Conference on Semantic Systems, 6–9 September 2021, Amsterdam, The Netherlands 2021 [Link] [DOI:10.3233/SSW53]
7. Reinforcement Learning–Based Collective Entity Alignment with Adaptive Features Zeng, Weixin, Zhao, Xiang, Tang, Jiuyang, Lin, Xuemin, and Groth, Paul ACM Trans. Inf. Syst. 2021 [Abs] [Link] [DOI:10.1145/3446428]

   Entity alignment (EA) is the task of identifying the entities that refer to the same real-world object but are located in different knowledge graphs (KGs). For entities to be aligned, existing EA solutions treat them separately and generate alignment results as ranked lists of entities on the other side. Nevertheless, this decision-making paradigm fails to take into account the interdependence among entities. Although some recent efforts mitigate this issue by imposing the 1-to-1 constraint on the alignment process, they still cannot adequately model the underlying interdependence and the results tend to be sub-optimal.To fill in this gap, in this work, we delve into the dynamics of the decision-making process, and offer a reinforcement learning (RL)–based model to align entities collectively. Under the RL framework, we devise the coherence and exclusiveness constraints to characterize the interdependence and restrict collective alignment. Additionally, to generate more precise inputs to the RL framework, we employ representative features to capture different aspects of the similarity between entities in heterogeneous KGs, which are integrated by an adaptive feature fusion strategy. Our proposal is evaluated on both cross-lingual and mono-lingual EA benchmarks and compared against state-of-the-art solutions. The empirical results verify its effectiveness and superiority.

8. Learnings from a Retail Recommendation System on Billions of Interactions at bol.com Kersbergen, B., and Schelter, S. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) 2021 [Link] [DOI:10.1109/ICDE51399.2021.00277]
9. Inductive Entity Representations from Text via Link Prediction Daza, Daniel, Cochez, Michael, and Groth, Paul In Proceedings of The Web Conference 2021 [arXiv] [DOI:10.1145/3442381.3450141] [Code]
10. Letter from the Special Issue Editor Schelter, Sebastian IEEE Data Engineering Bulletin (Special issue on Data validation for machine learning models and applications) 2021 [Link]
11. Complex Query Answering with Neural Link Predictors Arakelyan, Erik, Daza, Daniel, Minervini, Pasquale, and Cochez, Michael In International Conference on Learning Representations (ICLR) 2021 [arXiv] [Link] [ Outstanding Paper Award ICLR 2021 ]
12. Taming Technical Bias in Machine Learning Pipelines Schelter, Sebastian, and Stoyanovich, Julia IEEE Data Engineering Bulletin (Special Issue on Interdisciplinary Perspectives on Fairness and Artificial Intelligence Systems) 2021 [Link]
13. Talking datasets – Understanding data sensemaking behaviours Koesten, Laura, Gregory, Kathleen, Groth, Paul, and Simperl, Elena International Journal of Human-Computer Studies 2021 [Abs] [arXiv] [Link] [DOI:10.1016/j.ijhcs.2020.102562]

    The sharing and reuse of data are seen as critical to solving the most complex problems of today. Despite this potential, relatively little attention has been paid to a key step in data reuse: the behaviours involved in data-centric sensemaking. We aim to address this gap by presenting a mixed-methods study combining in-depth interviews, a think-aloud task and a screen recording analysis with 31 researchers from different disciplines as they summarised and interacted with both familiar and unfamiliar data. We use our findings to identify and detail common patterns of data-centric sensemaking across three clusters of activities that we present as a framework: inspecting data, engaging with content, and placing data within broader contexts. Additionally, we propose design recommendations for tools and documentation practices, which can be used to facilitate sensemaking and subsequent data reuse.

14. The Challenges of Cross-Document Coreference Resolution for Email Li, Xue, Magliacane, Sara, and Groth, Paul In Proceedings of the 11th on Knowledge Capture Conference 2021 [Abs] [Link] [DOI:10.1145/3460210.3493573]

    Long-form conversations such as email are an important source of information for knowledge capture. For tasks such as knowledge graph construction, conversational search, and entity linking, being able to resolve entities from across documents is important. Building on recent work on within document coreference resolution for email, we study for the first time a cross-document formulation of the problem. Our results show that the current state-of-the-art deep learning models for general cross-document coreference resolution are insufficient for email conversations. Our experiments show that the general task is challenging and, importantly for knowledge intensive tasks, coreference resolution models that only treat entity mentions perform worse. Based on these results, we outline the work needed to address this challenging task.

15. Supporting Ontology Maintenance with Contextual Word Embeddings and Maximum Mean Discrepancy Shroff, Natasha, Vandenbussche, Pierre-Yves, Moore, Véronique, and Groth, Paul In Joint Proceedings of the 2nd International Workshop on Deep Learning meets Ontologies and Natural Language Processing (DeepOntoNLP 2021) & 6th International Workshop on Explainable Sentiment Mining and Emotion Detection (X-SENTIMENT 2021) co-located with co-located with 18th Extended Semantic Web Conference 2021, Hersonissos, Greece, June 6th - 7th, 2021 (moved online) 2021 [Link]
16. Proceedings of Machine Learning with Symbolic Methods and Knowledge Graphs co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2021), Virtual, September 17, 2021 Alam, Mehwish, Ali, Mehdi, Groth, Paul, Hitzler, Pascal, Lehmann, Jens, Paulheim, Heiko, Rettinger, Achim, Sack, Harald, Sadeghi, Afshin, and Tresp, Volker 2021 [Link]
17. Verifiably Safe Exploration for End-to-End Reinforcement Learning Hunt, Nathan, Fulton, Nathan, Magliacane, Sara, Hoang, Trong Nghia, Das, Subhro, and Solar-Lezama, Armando In Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control 2021 [Abs] [Link] [DOI:10.1145/3447928.3456653] [ Best Paper Award ACM HSCC 2021 ]

    Deploying deep reinforcement learning in safety-critical settings requires developing algorithms that obey hard constraints during exploration. This paper contributes a first approach toward enforcing formal safety constraints on end-to-end policies with visual inputs. Our approach draws on recent advances in object detection and automated reasoning for hybrid dynamical systems. The approach is evaluated on a novel benchmark that emphasizes the challenge of safely exploring in the presence of hard constraints. Our benchmark draws from several proposed problem sets for safe learning and includes problems that emphasize challenges such as reward signals that are not aligned with safety constraints. On each of these benchmark problems, our algorithm completely avoids unsafe behavior while remaining competitive at optimizing for as much reward as is safe. We characterize safety constraints in terms of a refinement relation on Markov decision processes - rather than directly constraining the reinforcement learning algorithm so that it only takes safe actions, we instead refine the environment so that only safe actions are defined in the environment’s transition structure. This has pragmatic system design benefits and, more importantly, provides a clean conceptual setting in which we are able to prove important safety and efficiency properties. These allow us to transform the constrained optimization problem of acting safely in the original environment into an unconstrained optimization in a refined environment.

18. Summary of Tutorials at The Web Conference 2021 West, Robert, Bhagat, Smriti, Groth, Paul, Zitnik, Marinka, Couto, Francisco M., Lisena, Pasquale, Meroño-Peñuela, Albert, Zhao, Xiangyu, Fan, Wenqi, Yin, Dawei, Tang, Jiliang, Shou, Linjun, Gong, Ming, Pei, Jian, Geng, Xiubo, Zhou, Xingjie, Jiang, Daxin, Ricaud, Benjamin, Aspert, Nicolas, Miz, Volodymyr, Dy, Jennifer, Ioannidis, Stratis, Yıldız, undefinedlkay, Rezapour, Rezvaneh, Aref, Samin, Dinh, Ly, Diesner, Jana, Drutsa, Alexey, Ustalov, Dmitry, Popov, Nikita, Baidakova, Daria, Mishra, Shubhanshu, Gopalan, Arjun, Juan, Da-Cheng, Ilharco Magalhaes, Cesar, Ferng, Chun-Sung, Heydon, Allan, Lu, Chun-Ta, Pham, Philip, Yu, George, Fan, Yicheng, Wang, Yueqi, Laurent, Florian, Schraner, Yanick, Scheller, Christian, Mohanty, Sharada, Chen, Jiawei, Wang, Xiang, Feng, Fuli, He, Xiangnan, Teinemaa, Irene, Albert, Javier, Goldenberg, Dmitri, Vasile, Flavian, Rohde, David, Jeunen, Olivier, Benhalloum, Amine, Sakhi, Otmane, Rong, Yu, Huang, Wenbing, Xu, Tingyang, Bian, Yatao, Cheng, Hong, Sun, Fuchun, Huang, Junzhou, Fakhraei, Shobeir, Faloutsos, Christos, Çelebi, Onur, Müller, Martin, Schneider, Manuel, Altunina, Olesia, Wingerath, Wolfram, Wollmer, Benjamin, Gessert, Felix, Succo, Stephan, Ritter, Norbert, Courdier, Evann, Avram, Tudor Mihai, Cvetinovic, Dragan, Tsinadze, Levan, Jose, Johny, Howell, Rose, Koenig, Mario, Defferrard, Michaël, Kenthapadi, Krishnaram, Packer, Ben, Sameki, Mehrnoosh, and Sephus, Nashlie In Companion Proceedings of the Web Conference 2021 2021 [Abs] [Link] [DOI:10.1145/3442442.3453701]

    This report summarizes the 23 tutorials hosted at The Web Conference 2021: nine lecture-style tutorials and 14 hands-on tutorials.

19. HedgeCut: Maintaining Randomised Trees for Low-Latency Machine Unlearning Schelter, Sebastian, Grafberger, Stefan, and Dunning, Ted In Proceedings of the 2021 International Conference on Management of Data 2021 [Abs] [Link] [DOI:10.1145/3448016.3457239]

    Software systems that learn from user data with machine learning (ML) have become ubiquitous over the last years. Recent law such as the "General Data Protection Regulation" (GDPR) requires organisations that process personal data to delete user data upon request (enacting the "right to be forgotten"). However, this regulation does not only require the deletion of user data from databases, but also applies to ML models that have been learned from the stored data. We therefore argue that ML applications should offer users to unlearn their data from trained models in a timely manner. We explore how fast this unlearning can be done under the constraints imposed by real world deployments, and introduce the problem of low-latency machine unlearning: maintaining a deployed ML model in-place under the removal of a small fraction of training samples without retraining.We propose HedgeCut, a classification model based on an ensemble of randomised decision trees, which is designed to answer unlearning requests with low latency. We detail how to efficiently implement HedgeCut with vectorised operators for decision tree learning. We conduct an experimental evaluation on five privacy-sensitive datasets, where we find that HedgeCut can unlearn training samples with a latency of around 100 microseconds and answers up to 36,000 prediction requests per second, while providing a training time and predictive accuracy similar to widely used implementations of tree-based ML models such as Random Forests.

20. MLINSPECT: A Data Distribution Debugger for Machine Learning Pipelines Grafberger, Stefan, Guha, Shubha, Stoyanovich, Julia, and Schelter, Sebastian In Proceedings of the 2021 International Conference on Management of Data 2021 [Abs] [Link] [DOI:10.1145/3448016.3452759]

    Machine Learning (ML) is increasingly used to automate impactful decisions, and the risks arising from this wide-spread use are garnering attention from policymakers, scientists, and the media. ML applications are often very brittle with respect to their input data, which leads to concerns about their reliability, accountability, and fairness. While bias detection cannot be fully automated, computational tools can help pinpoint particular types of data issues.We recently proposed mlinspect, a library that enables lightweight lineage-based inspection of ML preprocessing pipelines. In this demonstration, we show how mlinspect can be used to detect data distribution bugs in a representative pipeline. In contrast to existing work, mlinspect operates on declarative abstractions of popular data science libraries like estimator/transformer pipelines, can handle both relational and matrix data, and does not require manual code instrumentation. The library is publicly available at https://github.com/stefan-grafberger/mlinspect.

21. Automating Data Quality Validation for Dynamic Data Ingestion Redyuk, Sergey, Kaoudi, Zoi, Markl, Volker, and Schelter, Sebastian In Proceedings of the 24th International Conference on Extending Database Technology, EDBT 2021, Nicosia, Cyprus, March 23 - 26, 2021 2021 [Link] [DOI:10.5441/002/edbt.2021.07]
22. JENGA - A Framework to Study the Impact of Data Errors on the Predictions of Machine Learning Models Schelter, Sebastian, Rukat, Tammo, and Biessmann, Felix In Proceedings of the 24th International Conference on Extending Database Technology, EDBT 2021, Nicosia, Cyprus, March 23 - 26, 2021 2021 [Link] [DOI:10.5441/002/edbt.2021.63]
23. Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines Grafberger, Stefan, Stoyanovich, Julia, and Schelter, Sebastian In 11th Conference on Innovative Data Systems Research, CIDR 2021, Virtual Event, January 11-15, 2021, Online Proceedings 2021 [Link]

2020

1. Towards Olfactory Information Extraction from Text: A Case Study on Detecting Smell Experiences in Novels Brate, Ryan, Groth, Paul, and Erp, Marieke In Proceedings of the The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature 2020 [Abs] [Link]

   Environmental factors determine the smells we perceive, but societal factors factors shape the importance, sentiment and biases we give to them. Descriptions of smells in text, or as we call them ‘smell experiences’, offer a window into these factors, but they must first be identified. To the best of our knowledge, no tool exists to extract references to smell experiences from text. In this paper, we present two variations on a semi-supervised approach to identify smell experiences in English literature. The combined set of patterns from both implementations offer significantly better performance than a keyword-based baseline.

2. Dataset Reuse: Toward Translating Principles to Practice Koesten, Laura, Vougiouklis, Pavlos, Simperl, Elena, and Groth, Paul Patterns 2020 [Link] [DOI:10.1016/j.patter.2020.100136]
3. Effective distributed representations for academic expert search Berger, Mark, Zavrel, Jakub, and Groth, Paul In Proceedings of the First Workshop on Scholarly Document Processing at EMNLP 2020 [Abs] [Link]

   Expert search aims to find and rank experts based on a user’s query. In academia, retrieving experts is an efficient way to navigate through a large amount of academic knowledge. Here, we study how different distributed representations of academic papers (i.e. embeddings) impact academic expert retrieval. We use the Microsoft Academic Graph dataset and experiment with different configurations of a document-centric voting model for retrieval. In particular, we explore the impact of the use of contextualized embeddings on search performance. We also present results for paper embeddings that incorporate citation information through retrofitting. Additionally, experiments are conducted using different techniques for assigning author weights based on author order. We observe that using contextual embeddings produced by a transformer model trained for sentence similarity tasks produces the most effective paper representations for document-centric expert retrieval. However, retrofitting the paper embeddings and using elaborate author contribution weighting strategies did not improve retrieval performance.

4. Dataset search: a survey Chapman, Adriane, Simperl, Elena, Koesten, Laura, Konstantinidis, George, Ibáñez, Luis-Daniel, Kacprzak, Emilia, and Groth, Paul The VLDB Journal 2020 [arXiv] [Link] [DOI:10.1007/s00778-019-00564-x]
5. Introduction – FAIR data, systems and analysis Groth, Paul, and Dumontier, Michel Data Science 2020 [Link] [DOI:10.3233/DS-200029]
6. Fairness-Aware Instrumentation of Preprocessing Pipelines for Machine Learning Yang, Ke, Huang, Biao, Stoyanovich, Julia, and Schelter, Sebastian In Workshop on Human-In-the-Loop Data Analytics (HILDA’20) 2020 [Link] [DOI:10.1145/3398730.3399194]
7. Towards Entity Spaces Erp, Marieke, and Groth, Paul In Proceedings of The 12th Language Resources and Evaluation Conference 2020 [Abs] [Link]

   Entities are a central element of knowledge bases and are important input to many knowledge-centric tasks including text analysis. For example, they allow us to find documents relevant to a specific entity irrespective of the underlying syntactic expression within a document. However, the entities that are commonly represented in knowledge bases are often a simplification of what is truly being referred to in text. For example, in a knowledge base, we may have an entity for Germany as a country but not for the more fuzzy concept of Germany that covers notions of German Population, German Drivers, and the German Government. Inspired by recent advances in contextual word embeddings, we introduce the concept of entity spaces - specific representations of a set of associated entities with near-identity. Thus, these entity spaces provide a handle to an amorphous grouping of entities. We developed a proof-of-concept for English showing how, through the introduction of entity spaces in the form of disambiguation pages, the recall of entity linking can be improved.

8. Lost or Found? Discovering Data Needed for Research Gregory, Kathleen, Groth, Paul, Scharnhorst, Andrea, and Wyatt, Sally Harvard Data Science Review 2020 [Link] [DOI:10.1162/99608f92.e38165eb]
9. PANDAcap: A Framework for Streamlining Collection of Full-System Traces Stamatogiannakis, Manolis, Bos, Herbert, and Groth, Paul In EuroSec 2020 [Link] [DOI:10.1145/3380786.3391396] [Code]
10. Estimating the imageability of words by mining visual characteristics from crawled image data Kastner, Marc A., Ide, Ichiro, Nack, Frank, Kawanishi, Yasutomo, Hirayama, Takatsugu, Deguchi, Daisuke, and Murase, Hiroshi Multimedia Tools and Applications 2020 [Link] [DOI:10.1007/s11042-019-08571-4]
11. FAIR Data Reuse – the Path through Data Citation Groth, Paul, Cousijn, Helena, Clark, Tim, and Goble, Carole Data Intelligence 2020 [Link] [DOI:10.1162/dint_a_00030]
12. Message Passing Query Embedding Daza, Daniel, and Cochez, Michael In ICML Workshop - Graph Representation Learning and Beyond 2020 [arXiv] [Link]
13. The state of altmetrics: a tenth anniversary celebration Altmetric Engineering, , Konkiel, Stacy, Priem, Jason, Adie, Euan, Derrick, Gemma, Didegah, Fereshteh, Groth, Paul, Neylon, Cameron, Shenmeng Xu, , Zahedi, Zohreh, Bowman, Timothy, Vanash M Patel, , Haunschild, Robin, Bornmann, Lutz, Taylor, Mike, Ross, Liesa, Theng, Yin-Leng, Hassan, Saeed-Ul, and Aljohani, Naif R. 2020 [Link] [DOI:10.6084/M9.FIGSHARE.13010000.V2]
14. CSSA’20: Workshop on Combining Symbolic and Sub-Symbolic Methods and Their Applications Alam, Mehwish, Groth, Paul, Hitzler, Pascal, Paulheim, Heiko, Sack, Harald, and Tresp, Volker In Proceedings of the 29th ACM International Conference on Information & Knowledge Management 2020 [Abs] [Link] [DOI:10.1145/3340531.3414072]

    There has been a rapid growth in the use of symbolic representations along with their applications in many important tasks. Symbolic representations, in the form of Knowledge Graphs (KGs), constitute large networks of real-world entities and their relationships. On the other hand, sub-symbolic artificial intelligence has also become a mainstream area of research. This workshop brought together researchers to discuss and foster collaborations on the intersection of these two areas.

15. ICIDS2020 Panel: Building the Discipline of Interactive Digital Narratives Bernstein, Mark, Palosaari Eladhari, Mirjam, Koenitz, Hartmut, Louchart, Sandy, Nack, Frank, Martens, Chris, Rossi, Giulia Carla, Bosser, Anne-Gwenn, and Millard, David E. In Interactive Storytelling 2020 [Abs] [DOI:10.1007/978-3-030-62516-0_1]

    Building our discipline has been an ongoing discussion since the early days of ICIDS. From earlier international joint efforts to integrate research from multiple fields of study to today’s endeavours by researchers to provide scholarly works of reference, the discussion on how to continue building Interactive Digital Narratives as a discipline with its own vocabulary, scope, evaluation and methods is far from over. This year, we have chosen to continue this discussion through a panel in order to explore what are the epistemological implications of the multiple disciplinary roots of our field, and what are the next steps we should take as a community.

16. Technical Perspective: Query Optimization for Faster Deep CNN Explanations Schelter, Sebastian ACM SIGMOD Record 2020 [Link]
17. Apache Mahout: Machine Learning on Distributed Dataflow Systems Anil, Robin, Capan, Gokhan, Drost-Fromm, Isabel, Dunning, Ted, Friedman, Ellen, Grant, Trevor, Quinn, Shannon, Ranjan, Paritosh, Schelter, Sebastian, and Yılmazel, Özgür Journal of Machine Learning Research 2020 [Link]
18. Semantic Systems. In the Era of Knowledge Graphs - 16th International Conference on Semantic Systems, SEMANTiCS 2020, Amsterdam, The Netherlands, September 7-10, 2020, Proceedings Blomqvist, Eva, Groth, Paul, Boer, Victor, Pellegrini, Tassilo, Alam, Mehwish, Käfer, Tobias, Kieseberg, Peter, Kirrane, Sabrina, Meroño-Peñuela, Albert, and Pandit, Harshvardhan J. 2020 [Link] [DOI:10.1007/978-3-030-59833-4]
19. A longitudinal analysis of university rankings Selten, Friso, Neylon, Cameron, Huang, Chun-Kai, and Groth, Paul Quantitative Science Studies 2020 [Link] [DOI:10.1162/qss_a_00052]

2019

1. How Relevant Is Your Choice? Kolhoff, Lobke, and Nack, Frank In ICIDS 2019. Lecture Notes in Computer Science, vol 11869 2019 [Abs]

   With the release of the film Black Mirror: Bandersnatch Netflix entered the area of interactive streamed narratives. We performed a qualitative analysis with 169 Netflix subscribers that had watched the episode. The key findings show (1) participants are initially engaged because of curiosity and the novelty value, and desire to explore the narrative regardless of satisfaction, (2) perceived agency is limited due to arbitrary choices and the lack of meaningful consequences, (3) the overall experience is satisfactory but adaptions are desirable in future design to make full use of the potential of the format.

2. Transfer Learning for Biomedical Named Entity Recognition with BioBERT Symeonidou, Anthi, Sazonau, Viachaslau, and Groth, Paul In Proceedings of the Posters and Demo Track of the 15th International Conference on Semantic Systems co-located with 15th International Conference on Semantic Systems (SEMANTiCS 2019), Karlsruhe, Germany, September 9th - to - 12th, 2019. 2019 [Link]
3. Understanding data search as a socio-technical practice Gregory, Kathleen M, Cousijn, Helena, Groth, Paul, Scharnhorst, Andrea, and Wyatt, Sally Journal of Information Science 2019 [Abs] [Link] [DOI:10.1177/0165551519837182]

   Open research data are heralded as having the potential to increase effectiveness, productivity and reproducibility in science, but little is known about the actual practices involved in data search. The socio-technical problem of locating data for reuse is often reduced to the technological dimension of designing data search systems. We combine a bibliometric study of the current academic discourse around data search with interviews with data seekers. In this article, we explore how adopting a contextual, socio-technical perspective can help to understand user practices and behaviour and ultimately help to improve the design of data discovery systems.

4. Searching Data: A Review of Observational Data Retrieval Practices in Selected Disciplines Gregory, Kathleen, Groth, Paul, Cousijn, Helena, Scharnhorst, Andrea, and Wyatt, Sally Journal of the Association for Information Science and Technology 2019 [Abs] [Link] [DOI:10.1002/asi.24165]

   A cross-disciplinary examination of the user behaviors involved in seeking and evaluating data is surprisingly absent from the research data discussion. This review explores the data retrieval literature to identify commonalities in how users search for and evaluate observational research data in selected disciplines. Two analytical frameworks, rooted in information retrieval and science and technology studies, are used to identify key similarities in practices as a first step toward developing a model describing data retrieval.

5. End-to-End Learning for Answering Structured Queries Directly over Text Groth, Paul T., Scerri, Antony, Daniel, Ron, and Allen, Bradley P. In Proceedings of the Workshop on Deep Learning for Knowledge Graphs (DL4KG2019) Co-located with the 16th Extended Semantic Web Conference 2019 (ESWC 2019), Portoroz, Slovenia, June 2, 2019. 2019 [arXiv] [Link]

2018

1. Open Information Extraction on Scientific Text: An Evaluation Groth, Paul T., Lauruhn, Michael, Scerri, Antony, and Daniel, Ron In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018 2018 [Link]
2. Elsevier’s Healthcare Knowledge Graph and the Case for Enterprise Level Linked Data Standards DeJong, Alex, Bord, Radmila, Dowling, Will, Hoekstra, Rinke, Moquin, Ryan, O, Charlie, Samarasinghe, Mevan, Snyder, Paul, Stanley, Craig, Tordai, Anna, Trefry, Michael, and Groth, Paul T. In Proceedings of the ISWC 2018 Posters & Demonstrations, Industry and Blue Sky Ideas Tracks co-located with 17th International Semantic Web Conference (ISWC 2018), Monterey, USA, October 8th - to - 12th, 2018. 2018 [Link]
3. Use of Internal Testing Data to Help Determine Compensation for Crowdsourcing Tasks Lauruhn, Michael, Groth, Paul T., Harper, Corey A., and Deus, Helena F. In Proceedings of the 2nd International Workshop on Augmenting Intelligence with Humans\--in-\-the-\-Loop co-located with 17th International Semantic Web Conference (ISWC 2018), Monterey, California, October 9th, 2018. 2018 [Link]

2017

1. Indicators for the use of robotic labs in basic biomedical research: a literature analysis Groth, Paul, and Cox, Jessica PeerJ 2017 [Abs] [Link] [DOI:10.7717/peerj.3997]

   Robotic labs, in which experiments are carried out entirely by robots, have the potential to provide a reproducible and transparent foundation for performing basic biomedical laboratory experiments. In this article, we investigate whether these labs could be applicable in current experimental practice. We do this by text mining 1,628 papers for occurrences of methods that are supported by commercial robotic labs. Using two different concept recognition tools, we find that 86%–89% of the papers have at least one of these methods. This and our other results provide indications that robotic labs can serve as the foundation for performing many lab-based experiments.

2. Storing, Tracking, and Querying Provenance in Linked Data Wylot, Marcin, Cudré-Mauroux, Philippe, Hauswirth, Manfred, and Groth, Paul T. IEEE Trans. Knowl. Data Eng. 2017 [Link] [DOI:10.1109/TKDE.2017.2690299]
3. PROV2R: Practical Provenance Analysis of Unstructured Processes Stamatogiannakis, Manolis, Athanasopoulos, Elias, Bos, Herbert, and Groth, Paul T. ACM Trans. Internet Techn. 2017 [Link] [DOI:10.1145/3062176]
4. Linked Data Management Hauswirth, Manfred, Wylot, Marcin, Grund, Martin, Groth, Paul T., and Cudré-Mauroux, Philippe 2017 [Link] [DOI:10.1007/978-3-319-49340-4_9]

2016

1. Sources of Change for Modern Knowledge Organization Systems Lauruhn, Michael, and Groth, Paul KNOWLEDGE ORGANIZATION 2016 [arXiv] [DOI:10.5771/0943-7444-2016-8-622]
2. Applying Universal Schemas for Domain Specific Ontology Expansion Groth, Paul T., Pal, Sujit, McBeath, Darin, Allen, Brad, and Daniel, Ron In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, AKBC@NAACL-HLT 2016, San Diego, CA, USA, June 17, 2016 2016 [Link]
3. The FAIR Guiding Principles for scientific data management and stewardship Wilkinson, Mark D., Dumontier, Michel, Aalbersberg, IJsbrand Jan, Appleton, Gabrielle, Axton, Myles, Baak, Arie, Blomberg, Niklas, Boiten, Jan Willem, da Silva Santos, Luiz Bonino, Bourne, Philip E., Bouwman, Jildau, Brookes, Anthony J., Clark, Tim, Crosas, Mercè, Dillo, Ingrid, Dumon, Olivier, Edmunds, Scott, Evelo, Chris T., Finkers, Richard, Gonzalez-Beltran, Alejandra, Gray, Alasdair J.G., Groth, Paul, Goble, Carole, Grethe, Jeffrey S., Heringa, Jaap, ’t Hoen, Peter A C, Hooft, Rob, Kuhn, Tobias, Kok, Ruben, Kok, Joost, Lusher, Scott J., Martone, Maryann E, Mons, Albert, Packer, Abel L., Persson, Bengt, Rocca-Serra, Philippe, Roos, Marco, van Schaik, Rene, Sansone, Susanna Assunta, Schultes, Erik, Sengstag, Thierry, Slater, Ted, Strawn, George, Swertz, Morris A., Thompson, Mark, Van Der Lei, Johan, Van Mulligen, Erik, Velterop, Jan, Waagmeester, Andra, Wittenburg, Peter, Wolstencroft, Katherine, Zhao, Jun, and Mons, Barend Scientific Data 2016 [Abs] [DOI:10.1038/sdata.2016.18]

   There is an urgent need to improve the infrastructure supporting the reuse of scholarly data. A diverse set of stakeholders-representing academia, industry, funding agencies, and scholarly publishers-have come together to design and jointly endorse a concise and measureable set of principles that we refer to as the FAIR Data Principles. The intent is that these may act as a guideline for those wishing to enhance the reusability of their data holdings. Distinct from peer initiatives that focus on the human scholar, the FAIR Principles put specific emphasis on enhancing the ability of machines to automatically find and use the data, in addition to supporting its reuse by individuals. This Comment is the first formal publication of the FAIR Principles, and includes the rationale behind them, and some exemplar implementations in the community.