Cookies Policy
The website need some cookies and similar means to function. If you permit us, we will use those means to collect data on your visits for aggregated statistics to improve our service. Find out More
Accept Reject
  • Menu
Facts & Numbers
000
Presentation

Telecommunications and Multimedia

At CTM, our vision is to promote a lively and sustainable world where networked intelligence enables ubiquitous interaction with sensory-rich content. Our mission is to develop advanced systems and technologies to enable high capacity, efficient, and secure communications, media knowledge extraction, and immersive ubiquitous multimedia applications.

We work in 4 main areas of research: Optical and Electronic Technologies, Wireless Networks, Multimedia and Communications Technologies, and VCMI (Visual Computing and Machine Intelligence).

Latest News
Communications

Europe discusses collaboration opportunities in high-frequency wireless communications

Smart propagation environments, improvements in signal processing for the sixth generation of mobile communications, and 6G-centred network and location developments were some of the topics discussed at an event organised by the European projects TERRAMETA (coordinated by INESC TEC), 6G-SHINE and TIMES, in collaboration with RESTART-IN – an Italian PRR.

06th March 2024

Artificial Intelligence

INESC TEC researchers work on the first prototype that applies AI to colorectal diagnosis developed in Portugal

The work behind the first prototype that uses Artificial Intelligence (AI) for colorectal diagnosis was fully developed by Portuguese researchers INESC TEC, and the IMP Diagnostics Molecular & Anatomic Pathology laboratory; the work featured in the renowned international scientific journal npj Precision Oncology (https://www.nature.com/articles/s41698-024-00539-4 ).

05th March 2024

INESC TEC researchers led discussion on wireless communications and computer vision at GLOBECOM

After almost one year, the CONVERGE project (coordinated by INESC TEC) has already showed relevant outcomes at one of the main conferences of the IEEE Communications Society, the GLOBECOM (Malaysia) – namely, through the organisation of a panel. “Convergence of wireless communications and computer vision: a new paradigm created by the CONVERGE project” sought to discuss the new opportunities and potential challenges associated with the use of tools that combine radio with computer vision.

23rd January 2024

“Sayonara, Porto”: INESC TEC researchers flew to Japan for an internship at an informatics institute

This month, two INESC TEC researchers started an internship at the National Institute of Informatics, in Tokyo. By leaving their “comfort zone”, they hope to explore “different points of view”, methods and elements to apply to their PhD projects.  

23rd January 2024

INESC TEC researcher collaborated with a German group to study the personification of Large Language Models

What happens when you ask Large Language Models, e.g., ChatGPT, to take on a certain role in different contexts? Isabel Rio-Torto, a researcher at INESC TEC, joined the Explainable Machine Learning (EML) group at the University of Tuebingen (Germany) for a study that concluded that the personification of models has an impact on performance, but can also show bias. The study was accepted in the 2023 edition of NeurIPS - Conference on Neural Information Processing Systems.

08th January 2024

187

Featured Projects

AICare4U

AI-based Robotic Solution Addressing Compensatory Patterns for Upper Limb Rehabilitation

2024-2025

AI4LUNGS

AI-BASED PERSONALISED CARE FOR RESPIRATORY DISEASE USING MULTI-MODAL DATA IN PATIENT STRATIFICATION

2024-2027

PHASE IV AI

Privacy compliant health data as a service for AI development

2023-2026

PFAI4_4eD

Programa de Formação Avançada Industria 4 - 4a edição

2023-2023

UNIFY

Compilation Abstraction and Hardware Adaptation for Specialized and General-Purpose Computing Unification

2023-2026

CELLO

The sound of cells: An acoustic platform aiming towards biophysical cell fingerprints for label-free precision medicine

2023-2026

LUCCA

AI-based Models for Lung Cancer Characterization: a Multimodal and Causal Approach

2023-2024

TORIS

Towards fully printed reconfigurable intelligent surfaces

2023-2024

Shielding

Medidas de shielding de materiais

2023-2023

WATSON

A holistic framework with Anticounterfeit and Intelligence-based technologies that will assist food chain stakeholders in rapidly identifying and preventing the spread of fraudulent practices

2023-2026

CONVERGE

Telecommunications and computer vision convergence tools for research infrastructures

2023-2026

CAGING

Causality-driven Generative Models for Privacy-preserving Case-based Explanations

2023-2024

TERRAMETA

Terahertz reconfigurable metasurfaces for ultra-high rate wireless communications

2023-2025

EADIGIFOLK

An European and Ibero-American approach for the digital collection, analysis and dissemination of folk music

2023-2026

A-IQ Ready

Artificial Intelligence using Quantum measured Information for realtime distributed systems at the edge

2023-2025

SuperIoT

Truly sustainable printed electronics-based IoT combining optical and radio wireless technologies

2023-2025

AEROGANP

Creación de un eje transfronterizo de investigación y transferencia de conocimiento en el sector aeronáutico y espacial en la Eurorregión Galicia-Norte de Portugal

2023-2026

A-MoVeR

Mobilizing Agenda for the Development of Products & Systems towards an Intelligent and Green Mobility

2022-2025

OVERWATCH

Integrated holographic management map for safety and crisis events

2022-2025

Vision2Control

Controlo Qualidade Rolamentos por Visão

2022-2023

NEXUS

Innovation Pact - Digital and Green Transition

2022-2025

AURORA

Deteção de atividade no interior do veículo

2022-2023

NewSpacePortugal

Agenda New Space Portugal

2022-2025

Produtech_R3

Agenda Mobilizadora da Fileira das Tecnologias de Produção para a Reindustrialização

2022-2025

SUSTAINABLE PLASTICS

Agenda Mobilizadora para os Plásticos Sustentáveis

2022-2025

IWOW2022

IWOW2022 - NEWFOCUS COST action meeting and workshop

2022-2022

CINDERELLA

Clinical Validation of an AI-based approach to improve the shared decision-making process and outcomes in Breast Cancer Patients proposed for Locoregional treatment

2022-2026

PFAI4_3ed

Programa de Formação Avançada Industria 4 - 3a edição

2022-2022

DivaX

Services for company Europeanisation

2022-2022

ABIS

Automated Biometric Identification System

2022-2022

FORM_I40

Formação Indústria 4.0

2022-2022

THEIA

Automated Perception Driving

2022-2023

CIRCUMSTANCE

Circulating Microbial Signatures for Early Diagnosis of Cancer

2022-2024

OpenMinds

Synchronising creative minds for social cohesion and radical inclusion

2021-2023

HfPT

Health from Portugal

2021-2025

vCardID4

Digital fingerprint enhanced model - 4

2021-2022

WaveCorkCal

Calibração do higrómetro de microondas

2021-2023

CholdaDigital

Consultoria Avançada em Sistemas de Informação e Redes de Comunicações para a Quinta da Cholda

2021-2022

CadPath

Computer-Aided Diagnosis in Pathology

2021-2022

MATinMOL

Matter Waves in Moiré Lattices

2021-2025

5GforUtilities

Tecnologia Celular 5G

2021-2022

DECARBONIZE

DEvelopment of strategies and policies based on energy and non-energy applications towards CARBON neutral cities via digitalization for citIZEns and society

2021-2023

Training4DS

Formação Avançada em Data Science - Altice Labs

2020-2020

PFAI4.0

Programa de Formação Avançada Industria 4.0

2020-2021

iiLab

Ampliação da Infraestrutura Tecnológica do INESC TEC para a Transformação Digital da Indústria

2020-2023

FLY_PT

Mobilizar a indústria aeronáutica nacional para a disrupção no transporte aéreo urbano do futuro

2020-2023

Continental FoF

Fábrica do Futuro da Continental Advanced Antenna

2020-2023

TAMI

Transparent Artificial Medical Intelligence

2020-2023

WiFi4DSO

Tecnologia Wi-Fi aplicada a cenários de um DSO

2019-2019

LeGeM

Learning Representations and Generative Models for 3D Breast Data

2019-2021

CorkNetmon

Network Infrastructure Monitoring for Remote Cork Manufacturing

2019-2020

Inphinit

Bolsa de Doutoramento ”LA CAIXA” Inphinit

2019-2022

SLID

Invitation to collaborate

2019-2022

ProLab

Consultoria profissional com recurso ao laboratório de electrónica

2019-2019

InterConnect

Interoperable Solutions Connecting Smart Homes, Buildings and Grids

2019-2024

TenisApp2

Aplicação móvel para análise de jogos de ténis - 2

2019-2021

NFCAD

Near Field Contact Antenna Development

2019-2020

EuConNeCts4

European Conferences on Networks and Communications

2019-2022

SCA

Serviço de caracterização antenas

2019-2019

RESPONDRONE

NOVEL INTEGRATED SOLUTION OF OPERATING A FLEET OF DRONES WITH MULTIPLE SYNCHRONIZED MISSIONS FOR DISASTER RESPONSES

2019-2022

STRx

Sistema de transmissão e receção de sinal de orientação eletrónica para a próxima geração de constelações de satélites (LEO e MEO).

2019-2022

InterCork

Remote Cork Manufacturing

2019-2019

CLOUD4CANDY

Cloud for CANDY

2019-2019

MetroRec

Avaliação do sistema gravação vídeo do Metro do Porto

2019-2019

Evo3DModel

Consultoria para melhoria do sistema INSIGHT

2019-2020

OpenInnoTrain

Research Translation and Applied Knowledge Exchange in Practice through University-Industry-Cooperation

2019-2024

FollicleCounter

Prestação de serviços de investigação e desenvolvimento em matéria de processamento de imagem para maior fiabilidade de aferição dos folículos implantados

2018-2021

NB-IoT

Consultoria no âmbito da tecnolofia Narrowband-internet of Things

2018-2019

XPERIMUS

Experimentação em música na cultura portuguesa: História, contextos e práticas nos séculos XX e XXI

2018-2022

Blueenergy

Blue energy generation using hybrid triboelectric/photovoltaic systems for the long term deployment of Autonomous Underwater Vehicles

2018-2020

AUTOMOTIVE

AUTOmatic multiMOdal drowsiness detecTIon for smart Vehicles

2018-2021

GROW

Long-range broadband underwater wireless communications

2018-2021

HELP-MD

O poder emocional e curativo da música e da dança

2018-2022

NeurOxide

Integration of oxide thin film transistors and memristors in neuromorphic networks

2018-2022

PEPCC

Power efficiency and performance for embedded and HPC systems with custom CGRAs

2018-2021

LUCAS

Lung cancer screening - A non-invasive methodology for early diagnosis

2018-2022

HEMOSwimmers

Hemodynamic optimization around 3D swimming microbots

2018-2022

CLARE

Computer-aided cervical cancer screening

2018-2021

ENDURANCE

Underwater wireless energy and communications enabling long-term deep-sea presence

2018-2020

S-MODE

Screening of antibiotic contamination by mobile devices

2018-2021

UnWSNet

Underwater Wireless Sensor Networks

2018-2018

SIMBED

Fed4fire testbed for experimentation

2018-2019

FotoInMotion

Repurposing and enriching images for immersive storytelling through smart digital tools

2018-2020

ConnectedRefinery

Rede de comunicações sem fios para as instalações da Galp Energia em Leixões

2018-2019

5G

Componentes e Serviços para Redes 5G

2018-2021

Arquitetura_IoT

Consultoria sobre a arquitetura de referência para implementação serviços de informação baseados em IoT

2017-2018

CompMash

Music compatibility models for interactive mashup applications

2017-2019

CHIC

Cooperative Holistic view on Internet and Content

2017-2020

UGREEN

Otimização do Consumo Energético de Redes LTE-U e Wi-Fi em Cenários de Coexistência

2017-2019

SURGEONMATE

Video processing for surgery analysis

2017-2017

TERAPOD

Terahertz based Ultra High Bandwidth Wireless Access Networks

2017-2021

TEC4Sea

Modular Platform for Research, Test and Validation of Technologies supporting a Sustainable Blue Economy

2017-2022

ROMOVI

ROMOVI: Robô Modular e cooperativo para Vinhas de encosta

2017-2019

BCCT.Plan

BCCT.plan: Ferramenta 3D para o planeamento do tratamento conservador do cancro da mama

2016-2020

WI-GREEN

WI-GREEN .: Otimização do consumo energético de redes Wi-Fi sensível aos padrões de tráfego

2016-2018

RAWFIE

Road-, Air- and Water-based Future Internet Experimentation

2016-2019

Cloud-Setup

PLATAFORMA DE PREPARAÇÃO DE CONTEÚDOS AUDIOVISUAIS PARA INGEST NA CLOUD

2016-2019

EVOXANT

Bacterial evolution beyond the cultured isolates - Xanthomonas arboricola pv. juglandis as a paradigm

2016-2020

WISE

TrafficAware Flying Backhaul Mesh Networks

2016-2019

MareCom

Redes e serviços marítimos comunitários

2016-2018

CORAL-TOOLS

CORAL – Sustainable Ocean Exploitation: Tools and Sensors

2016-2018

STRONGMAR-CRAS

STRengthening MARritime Technology Research Center

2016-2018

BLUECOM+

Connecting Humans and Systems at Remote Ocean Areas using Cost-effective Broadband Communications

2015-2017

ENDURE

Enabling Long-Term Deployments of Underwater Robotic Platforms in Remote Oceanic Locations

2015-2017

FOUREYES

TEC4Growth - RL FourEyes - Intelligence, Interaction, Immersion and Innovation for media industries

2015-2019

NanoStima-RL5

NanoSTIMA - Advanced Methodologies for Computer-Aided Detection and Diagnosis

2015-2019

NanoStima-RL1

NanoSTIMA - Macro-to-Nano Human Sensing Technologies

2015-2019

SMILES

SMILES - Smart, Mobile, Intelligent and Large scale Sensing and analytics

2015-2019

VAMOS

Viable Alternative Mine Operating System

2015-2019

SCREEN

Space Cognitive Radio for Electromagnetic Environment maNagement

2015-2016

iBROW

Innovative ultra-BROadband ubiquitous Wireless communications through terahertz transceivers

2015-2018

AnyPLACE

Adaptable Platform for Active Services Exchange

2015-2018

SmarterEMC2

Smarter Grid: Empowering SG Market Actors through Information and Communication Technologies

2015-2017

SEAD

Statistically Enhanced Mixed-Signaland Analog Design

2014-2016

MDX

Simulation Models

2014-2016

Unisat

Serviços de banda-larga em simultâneo com receção de televisão por satélite

2014-2014

TWAVE

Phase conjugated twin waves to unlock the potential of future spatial division multiplexed systems (TWave)

2014-2015

HiperWireless

Microwave Point-to-Multipoint Communications in Free Hiperlan Band (17GHZ)

2014-2015

PGLobal

Desenvolvimento de software para ser integrado numa plataforma de recolha automática e selecção de conteúdos de jornais participantes de vários países

2014-2015

SUNNY

Smart UNmanned aerial vehicle sensor Network for detection of border crossing and illegal entrY

2014-2018

PCSA

Place characterisation from sensing and acting

2013-2014

SIVIC

Wearable Integrated Cardiovascular Surveillance System

2013-2015

Creation

Cognitive Radio Transceiver Design for energy Efficient Daa Transmission

2013-2016

TDT

Digital terrestrial television (DTT) signal monitoring

2013-2014

PICTURE

Patient Information Combined for the Assessment of specific surgical outcomes in breast cancer

2013-2016

Confine

Community Networks Testbed for the Future Internet

2013-2015

MAT

Media Arts and Technology

2013-2015

Sensing

Network Sensing for Critical Systems Monitoring

2013-2015

SmartGrids

Smart Grids

2013-2015

Cooperation

Cooperation and Perception for Augmented Autonomy

2013-2015

ASSIST

Retail futsal statistics

2012-2013

MOGTIDT

Station to obtain 3D images

2012-2012

MTGrid

Multi Technology Communication Infrastructure for the Smart Grid

2012-2015

RETAIL_PRO

Integrated Platform to Strategically Manage Retail Environments

2012-2015

WiSAT

Waveguide passive devices in S and Ka band

2012-2012

SARA

Asset Management System for Road Networks

2012-2015

CPT

Cartesian Polar Transmitter

2012-2014

MC-WMNs

Multiple Context-based Wireless Mesh Networks (MC-WMNs)

2012-2015

SENSEIVER

Low-cost and energy-efficient LTCC sensor/IR-UWB transceiver solutions for a sustainable healthy environment

2011-2015

MIRes

Roadmap for Music Information Research

2011-2013

AdChrono

Automatic optimisation of online advertising

2011-2013

3dBCT

3D Models for Aesthetic Evaluations and Result Predictions in Breast Cancer Procedures

2011-2014

CASA

Computational Auditory Scene Analysis Framework for Sound Segregation in Music Signals

2011-2014

SHAKEIT

Mechanisms of Musical Groove and applications

2011-2013

MultiRadioAccess

Multi-radio (HSPA and WiFi) aggregation to provide a higher rate than those of the individual networks

2011-2011

Steering

Steering of light in nonlinear waveguides with resonant interactions

2011-2014

CNG

New Generation Contents for Education and Vocational Training

2011-2014

AAL4ALL

Ambient Assisted Living for All

2011-2015

SUM

Sensing and Understanding human Motion dynamics

2011-2013

User-Tracking2.0

User-Tracking for Web traffic

2010-2011

SELF-PVP

Self-organizing power management for photovoltaic power plants

2010-2014

ImTV

On demand Immersive-TV for communities of Media Producers and Consumers

2010-2014

NeTS

Next Generation Network Operations and Management

2010-2013

EscolinhasCriativas

Creative Spaces for Creative Kids

2010-2013

Convergence

Content-centric, publish-subscribe service model for the Internet

2010-2013

SafeHomeHealthCare

Interference-free Home Health-Care Smart Spaces using Search Algorithms and Meta-Reality Reflection

2010-2013

OSP

Optical Signal Processing Using Highly Nonlinear Fibers

2010-2013

ProLimb

Electronic sensing for the prophylaxis of lower limb pathologies

2010-2013

ContextAware

Context-aware and personalized multimedia services

2010-2011

REIVE

Intelligent electric networks with plug-in electric vehicles

2010-2012

Alicante

MediA Ecosystem Deployment through Ubiquitous Content-Aware Network Environments

2010-2013

P3.net

On-line daily news platform for young people

2010-2012

LUL

Living Usability Lab for Next Generation Networks

2010-2012

Hotel3.0

Web3.0 Platform for the hospitality market

2010-2012

WOWI

Wireless-optical-wireless interfaces for picocellular access networks

2010-2013

RobVigil

Collaborative and intelligent surveillance robot for the security area

2010-2012

Daphne

Developing aircraft photonic networks

2009-2013

SWIOP

Intelligent and secure Webmail System to Support Personal Organization

2009-2011

SITMe

Metropolitan multi-technology wireless network for public transportation systems

2009-2012

Mobiles

Sustainable electric mobility - Solutions for the logistics associated with electric vehicle battery charging

2009-2012

V-SAT

Passive devices for VSAT (Very Small Aperture Terminal) applications

2009-2009

PortalDouro

Tourism portal for the Douro region

2009-2011

KINETIC

Controller driven adaptive and dynamic music composition systems

2009-2011

ASP-RedesDomesticas

Authentication, Security and Privacy (ASP) Solutions for home networks

2009-2010

MuMoMgt

Multicast and mobility management in heterogeneous access networks

2009-2012

ReCoop

Cooperative Wireless Networks

2009-2012

SemanticPACS

Picture Archiving and Communication System with Semantic Search Engine

2009-2011

Palco3.0

Intelligent Web system to support the management of a social network on music

2008-2011

AHRS

Attitude-Heading Reference System based on MEMS technology

2008-2010

GeCLIFmcast

Management of multicast sessions for IP based services created by users

2008-2009

DR-Vids

Dynamic reconfiguration of logical resources for real time foreground/background video segmentation

2007-2010

Vector

Compilation and Synthesis of Image Processing Algorithms in MATLAB for FPGA-based Custom Vector Units

2007-2011

OMR

Optical recognition system for handwritten music scores

2007-2010

BCCT

Advanced objective method for the evaluation of the aesthetic result of Breast Cancer Conservative Treatment

2007-2010

ROFWDM

Design and Optimisation of WDM Millimetre-Wave Fibre-Radio Systems

2007-2010

EDCine

Enhanced Digital Cinema

2007-2009

VISNETII

Networked Audiovisual Media Technologies

2006-2009

Team
002

Laboratories

Laboratory of Sound and Music Computing

Optical and Electronic Technologies Research Laboratory

Publications

CTM Publications

View all Publications

2024

A Machine Learning App for Monitoring Physical Therapy at Home

Authors
Pereira, B; Cunha, B; Viana, P; Lopes, M; Melo, ASC; Sousa, ASP;

Publication
SENSORS

Abstract
Shoulder rehabilitation is a process that requires physical therapy sessions to recover the mobility of the affected limbs. However, these sessions are often limited by the availability and cost of specialized technicians, as well as the patient's travel to the session locations. This paper presents a novel smartphone-based approach using a pose estimation algorithm to evaluate the quality of the movements and provide feedback, allowing patients to perform autonomous recovery sessions. This paper reviews the state of the art in wearable devices and camera-based systems for human body detection and rehabilitation support and describes the system developed, which uses MediaPipe to extract the coordinates of 33 key points on the patient's body and compares them with reference videos made by professional physiotherapists using cosine similarity and dynamic time warping. This paper also presents a clinical study that uses QTM, an optoelectronic system for motion capture, to validate the methods used by the smartphone application. The results show that there are statistically significant differences between the three methods for different exercises, highlighting the importance of selecting an appropriate method for specific exercises. This paper discusses the implications and limitations of the findings and suggests directions for future research.

2024

Classification of Pulmonary Nodules in 2-[<SUP>18</SUP>F]FDG PET/CT Images with a 3D Convolutional Neural Network

Authors
Alves, VM; Cardoso, JD; Gama, J;

Publication
NUCLEAR MEDICINE AND MOLECULAR IMAGING

Abstract
Purpose 2-[F-18]FDG PET/CT plays an important role in the management of pulmonary nodules. Convolutional neural networks (CNNs) automatically learn features from images and have the potential to improve the discrimination between malignant and benign pulmonary nodules. The purpose of this study was to develop and validate a CNN model for classification of pulmonary nodules from 2-[F-18]FDG PET images.Methods One hundred thirteen participants were retrospectively selected. One nodule per participant. The 2-[F-18]FDG PET images were preprocessed and annotated with the reference standard. The deep learning experiment entailed random data splitting in five sets. A test set was held out for evaluation of the final model. Four-fold cross-validation was performed from the remaining sets for training and evaluating a set of candidate models and for selecting the final model. Models of three types of 3D CNNs architectures were trained from random weight initialization (Stacked 3D CNN, VGG-like and Inception-v2-like models) both in original and augmented datasets. Transfer learning, from ImageNet with ResNet-50, was also used.Results The final model (Stacked 3D CNN model) obtained an area under the ROC curve of 0.8385 (95% CI: 0.6455-1.0000) in the test set. The model had a sensibility of 80.00%, a specificity of 69.23% and an accuracy of 73.91%, in the test set, for an optimised decision threshold that assigns a higher cost to false negatives.Conclusion A 3D CNN model was effective at distinguishing benign from malignant pulmonary nodules in 2-[F-18]FDG PET images.

2024

Active Supervision: Human in the Loop

Authors
Cruz, RPM; Shihavuddin, ASM; Maruf, MH; Cardoso, JS;

Publication
PROGRESS IN PATTERN RECOGNITION, IMAGE ANALYSIS, COMPUTER VISION, AND APPLICATIONS, CIARP 2023, PT I

Abstract
After the learning process, certain types of images may not be modeled correctly because they were not well represented in the training set. These failures can then be compensated for by collecting more images from the real-world and incorporating them into the learning process - an expensive process known as active learning. The proposed twist, called active supervision, uses the model itself to change the existing images in the direction where the boundary is less defined and requests feedback from the user on how the new image should be labeled. Experiments in the context of class imbalance show the technique is able to increase model performance in rare classes. Active human supervision helps provide crucial information to the model during training that the training set lacks.

2024

Explaining Bounding Boxes in Deep Object Detectors Using Post Hoc Methods for Autonomous Driving Systems

Authors
Nogueira, C; Fernandes, L; Fernandes, JND; Cardoso, JS;

Publication
SENSORS

Abstract
Deep learning has rapidly increased in popularity, leading to the development of perception solutions for autonomous driving. The latter field leverages techniques developed for computer vision in other domains for accomplishing perception tasks such as object detection. However, the black-box nature of deep neural models and the complexity of the autonomous driving context motivates the study of explainability in these models that perform perception tasks. Moreover, this work explores explainable AI techniques for the object detection task in the context of autonomous driving. An extensive and detailed comparison is carried out between gradient-based and perturbation-based methods (e.g., D-RISE). Moreover, several experimental setups are used with different backbone architectures and different datasets to observe the influence of these aspects in the explanations. All the techniques explored consist of saliency methods, making their interpretation and evaluation primarily visual. Nevertheless, numerical assessment methods are also used. Overall, D-RISE and guided backpropagation obtain more localized explanations. However, D-RISE highlights more meaningful regions, providing more human-understandable explanations. To the best of our knowledge, this is the first approach to obtaining explanations focusing on the regression of the bounding box coordinates.

2024

Intrinsic Explainability for End-to-End Object Detection

Authors
Fernandes, L; Fernandes, JND; Calado, M; Pinto, JR; Cerqueira, R; Cardoso, JS;

Publication
IEEE ACCESS

Abstract
Deep Learning models are automating many daily routine tasks, indicating that in the future, even high-risk tasks will be automated, such as healthcare and automated driving areas. However, due to the complexity of such deep learning models, it is challenging to understand their reasoning. Furthermore, the black box nature of the designed deep learning models may undermine public confidence in critical areas. Current efforts on intrinsically interpretable models focus only on classification tasks, leaving a gap in models for object detection. Therefore, this paper proposes a deep learning model that is intrinsically explainable for the object detection task. The chosen design for such a model is a combination of the well-known Faster-RCNN model with the ProtoPNet model. For the Explainable AI experiments, the chosen performance metric was the similarity score from the ProtoPNet model. Our experiments show that this combination leads to a deep learning model that is able to explain its classifications, with similarity scores, using a visual bag of words, which are called prototypes, that are learned during the training process. Furthermore, the adoption of such an explainable method does not seem to hinder the performance of the proposed model, which achieved a mAP of 69% in the KITTI dataset and a mAP of 66% in the GRAZPEDWRI-DX dataset. Moreover, our explanations have shown a high reliability on the similarity score.

Facts & Figures

82Researchers

2016

2Book Chapters

2020

2R&D Employees

2020

Contacts