Deep Learning neural network models have been successfully applied to natural language processing, and are now changing radically how we interact with machines (Siri, Alexa, machine translation or the Google search engine). Large Language Models are at the core of these developments, and are being used to crack "languages" in other disciplines, ranging from programming languages (Copilot) to proteins (AlphaFold) and gene sequences (GenSLM).

This course introduces in detail the machinery that makes Deep Learning work for NLP, including the latest Transformers and Large Language Models like GPT, BERT and T5. It also covers the use of prompts for zero-shot and few-shot learning, as well as multimodal text-image models like GPT-4. The course combines theoretical and practical hands-on classes. Attendants will be able to understand the internal working of the models, and implement them nearly from scratch in Tensorflow. The aim is to allow attendees to acquire the ability to understand, modify and apply current and future Deep Learning models to NLP and other areas.

The course is part of the NLP master hosted by the Ixa NLP research group at the HiTZ research center of the University of the Basque Country (UPV/EHU).

NOTE on online attendance: In addition to onsite attendance, the classes will be broadcasted live online. The practical labs are also available online, with split groups with one lecturer in each. We offer a high-quality and engaging course, both at the theoretical and hands-on practical sessions.

Student profile

Addressed to professionals, researchers and students who want to understand and apply deep learning techniques to text. The practical part requires basic programming experience, a university-level course in computer science and experience in Python. Basic math skills (algebra or pre-calculus) are also needed.

Contents

Introduction to machine learning and NLP with Tensorflow

Machine learning, Deep learning
Natural Language Processing
A sample NLP task with ML
. Sentiment analysis
. Features
. Logistic Regression
LABORATORY: Sentiment analysis with logistic regression

Multilayer Perceptron

Multiple layers ~ Deep: MLP
Backpropagation and gradients
Learning rate
More regularization
Hyperparameters
LABORATORY: Sentiment analysis with Multilayer Perceptron

Embeddings and Recurrent Neural Networks

Representation learning
Word embeddings
From words to sequences: Recurrent Neural Networks (RNN)
LABORATORY: Sentiment analysis with RNNs

Seq2seq, Neural Machine Translation and better RNNs

Application of RNN:
. Language Models (sentence encoders)
. Language Generation (sentence decoders)
. Sequence to sequence models and Neural Machine Translation (I)
Problems with gradients in RNN
LSTM and GRU
LABORATORY: Sentiment analysis with GRUs

Attention, Transformers and Natural Language Inference

Re-thinking seq2seq:
. Attention, memory, Transformers
. State of the art NMT
Natural Language Inference with siamese networks
LABORATORY: Attention Model for NLI

Pre-trained Transformers, BERT, GPT.

Pre-trained language models.
. Multilingual transfer learning
. Fine-tuning, prompting
Deep learning frameworks
Last words
LABORATORY: Pre-trained transformers for sentiment analysis and NLI

Bridging the gap between natural languages and the visual world

Brief introduction to Deep Learning for Computer Vision
Convolutional Neural Networks
Transformers
Image captioning
Visual question answering
Text-based image generation
LABORATORY: Image captioning with CNNs and Transformers

Instructors

Person 1

Eneko Agirre

Professor, member of IXA
Director of HiTZ

Person 2

Oier Lopez de Lacalle

Assistant Professor, member of IXA
and HiTZ

Person 2

Gorka Azkune

Associate Professor, member of IXA
and HiTZ

Person 2

Ander Barrena

Asisstant Professor, member of IXA
and HiTZ

Practical details

General information

Part of the Language Analysis and Processing master program.
  • Bring your own laptop (in order to do the practical side).
  • 14 sessions, theoretical and hand-on labs (35 hours).
  • Scheduled from January 8th to the 25th 2024, 17:00-19:30.
  • Practical classes will be in two split groups.
  • Where: E4 laboratory, Computer science faculty, San Sebastian. Online link will be provided in due time.
  • The university provides some limited information about accommodation in San Sebastian (Basque/Spanish) and the Basque Country (English).
  • Teaching language: English.
  • Capacity: 25 attendants total (First-come first-served).
  • Cost: 270€ + 4€ insurance = 274€
    (If you are an UPV/EHU member or have already registered for another course, it is 270€).

Registration

Registration is open up to the 20th of December 2023 (or until full).
  • Please register by email to amaia.lorenzo@ehu.eus (subject "Registration to DL4NLP", please CC gorka.azcune@ehu.eus).
  • After you receive the payment instructions you will have three days to formalize the payment.
  • Plese use the same email for any enquiry you might have.
  • The university provides official certificates for an additional fee. Please apply AFTER completing the course.
  • The university can provide invoices addressed to universities or companies. More details are provided after registration is made.

Prerequisites
  • Basic programming experience, a university-level course in computer science and experience in Python. Basic math skills (algebra or pre-calculus) are also needed.
  • Bring your own laptop (no need to install anything).

Previous editions

Online clas of July 2021.

Handful of the 70 participants of the class of July 2020. To the right the screen lecturers had to talk to :-)

Class of January 2020

Class of July 2019