Speech Pre-training

 

W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

Comments

Post a Comment

Popular posts from this blog

Humans Learn From Task Descriptions and So Should Our Models

Image
An excellent talk on PE T: https://www.youtube.com/watch?v=_YOaRLQnBjc&t=246s TDLR; They provide a task description (patterns + class label verbalizer; abbreviated by PET) plus a few examples of the target task of interest to a pre-trained LLM, and train the LLM based on the few examples. Their main difference to GPT3 prompting is that they allow the model to be trained based on the few examples in the prompt. They show that it leads to big improvements compared to GPT3, even if the size of their LLM is much smaller compared to GPT3. The intuition is that, as humans, we update our brains and 'learn' after receiving examples. Can ‘one’ model be trained for a large number of patterns? Yes, maybe in a sequential manner. That is, the problem is now reduced to ‘continual learning', or even multitask learning? Maybe subnetwork pruning can help here, to identify the relevant subnetwork and only update those parameters.   Can the unlabeled data be used to filter our good patte...
Read more

Divergence Frontiers for Evaluating Deep Generative Models

I was reading a collection of interesting papers on the evaluation of deep generative models, which I have summarised in the following slides .  The papers include:  Assessing Generative Models via Precision and Recall " (NeurIPS 2018) Precision-Recall Curves Using Information Divergence Frontiers  (AISTATS 2020) Divergence Frontiers for Generative Models:Sample Complexity, Quantization Effects,and Frontier Integrals  (NuerIPS 2021) MAUVE: Measuring the GapBetween Neural Text and Human Textusing Divergence Frontiers  (NuerIPS 2021)
Read more