Humans Learn From Task Descriptions and So Should Our Models

An excellent talk on PET: https://www.youtube.com/watch?v=_YOaRLQnBjc&t=246s

TDLR; They provide a task description (patterns + class label verbalizer; abbreviated by PET) plus a few examples of the target task of interest to a pre-trained LLM, and train the LLM based on the few examples.

Their main difference to GPT3 prompting is that they allow the model to be trained based on the few examples in the prompt. They show that it leads to big improvements compared to GPT3, even if the size of their LLM is much smaller compared to GPT3. The intuition is that, as humans, we update our brains and 'learn' after receiving examples.

• Can ‘one’ model be trained for a large number of patterns?
• Yes, maybe in a sequential manner. That is, the problem is now reduced to ‘continual learning', or even multitask learning?
• Maybe subnetwork pruning can help here, to identify the relevant subnetwork and only update those parameters.  
• Can the unlabeled data be used to filter our good patterns?
• Yes, based on the consistency criterion (quality estimation for patterns)
• Can PET be used for other tasks?
• Yes, currently under development. But it is not trivial to design descriptions for all tasks. 
• PET provides a framework for involving ‘human in the loop’ for learning
• As opposed to active learning. 
• Automatic construction of patterns is good, but we need them to be interpretable by humans (no problem of having humans in the loop) → can we do AL for getting patterns?

There are some relevant works on prompting from Google Research (Quoc V Le), CMU (Graham Neubig), and UW (Hannaneh Hajishirazi) which hopefully I can blog about in the future.