Scaling Laws for Transformers (TBF)
TBF
SCALE EFFICIENTLY: INSIGHTS FROM PRE-TRAINING AND FINE-TUNING TRANSFORMERS
Scaling Laws for Neural Machine Translation
Humans Learn From Task Descriptions and So Should Our Models
An excellent talk on PE T: https://www.youtube.com/watch?v=_YOaRLQnBjc&t=246s TDLR; They provide a task description (patterns + class label verbalizer; abbreviated by PET) plus a few examples of the target task of interest to a pre-trained LLM, and train the LLM based on the few examples. Their main difference to GPT3 prompting is that they allow the model to be trained based on the few examples in the prompt. They show that it leads to big improvements compared to GPT3, even if the size of their LLM is much smaller compared to GPT3. The intuition is that, as humans, we update our brains and 'learn' after receiving examples. Can ‘one’ model be trained for a large number of patterns? Yes, maybe in a sequential manner. That is, the problem is now reduced to ‘continual learning', or even multitask learning? Maybe subnetwork pruning can help here, to identify the relevant subnetwork and only update those parameters. Can the unlabeled data be used to filter our good patte...
Comments
Post a Comment