GPT 原理解析

GPT 原理解析

GPT锛圙enerative Pre-Training锛夋槸涓涓吀鍨嬬殑涓ら樁娈靛紡妯″瀷锛氭棤鐩戠潱棰勮缁+鐩戠潱涓嬫父浠诲姟寰皟銆傝屽湪 NLP 浠诲姟涓紝涓婃父鐨勬棤鐩戠潱棰勮缁冧换鍔″鍗婇噰鐢ㄨ瑷妯″瀷鏉ュ疄鐜帮紝涓嬫父鐨勭洃鐫ｄ换鍔＄洿鎺ュ涓婃父鐨勬ā鍨嬭繘琛岀畝鍗曟敼閫犲嵆鍙娇鐢ㄣ
   GPT 涓轰簡鑳藉鏈夋晥鐨勬姄鍙栨枃鏈腑鐨勮涔変俊鎭紝浣跨敤浜嗗崟鍚戠殑 Transformer Decoder 妯″潡鏋勫缓鏍囧噯鐨勮瑷妯″瀷锛屽啀浣跨敤棰勮缁冨緱鍒扮殑缃戠粶鏋舵瀯涓庡弬鏁拌繘琛屼笅娓哥洃鐫ｄ换鍔＄殑寰皟锛屽彇寰椾簡涓嶉敊鐨勬晥鏋溿
   璁烘枃锛 Improving Language Understanding by Generative Pre-Training
  
瀵逛簬璇枡   锛孏PT 鏋勫缓鏍囧噯鐨勮瑷妯″瀷閫楄春妫氾細
     
   鏂囩珷涓娇鐢 Transformer Decoder 妯″潡鍔犱笂鍓嶉绁炵粡缃戠粶锛屾渶鍚庝娇鐢 softmax 杈撳嚭鐩爣璇嶇殑鍒嗗竷锛

瀵逛簬閫氳繃绗竴闃舵鐨勯璁粌寰楀埌鐨勮瑷妯″瀷锛屽浜庣壒瀹氱殑浠诲姟杩涜 fine-tuning銆
   瀵逛簬涓涓洃鐫ｆ暟鎹泦   锛屽叾涓殑鏁版嵁涓轰竴涓簭鍒    鍜屼竴涓爣绛   銆傚皢搴忓垪杈撳叆棰勮缁冩ā鍨嬪悗寰楀埌杈撳嚭鍚戦噺涓   锛屾帴鐫浣跨敤涓涓嚎鎬у眰鏉ラ娴嬫爣绛撅細
     
   闇鏋佸ぇ鍖栫殑浼肩劧鍑芥暟涓猴細
     
   鍙﹀锛屼綔鑰呭彂鐜帮紝浣跨敤璇█妯″瀷鏉ヨ緟鍔╃洃鐫ｅ涔犵殑浠诲姟杩涜寰皟锛屾湁涓や釜濂藉锛
  
鎵浠ワ紝鏈缁堜笅娓镐娇鐢ㄧ殑鐩戠潱妯″瀷鐩爣鍑芥暟涓猴細

GPT 浣跨敤涓ら樁娈靛紡妯″瀷鐨勫彟澶栦竴涓ソ澶勬槸锛屼綔鑰呮湡鏈涢氳繃绗竴灞卞垯闃舵鐨勬媿鐚滈璁粌璇█妯″瀷锛屽涔犲埌灏藉彲鑳藉鐨勮嚜鐒惰瑷淇℃伅锛屼笖瀵逛簬澶у鏁颁笅娓镐换鍔★紝鍙渶瑕佺畝鍗曠殑淇敼杈撳叆鑰屼笉闇瑕佷慨鏀规ā鍨嬫灦鏋勫嵆鍙畬鎴愬井璋冦傚浜 NLP 涓殑鍑犵涓绘祦浠诲姟锛孏PT 鍒嗗埆鍋氫簡濡備笅鐨勫彉鎹㈢瓥鐣ワ細
                                          
妯″瀷杩樺寘鎷竴浜涚粏鑺傦細
  
璁烘枃锛 Language Models are Unsupervised Multitask Learners
   GPT-2 鏄 GPT 鐨勭洿鎺ュ崌绾х増锛屾晥鏋滄儕浜恒傜浉姣斾箣涓嬶紝GPT-2 鏈夊涓嬪嚑鐐规敼杩涳細