stable diffusion Dreambooth训练过程提示错误?

杩欎釜閿欒鎻愮ず閫氬父鍑虹幇鍦ㄤ娇鐢∟VIDIA GPU杩涜璁粌鏃讹紝瀹冩彁绀轰綘鍙兘浼氬湪鍏朵粬API璋冪敤鏃跺紓姝ユ姤鍛奀UDA鏍稿績閿欒锛屽洜姝や笅闈㈢殑鍫嗘爤璺熻釜鍙兘鏄笉姝ｇ‘鐨勩傝繖涓敊璇氬父鏄敱浜嶤UDA鏍稿績鍑虹幇闂鑰岃寗璐虹┖瀵艰嚧鐨勶紝鍙兘鐨勫師鍥犲寘鎷細
1. GPU杩囩儹鎴栬呴鐜囦笉绋冲畾閫犳垚鐨勭‖浠堕棶棰橈紱
2. 鍙傛暟涓嶆纭鑷寸殑閰嶇疆閿欒锛
3. 纭欢鎴栬呴┍鍔ㄤ笉鍏煎瀵艰嚧鐨勯棶棰樸
瑕佽В鍐宠繖涓棶棰橈紝浣犲彲浠ュ皾璇曚互涓嬪嚑涓楠わ細
1. 妫鏌PU鏄惁杩囩儹鎴栬呴鐜囦笉绋冲畾锛屽鏈夊繀瑕侊紝璇疯繘琛屾暎棰ょ瀻鐑垨鑰呰皟鏁撮鐜囥
2. 妫鏌ラ厤缃枃浠跺拰鍙傛暟鏄惁姝ｇ‘璁剧疆銆傜‘淇濇墍鏈夊奸兘鍦ㄦ寚瀹氱殑鑼冨洿鍐咃紝骞朵笖娌℃湁閿欏埆瀛楁垨鑰呮嫾鍐欓敊璇
3. 鏇存柊GPU椹卞姩绋嬪簭骞堕噸鍚绠楁満锛岀‘淇濈‖浠跺拰椹卞姩绋嬪簭鍏煎鎬ц壇濂姐
鍙﹀锛屼綘鎷嶅尽涔熷彲浠ユ寜鐓ф彁绀猴紝娣诲姞鐜鍙橀噺CUDA_LAUNCH_BLOCKING=1鏉ュ惎鐢–UDA debug妯″紡锛屼互鏇村ソ鍦拌皟璇旵UDA绋嬪簭銆

杩欎釜閿欒琛ㄦ槑鍦ㄤ娇鐢 CUDA 杩涜璁＄畻鏃堕亣鍒颁簡闈炴硶鍐呭瓨璁块棶銆備互涓嬫槸涓浜涘缓璁拰鍙兘鐨勮В鍐虫柟妗堬細

鏇存柊椹卞姩鍜 CUDA锛氱‘淇濇偍鐨勬樉鎺樺ぇ鍗￠┍鍔ㄥ拰 CUDA 宸ュ叿鍖呴兘鏄渶鏂扮増鏈傛洿鏂板畠浠彲鑳戒細瑙ｅ喅姝ら棶棰樸
璁剧疆 CUDA_LAUNCH_BLOCKING锛氬湪杩愯绋嬪簭涔嬪墠锛岃缃幆澧冨彉閲 CUDA_LAUNCH_BLOCKING=1锛岃繖灏嗕娇 CUDA 鍐呮牳鍚屾杩愯锛屽彲鑳藉府鍔╂偍鎵惧埌闂鎵鍦ㄣ傚湪缁堢鎴栧懡浠ゆ彁绀虹涓緭鍏ヤ互涓嬪唴瀹癸細

export CUDA_LAUNCH_BLOCKING=1

瀵逛簬 Windows PowerShell锛屼娇鐢細
$env:CUDA_LAUNCH_BLOCKING = 1

妫鏌ュ唴瀛樺垎閰嶏細纭繚鎮ㄦ病鏈夊垎閰嶈秴杩 GPU 涓婂彲鐢ㄥ唴瀛樼殑寮犻噺銆傛偍鍙互浣跨敤 torch.cuda.memory_allocated() 鍜 torch.cuda.memory_reserved() 鍑芥暟妫鏌ュ凡鍒嗛厤鍜岄鐣欑殑 GPU 鍐呭瓨銆
妫鏌ヤ唬鐮佷腑鐨勯敊璇細浠旂粏妫鏌ユ偍鐨勪唬鐮侊紝纭繚娌℃湁閿欒鐨勫唴瀛樿闂紝渚嬪瓒婄晫璁块棶鎴栨湭鍒濆鍖栫殑鍙橀噺銆傝繖鍙兘闇瑕佷粩缁嗗鏌ヤ笌 CUDA 鐩稿叧鐨勬墍鏈夋搷浣溿
闄嶄綆 GPU 鐨勮秴棰戯細濡傛灉鎮ㄧ殑 GPU 琚秴棰戯紝鍙兘浼氬鑷翠笉绋冲畾鐨勮涓恒傚皾璇曞皢 GPU 鐨勮秴棰戣缃仮澶嶅埌榛樿鍊笺
浣跨敤鏇翠綆鐨勫苟琛屽害锛氬皾璇曢檷浣庢ā鍨嬪拰鏁版嵁鍔犺浇鍣ㄧ殑鎵规澶у皬鍜屽伐浣滅嚎绋嬫暟銆傝繖鍙兘鏈夊姪浜庡噺杞绘灟娑 GPU 鍐呭瓨鍘嬪姏銆

濡傛灉鎮ㄤ娇鐢ㄧ殑鏄鍙朵竴閿寘锛堜竴涓鍏堥厤缃ソ鐨勭幆澧冿級锛屾偍鍙互灏濊瘯鍦ㄨ繍琛岃缁冭剼鏈箣鍓嶈缃幆澧冨彉閲忋備互涓嬫槸濡備綍鍦ㄤ笉鍚屾搷浣滅郴缁熶腑璁剧疆鐜鍙橀噺鐨勬柟娉曪細
瀵逛簬 Windows 鍛戒护鎻愮ず绗︼細

鎵撳紑鍛戒护鎻愮ず绗︺
鍦ㄥ懡浠ゆ彁绀虹涓紝杈撳叆浠ヤ笅鍛戒护骞舵寜 Enter锛

set CUDA_LAUNCH_BLOCKING=1

鍦ㄥ悓涓涓懡浠ゆ彁绀哄垽璐ョ珫绗︾獥鍙ｄ腑锛岃繍琛岀鍙朵竴閿寘鐨勮缁冭剼鏈
瀵逛簬 Windows PowerShell锛
鎵撳紑 PowerShell銆
鍦 PowerShell 涓紝杈撳叆浠ヤ笅鍛戒护骞舵寜 Enter锛

$env:CUDA_LAUNCH_BLOCKING = 1

鍦ㄥ悓涓涓 PowerShell 绐楀彛涓紝杩愯绉嬪彾涓閿寘鐨勮缁冭剼鏈
瀵逛簬 Linux 鎴 macOS 缁堢锛
鎵撳紑缁堢銆
鍦ㄧ粓绔腑锛岃緭鍏ヤ互涓嬪懡浠ゅ苟鎸 Enter锛

export CUDA_LAUNCH_BLOCKING=1

鍦ㄥ悓涓涓粓绔獥鍙ｄ腑锛岃繍琛岀鍙朵竴閿寘鐨勮缁冭剼鏈
璁剧疆鐜鍙橀噺鍚庯紝灏濊瘯鍐嶆杩愯璁粌鑴氭湰銆

杩欎釜閿欒鎻愮ず閫氬父涓嶱yTorch浣跨敤CUDA鏈夊叧銆"CUDA kernel errors might be asynchronously reported at some other API call"鎰忓懗鐫鍦≒yTorch鐨勬煇涓紓姝PI璋冪敤鏃跺彲鑳戒細鍙戠敓CUDA kernel閿欒锛岃繖鍙兘瀵艰嚧涓嬮潰鐨勫爢鏍堣窡韪笉姝ｇ‘銆
"consider passing CUDA_LAUNCH_BLOCKING=1"鍛婇殣鏄竴涓缓璁紝鍙互閫氳繃鍦ㄨ缁冭剼鏈腑璁剧疆鐜鍙橀噺鏉ュ惎鐢–UDA_LAUNCH_BLOCKING銆傝繖灏嗕娇PyTorch鍦ㄦ瘡涓狢UDA kernel鐨勫惎鍔ㄦ椂杩涜鍚屾锛屽苟涓斿彲鑳芥湁鍔╀簬鏇村噯纭湴纭畾閿欒銆
鍦ㄧ粓绔仛寮婃垨鑴氭湰涓墽琛屼互涓嬪懡浠ゅ嵆鍙缃幆澧冨彉閲忥細
export CUDA_LAUNCH_BLOCKING=1
鐒跺悗鍐嶉噸鏂拌繍琛屼綘鐨勮缁冭剼鏈嵆绾弸鏃忓彲銆

杩欎釜閿欒鎻愮ず鏄湪浣夸綑鎼忕敤CUDA杩涜娣卞害瀛︿範璁粌鏃跺嚭鐜扮殑銆傚畠鐨勬剰鎬濇槸CUDA鍐呮牳閿欒鍙兘浼氬湪鍏朵粬API璋冪敤鏃跺紓姝ユ姤鍛婏紝鍥犳涓嬮潰鐨勫爢鏍堣窡韪彲鑳戒笉姝ｇ‘銆備负浜嗚繘琛岃皟璇曪紝鍙互鑰冭檻浼犻扖UDA_LAUNCH_BLOCKING=1銆
杩欎釜閿欒鎻愮ず寤鸿鎮ㄥ湪杩涜CUDA娣卞害瀛︿範璁粌鏃讹紝灏咰UDA_LAUNCH_BLOCKING璁剧疆涓1锛屼互渚垮湪鍐呮牳閿欒鍙戠敓鏃剁珛鍗冲仠姝㈢▼搴忓苟鎵撳嵃閿欒淇℃伅锛屼互渚挎洿濂藉湴杩涜璋冭瘯銆傚彲浠ュ湪璁粌浠ｇ爜涓坊鍔犱互涓嬩唬鐮佹潵璁剧疆CUDA_LAUNCH_BLOCKING锛
```python
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
```
杩欐牱锛屽湪璁粌杩囩▼涓鏋滃嚭鐜板唴鏍搁敊璇抗姣侊紝绋嬪簭浼氱珛鍗冲仠姝㈠苟鎵撳嵃閿欒淇℃伅锛屾柟渚胯繘濮挎瘉澶囪璋冭瘯銆