澶辨晥閾炬帴澶勭悊 |
Layer normalization 綃?nbsp; PDF 涓嬭澆
杞澆鑷細(xì)http://www.python222.com/article/1162
鐩稿叧鎴浘錛?/strong>
![]() 涓昏鍐呭錛?/strong>
Layer normalization-鏂規(guī)硶綃?/strong>
涓€銆?/strong>Layer Norm 綃?/strong>
1.1 Layer Norm 鐨勮綆楀叕寮忓啓涓€涓嬶紵
浜屻€?/strong>RMS Norm 綃?錛堝潎鏂規(guī)牴 Norm錛?/strong>
2.1 RMS Norm 鐨勮綆楀叕寮忓啓涓€涓嬶紵
2.2 RMS Norm 鐩告瘮浜?/strong> Layer Norm 鏈変粈涔堢壒鐐癸紵
RMS Norm 綆€鍖栦簡(jiǎn) Layer Norm 錛屽幓闄ゆ帀璁$畻鍧囧€艱繘琛屽鉤縐葷殑閮ㄥ垎銆?/span>
瀵規(guī)瘮LN錛?/span>RMS Norm鐨勮綆楅€熷害鏇村揩銆傛晥鏋滃熀鏈浉褰擄紝鐢氳嚦鐣ユ湁鎻愬崌銆?/span>
涓夈€?/strong>Deep Norm 綃?/strong>
3.1 Deep Norm 鎬濊礬錛?/strong>
Deep Norm鏂規(guī)硶鍦ㄦ墽琛?/span>Layer Norm涔嬪墠錛?/span>up-scale浜?jiǎn)娈?huà)宸繛鎺?/span> (alpha>1)錛涘彟澶栵紝鍦ㄥ垵濮嬪寲闃舵down-scale浜?jiǎn)妯?/span>
鍨嬪弬鏁?/span>(beta<1)銆?/span>
3.2 鍐欎竴涓?/strong> Deep Norm 浠g爜瀹炵幇錛?/strong>
|