Instructions to use palmfuture/Qwen3.8-27B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Local Apps Settings

How to use palmfuture/Qwen3.8-27B-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "palmfuture/Qwen3.8-27B-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.8-27B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/palmfuture/Qwen3.8-27B-GPTQ-Int4

SGLang

How to use palmfuture/Qwen3.8-27B-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "palmfuture/Qwen3.8-27B-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.8-27B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "palmfuture/Qwen3.8-27B-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.8-27B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use palmfuture/Qwen3.8-27B-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/palmfuture/Qwen3.8-27B-GPTQ-Int4
```

Qwen3.8-27B-GPTQ-Int4 / quant_log.csv

palmfuture

Re-quantize with Thai calibration (60 Thai samples) + official Qwen3.8 tokenizer; backfilled mtp.* into index

8fd3b71 verified 3 months ago

Raw History Blame Contribute Delete

19.2 kB

layer,module,loss,samples,damp,time
0,linear_attn.in_proj_qkv,0.0003368612,0.05000,2.241
0,linear_attn.in_proj_z,0.0002161865,0.05000,1.935
0,linear_attn.out_proj,0.0000001807,0.05000,2.306
0,mlp.up_proj,0.0000019150,0.05000,2.592
0,mlp.gate_proj,0.0000020682,0.05000,2.739
0,mlp.down_proj,0.0000000214,0.05000,9.331
1,linear_attn.in_proj_qkv,0.0000155794,0.05000,1.837
1,linear_attn.in_proj_z,0.0000095438,0.05000,1.921
1,linear_attn.out_proj,0.0000000712,0.05000,2.304
1,mlp.gate_proj,0.0000045172,0.05000,2.787
1,mlp.up_proj,0.0000043109,0.05000,2.766
1,mlp.down_proj,0.0000000344,0.05000,9.505
2,linear_attn.in_proj_qkv,0.0000233371,0.05000,2.025
2,linear_attn.in_proj_z,0.0000140999,0.05000,1.871
2,linear_attn.out_proj,0.0000001037,0.05000,2.299
2,mlp.up_proj,0.0000079759,0.05000,2.765
2,mlp.gate_proj,0.0000084705,0.05000,2.862
2,mlp.down_proj,0.0000001229,0.05000,9.616
3,self_attn.k_proj,0.0000173189,0.05000,3.799
3,self_attn.v_proj,0.0000173191,0.05000,3.854
3,self_attn.q_proj,0.0002429218,0.05000,3.987
3,self_attn.o_proj,0.0000001062,0.05000,2.506
3,mlp.gate_proj,0.0000102965,0.05000,2.686
3,mlp.up_proj,0.0000099161,0.05000,2.798
3,mlp.down_proj,0.0000000954,0.05000,9.555
4,linear_attn.in_proj_qkv,0.0000326377,0.05000,1.977
4,linear_attn.in_proj_z,0.0000207677,0.05000,1.802
4,linear_attn.out_proj,0.0000001566,0.05000,2.534
4,mlp.up_proj,0.0000133052,0.05000,2.883
4,mlp.gate_proj,0.0000137436,0.05000,2.941
4,mlp.down_proj,0.0000001336,0.05000,9.328
5,linear_attn.in_proj_qkv,0.0000416407,0.05000,2.109
5,linear_attn.in_proj_z,0.0000270064,0.05000,1.975
5,linear_attn.out_proj,0.0000002095,0.05000,2.415
5,mlp.gate_proj,0.0000176608,0.05000,2.717
5,mlp.up_proj,0.0000166580,0.05000,2.812
5,mlp.down_proj,0.0000002073,0.05000,9.511
6,linear_attn.in_proj_qkv,0.0000611388,0.05000,2.120
6,linear_attn.in_proj_z,0.0000364240,0.05000,1.807
6,linear_attn.out_proj,0.0000002775,0.05000,2.352
6,mlp.up_proj,0.0000216415,0.05000,3.333
6,mlp.gate_proj,0.0000231893,0.05000,3.383
6,mlp.down_proj,0.0000003309,0.05000,9.251
7,self_attn.v_proj,0.0000148674,0.05000,3.699
7,self_attn.k_proj,0.0000158531,0.05000,3.816
7,self_attn.q_proj,0.0002016444,0.05000,3.953
7,self_attn.o_proj,0.0000001799,0.05000,2.435
7,mlp.gate_proj,0.0000262628,0.05000,2.759
7,mlp.up_proj,0.0000245137,0.05000,2.858
7,mlp.down_proj,0.0000003872,0.05000,9.828
8,linear_attn.in_proj_qkv,0.0000634480,0.05000,1.994
8,linear_attn.in_proj_z,0.0000399709,0.05000,1.994
8,linear_attn.out_proj,0.0000003959,0.05000,2.335
8,mlp.up_proj,0.0000273954,0.05000,2.748
8,mlp.gate_proj,0.0000291099,0.05000,2.848
8,mlp.down_proj,0.0000004538,0.05000,9.649
9,linear_attn.in_proj_qkv,0.0000653086,0.05000,1.925
9,linear_attn.in_proj_z,0.0000397026,0.05000,2.071
9,linear_attn.out_proj,0.0000004228,0.05000,2.493
9,mlp.up_proj,0.0000296803,0.05000,2.737
9,mlp.gate_proj,0.0000313414,0.05000,2.830
9,mlp.down_proj,0.0000005040,0.05000,9.339
10,linear_attn.in_proj_qkv,0.0000675400,0.05000,2.034
10,linear_attn.in_proj_z,0.0000399367,0.05000,1.949
10,linear_attn.out_proj,0.0000004706,0.05000,2.441
10,mlp.gate_proj,0.0000315122,0.05000,2.953
10,mlp.up_proj,0.0000300201,0.05000,2.959
10,mlp.down_proj,0.0000005424,0.05000,9.450
11,self_attn.v_proj,0.0000137915,0.05000,3.864
11,self_attn.k_proj,0.0000148768,0.05000,3.911
11,self_attn.q_proj,0.0001777447,0.05000,4.005
11,self_attn.o_proj,0.0000002979,0.05000,2.480
11,mlp.up_proj,0.0000317410,0.05000,2.748
11,mlp.gate_proj,0.0000331847,0.05000,2.749
11,mlp.down_proj,0.0000006230,0.05000,9.330
12,linear_attn.in_proj_qkv,0.0000752975,0.05000,1.996
12,linear_attn.in_proj_z,0.0000445512,0.05000,1.923
12,linear_attn.out_proj,0.0000006823,0.05000,2.228
12,mlp.up_proj,0.0000328472,0.05000,2.789
12,mlp.gate_proj,0.0000343769,0.05000,2.839
12,mlp.down_proj,0.0000007140,0.05000,9.529
13,linear_attn.in_proj_qkv,0.0000792920,0.05000,2.077
13,linear_attn.in_proj_z,0.0000446594,0.05000,1.928
13,linear_attn.out_proj,0.0000007346,0.05000,2.322
13,mlp.gate_proj,0.0000368082,0.05000,3.155
13,mlp.up_proj,0.0000351891,0.05000,3.251
13,mlp.down_proj,0.0000008116,0.05000,9.204
14,linear_attn.in_proj_qkv,0.0000863997,0.05000,2.020
14,linear_attn.in_proj_z,0.0000501994,0.05000,1.914
14,linear_attn.out_proj,0.0000008711,0.05000,2.414
14,mlp.up_proj,0.0000371291,0.05000,2.753
14,mlp.gate_proj,0.0000387172,0.05000,2.797
14,mlp.down_proj,0.0000008954,0.05000,9.483
15,self_attn.k_proj,0.0000132157,0.05000,3.811
15,self_attn.v_proj,0.0000130554,0.05000,3.910
15,self_attn.q_proj,0.0001511661,0.05000,4.006
15,self_attn.o_proj,0.0000004878,0.05000,2.336
15,mlp.gate_proj,0.0000436905,0.05000,2.710
15,mlp.up_proj,0.0000422585,0.05000,2.713
15,mlp.down_proj,0.0000010513,0.05000,9.351
16,linear_attn.in_proj_qkv,0.0000917909,0.05000,1.968
16,linear_attn.in_proj_z,0.0000523076,0.05000,2.030
16,linear_attn.out_proj,0.0000009318,0.05000,2.450
16,mlp.gate_proj,0.0000481896,0.05000,2.843
16,mlp.up_proj,0.0000463708,0.05000,2.898
16,mlp.down_proj,0.0000012755,0.05000,9.592
17,linear_attn.in_proj_qkv,0.0001043242,0.05000,1.991
17,linear_attn.in_proj_z,0.0000535001,0.05000,1.919
17,linear_attn.out_proj,0.0000012644,0.05000,2.200
17,mlp.up_proj,0.0000533127,0.05000,2.742
17,mlp.gate_proj,0.0000554380,0.05000,2.888
17,mlp.down_proj,0.0000017082,0.05000,9.680
18,linear_attn.in_proj_qkv,0.0001136891,0.05000,1.993
18,linear_attn.in_proj_z,0.0000628993,0.05000,1.917
18,linear_attn.out_proj,0.0000015056,0.05000,2.480
18,mlp.up_proj,0.0000634088,0.05000,2.813
18,mlp.gate_proj,0.0000674569,0.05000,2.868
18,mlp.down_proj,0.0000027068,0.05000,9.709
19,self_attn.k_proj,0.0000166268,0.05000,3.937
19,self_attn.v_proj,0.0000189011,0.05000,3.965
19,self_attn.q_proj,0.0001865257,0.05000,3.992
19,self_attn.o_proj,0.0000019393,0.05000,2.402
19,mlp.up_proj,0.0000810440,0.05000,2.662
19,mlp.gate_proj,0.0000855590,0.05000,2.747
19,mlp.down_proj,0.0000037328,0.05000,9.433
20,linear_attn.in_proj_qkv,0.0002039232,0.05000,2.144
20,linear_attn.in_proj_z,0.0001151243,0.05000,1.996
20,linear_attn.out_proj,0.0000025767,0.05000,2.331
20,mlp.up_proj,0.0000985494,0.05000,2.907
20,mlp.gate_proj,0.0001057696,0.05000,3.172
20,mlp.down_proj,0.0000044256,0.05000,9.291
21,linear_attn.in_proj_qkv,0.0002524438,0.05000,1.994
21,linear_attn.in_proj_z,0.0001616186,0.05000,1.874
21,linear_attn.out_proj,0.0000033764,0.05000,2.406
21,mlp.up_proj,0.0001090591,0.05000,2.887
21,mlp.gate_proj,0.0001163682,0.05000,2.889
21,mlp.down_proj,0.0000048190,0.05000,9.750
22,linear_attn.in_proj_qkv,0.0002218586,0.05000,2.049
22,linear_attn.in_proj_z,0.0001359311,0.05000,1.984
22,linear_attn.out_proj,0.0000031948,0.05000,2.406
22,mlp.gate_proj,0.0001366316,0.05000,2.844
22,mlp.up_proj,0.0001217416,0.05000,2.862
22,mlp.down_proj,0.0000054926,0.05000,9.431
23,self_attn.v_proj,0.0000261538,0.05000,3.910
23,self_attn.k_proj,0.0000240889,0.05000,3.960
23,self_attn.q_proj,0.0002466288,0.05000,4.048
23,self_attn.o_proj,0.0000025475,0.05000,2.350
23,mlp.gate_proj,0.0001418136,0.05000,2.799
23,mlp.up_proj,0.0001317739,0.05000,2.805
23,mlp.down_proj,0.0000056953,0.05000,9.682
24,linear_attn.in_proj_qkv,0.0002171155,0.05000,1.960
24,linear_attn.in_proj_z,0.0001230112,0.05000,1.877
24,linear_attn.out_proj,0.0000038757,0.05000,2.360
24,mlp.gate_proj,0.0001553255,0.05000,2.868
24,mlp.up_proj,0.0001441100,0.05000,2.875
24,mlp.down_proj,0.0000068346,0.05000,9.680
25,linear_attn.in_proj_qkv,0.0002227245,0.05000,2.002
25,linear_attn.in_proj_z,0.0001177830,0.05000,1.969
25,linear_attn.out_proj,0.0000040613,0.05000,2.318
25,mlp.gate_proj,0.0001498546,0.05000,2.806
25,mlp.up_proj,0.0001460494,0.05000,2.839
25,mlp.down_proj,0.0000068895,0.05000,9.662
26,linear_attn.in_proj_qkv,0.0002012057,0.05000,2.061
26,linear_attn.in_proj_z,0.0001023811,0.05000,2.082
26,linear_attn.out_proj,0.0000043126,0.05000,2.332
26,mlp.up_proj,0.0001428793,0.05000,2.798
26,mlp.gate_proj,0.0001395380,0.05000,2.837
26,mlp.down_proj,0.0000072643,0.05000,9.651
27,self_attn.v_proj,0.0000215405,0.05000,3.914
27,self_attn.k_proj,0.0000227641,0.05000,3.969
27,self_attn.q_proj,0.0002286453,0.05000,4.034
27,self_attn.o_proj,0.0000055269,0.05000,2.383
27,mlp.up_proj,0.0001534065,0.05000,2.776
27,mlp.gate_proj,0.0001449911,0.05000,2.807
27,mlp.down_proj,0.0000078070,0.05000,9.142
28,linear_attn.in_proj_qkv,0.0001981716,0.05000,1.882
28,linear_attn.in_proj_z,0.0000941573,0.05000,2.102
28,linear_attn.out_proj,0.0000055053,0.05000,2.342
28,mlp.up_proj,0.0001502735,0.05000,2.673
28,mlp.gate_proj,0.0001394303,0.05000,3.035
28,mlp.down_proj,0.0000079834,0.05000,9.409
29,linear_attn.in_proj_qkv,0.0002155416,0.05000,1.972
29,linear_attn.in_proj_z,0.0000993961,0.05000,1.883
29,linear_attn.out_proj,0.0000050188,0.05000,2.338
29,mlp.up_proj,0.0001521553,0.05000,2.986
29,mlp.gate_proj,0.0001403565,0.05000,3.014
29,mlp.down_proj,0.0000080845,0.05000,9.294
30,linear_attn.in_proj_qkv,0.0002148425,0.05000,2.097
30,linear_attn.in_proj_z,0.0001015619,0.05000,2.084
30,linear_attn.out_proj,0.0000061994,0.05000,2.395
30,mlp.gate_proj,0.0001370147,0.05000,2.795
30,mlp.up_proj,0.0001523059,0.05000,2.857
30,mlp.down_proj,0.0000082668,0.05000,9.676
31,self_attn.k_proj,0.0000231682,0.05000,3.895
31,self_attn.v_proj,0.0000273828,0.05000,3.996
31,self_attn.q_proj,0.0002229610,0.05000,4.106
31,self_attn.o_proj,0.0000073854,0.05000,2.383
31,mlp.up_proj,0.0001673358,0.05000,2.796
31,mlp.gate_proj,0.0001468095,0.05000,2.854
31,mlp.down_proj,0.0000091737,0.05000,9.578
32,linear_attn.in_proj_qkv,0.0002296537,0.05000,2.166
32,linear_attn.in_proj_z,0.0001012839,0.05000,1.941
32,linear_attn.out_proj,0.0000065890,0.05000,2.337
32,mlp.up_proj,0.0001791505,0.05000,2.758
32,mlp.gate_proj,0.0001558562,0.05000,2.892
32,mlp.down_proj,0.0000099460,0.05000,9.649
33,linear_attn.in_proj_qkv,0.0002735911,0.05000,2.182
33,linear_attn.in_proj_z,0.0001131769,0.05000,1.949
33,linear_attn.out_proj,0.0000076368,0.05000,2.344
33,mlp.up_proj,0.0001883489,0.05000,2.766
33,mlp.gate_proj,0.0001620953,0.05000,2.871
33,mlp.down_proj,0.0000102629,0.05000,9.698
34,linear_attn.in_proj_qkv,0.0002683308,0.05000,2.014
34,linear_attn.in_proj_z,0.0001216698,0.05000,1.799
34,linear_attn.out_proj,0.0000077787,0.05000,2.232
34,mlp.up_proj,0.0002016926,0.05000,3.033
34,mlp.gate_proj,0.0001748909,0.05000,3.043
34,mlp.down_proj,0.0000134868,0.05000,9.760
35,self_attn.k_proj,0.0000294353,0.05000,3.739
35,self_attn.v_proj,0.0000423436,0.05000,3.832
35,self_attn.q_proj,0.0002818952,0.05000,3.964
35,self_attn.o_proj,0.0000129746,0.05000,2.268
35,mlp.up_proj,0.0002112278,0.05000,2.812
35,mlp.gate_proj,0.0001868287,0.05000,2.878
35,mlp.down_proj,0.0000149566,0.05000,9.501
36,linear_attn.in_proj_qkv,0.0003793859,0.05000,1.995
36,linear_attn.in_proj_z,0.0001906052,0.05000,2.056
36,linear_attn.out_proj,0.0000076288,0.05000,2.398
36,mlp.gate_proj,0.0002082069,0.05000,2.823
36,mlp.up_proj,0.0002087939,0.05000,2.846
36,mlp.down_proj,0.0000134904,0.05000,9.323
37,linear_attn.in_proj_qkv,0.0004087727,0.05000,2.040
37,linear_attn.in_proj_z,0.0002336417,0.05000,1.937
37,linear_attn.out_proj,0.0000079233,0.05000,2.371
37,mlp.up_proj,0.0002071794,0.05000,2.924
37,mlp.gate_proj,0.0002113439,0.05000,2.940
37,mlp.down_proj,0.0000126032,0.05000,9.261
38,linear_attn.in_proj_qkv,0.0003641806,0.05000,1.933
38,linear_attn.in_proj_z,0.0002126180,0.05000,1.927
38,linear_attn.out_proj,0.0000062375,0.05000,2.351
38,mlp.gate_proj,0.0002730919,0.05000,2.693
38,mlp.up_proj,0.0002319409,0.05000,3.058
38,mlp.down_proj,0.0000126153,0.05000,9.272
39,self_attn.q_proj,0.0003205566,0.05000,3.990
39,self_attn.v_proj,0.0000426120,0.05000,3.989
39,self_attn.k_proj,0.0000334579,0.05000,3.999
39,self_attn.o_proj,0.0000080916,0.05000,2.274
39,mlp.up_proj,0.0002231741,0.05000,2.852
39,mlp.gate_proj,0.0002467629,0.05000,2.966
39,mlp.down_proj,0.0000115561,0.05000,9.532
40,linear_attn.in_proj_qkv,0.0003428894,0.05000,1.878
40,linear_attn.in_proj_z,0.0001902024,0.05000,1.974
40,linear_attn.out_proj,0.0000068595,0.05000,2.359
40,mlp.up_proj,0.0002364197,0.05000,2.828
40,mlp.gate_proj,0.0002664747,0.05000,2.845
40,mlp.down_proj,0.0000120810,0.05000,9.456
41,linear_attn.in_proj_qkv,0.0003137499,0.05000,2.138
41,linear_attn.in_proj_z,0.0001716255,0.05000,1.880
41,linear_attn.out_proj,0.0000059280,0.05000,2.473
41,mlp.up_proj,0.0002241652,0.05000,2.721
41,mlp.gate_proj,0.0002274599,0.05000,2.739
41,mlp.down_proj,0.0000115259,0.05000,9.483
42,linear_attn.in_proj_qkv,0.0002912149,0.05000,1.879
42,linear_attn.in_proj_z,0.0001538357,0.05000,2.045
42,linear_attn.out_proj,0.0000066113,0.05000,2.447
42,mlp.gate_proj,0.0002051962,0.05000,2.746
42,mlp.up_proj,0.0002163007,0.05000,2.847
42,mlp.down_proj,0.0000123591,0.05000,9.604
43,self_attn.k_proj,0.0000328653,0.05000,3.945
43,self_attn.v_proj,0.0000419066,0.05000,3.974
43,self_attn.q_proj,0.0003056384,0.05000,4.065
43,self_attn.o_proj,0.0000111928,0.05000,2.405
43,mlp.up_proj,0.0002239182,0.05000,2.926
43,mlp.gate_proj,0.0002035338,0.05000,2.931
43,mlp.down_proj,0.0000135572,0.05000,9.248
44,linear_attn.in_proj_qkv,0.0002801879,0.05000,2.008
44,linear_attn.in_proj_z,0.0001365148,0.05000,1.790
44,linear_attn.out_proj,0.0000092356,0.05000,2.407
44,mlp.up_proj,0.0002258869,0.05000,2.809
44,mlp.gate_proj,0.0002031547,0.05000,2.878
44,mlp.down_proj,0.0000151829,0.05000,9.601
45,linear_attn.in_proj_qkv,0.0002850569,0.05000,1.954
45,linear_attn.in_proj_z,0.0001386788,0.05000,1.806
45,linear_attn.out_proj,0.0000080556,0.05000,2.429
45,mlp.gate_proj,0.0002044231,0.05000,2.805
45,mlp.up_proj,0.0002285339,0.05000,3.046
45,mlp.down_proj,0.0000157144,0.05000,9.286
46,linear_attn.in_proj_qkv,0.0002990598,0.05000,1.973
46,linear_attn.in_proj_z,0.0001469002,0.05000,1.927
46,linear_attn.out_proj,0.0000117627,0.05000,2.395
46,mlp.gate_proj,0.0002012278,0.05000,2.789
46,mlp.up_proj,0.0002300471,0.05000,2.827
46,mlp.down_proj,0.0000171396,0.05000,9.654
47,self_attn.k_proj,0.0000347433,0.05000,3.852
47,self_attn.v_proj,0.0000558419,0.05000,3.955
47,self_attn.q_proj,0.0003292233,0.05000,4.001
47,self_attn.o_proj,0.0000132204,0.05000,2.237
47,mlp.up_proj,0.0002564880,0.05000,2.770
47,mlp.gate_proj,0.0002245832,0.05000,2.791
47,mlp.down_proj,0.0000213101,0.05000,9.215
48,linear_attn.in_proj_qkv,0.0003306475,0.05000,2.080
48,linear_attn.in_proj_z,0.0001571578,0.05000,1.962
48,linear_attn.out_proj,0.0000139544,0.05000,2.476
48,mlp.up_proj,0.0002781727,0.05000,2.887
48,mlp.gate_proj,0.0002438090,0.05000,2.900
48,mlp.down_proj,0.0000261321,0.05000,9.646
49,linear_attn.in_proj_qkv,0.0003951708,0.05000,2.102
49,linear_attn.in_proj_z,0.0001682103,0.05000,2.024
49,linear_attn.out_proj,0.0000192123,0.05000,2.427
49,mlp.gate_proj,0.0002650328,0.05000,2.760
49,mlp.up_proj,0.0003012208,0.05000,2.796
49,mlp.down_proj,0.0000339468,0.05000,9.749
50,linear_attn.in_proj_qkv,0.0004090191,0.05000,1.981
50,linear_attn.in_proj_z,0.0001887876,0.05000,1.814
50,linear_attn.out_proj,0.0000276691,0.05000,2.365
50,mlp.up_proj,0.0003447875,0.05000,2.891
50,mlp.gate_proj,0.0003162317,0.05000,2.927
50,mlp.down_proj,0.0000570232,0.05000,9.744
51,self_attn.q_proj,0.0004390058,0.05000,3.946
51,self_attn.v_proj,0.0001044389,0.05000,3.971
51,self_attn.k_proj,0.0000458526,0.05000,3.982
51,self_attn.o_proj,0.0000393520,0.05000,2.430
51,mlp.up_proj,0.0003995472,0.05000,2.695
51,mlp.gate_proj,0.0003556540,0.05000,2.800
51,mlp.down_proj,0.0000732886,0.05000,9.655
52,linear_attn.in_proj_qkv,0.0006285047,0.05000,2.038
52,linear_attn.in_proj_z,0.0002888460,0.05000,1.926
52,linear_attn.out_proj,0.0000387001,0.05000,2.393
52,mlp.up_proj,0.0004349370,0.05000,2.843
52,mlp.gate_proj,0.0004419015,0.05000,2.924
52,mlp.down_proj,0.0000731234,0.05000,9.152
53,linear_attn.in_proj_qkv,0.0006293647,0.05000,2.033
53,linear_attn.in_proj_z,0.0003074905,0.05000,1.847
53,linear_attn.out_proj,0.0000427141,0.05000,2.380
53,mlp.gate_proj,0.0004943917,0.05000,2.842
53,mlp.up_proj,0.0004622656,0.05000,2.919
53,mlp.down_proj,0.0000799335,0.05000,9.762
54,linear_attn.in_proj_qkv,0.0006097221,0.05000,2.126
54,linear_attn.in_proj_z,0.0003273230,0.05000,1.937
54,linear_attn.out_proj,0.0000440077,0.05000,2.526
54,mlp.up_proj,0.0005474443,0.05000,2.852
54,mlp.gate_proj,0.0006205177,0.05000,2.870
54,mlp.down_proj,0.0001119038,0.05000,9.327
55,self_attn.q_proj,0.0005899858,0.05000,3.978
55,self_attn.k_proj,0.0000637752,0.05000,4.002
55,self_attn.v_proj,0.0001395592,0.05000,4.035
55,self_attn.o_proj,0.0000379650,0.05000,2.496
55,mlp.up_proj,0.0005508717,0.05000,2.840
55,mlp.gate_proj,0.0005990099,0.05000,2.841
55,mlp.down_proj,0.0000952430,0.05000,9.324
56,linear_attn.in_proj_qkv,0.0007110504,0.05000,2.040
56,linear_attn.in_proj_z,0.0003553513,0.05000,1.851
56,linear_attn.out_proj,0.0000582206,0.05000,2.566
56,mlp.gate_proj,0.0006629710,0.05000,2.838
56,mlp.up_proj,0.0005916605,0.05000,2.924
56,mlp.down_proj,0.0000945713,0.05000,9.728
57,linear_attn.in_proj_qkv,0.0007448315,0.05000,1.959
57,linear_attn.in_proj_z,0.0003739022,0.05000,1.902
57,linear_attn.out_proj,0.0000515008,0.05000,2.429
57,mlp.gate_proj,0.0007721504,0.05000,2.871
57,mlp.up_proj,0.0006770504,0.05000,2.884
57,mlp.down_proj,0.0000980713,0.05000,9.658
58,linear_attn.in_proj_qkv,0.0007159035,0.05000,2.370
58,linear_attn.in_proj_z,0.0003891521,0.05000,2.025
58,linear_attn.out_proj,0.0000555304,0.05000,2.422
58,mlp.gate_proj,0.0008880539,0.05000,2.742
58,mlp.up_proj,0.0007787612,0.05000,2.753
58,mlp.down_proj,0.0001309379,0.05000,9.574
59,self_attn.k_proj,0.0000801485,0.05000,3.907
59,self_attn.v_proj,0.0002913585,0.05000,3.955
59,self_attn.q_proj,0.0007040162,0.05000,4.042
59,self_attn.o_proj,0.0001141125,0.05000,2.312
59,mlp.gate_proj,0.0008777454,0.05000,2.820
59,mlp.up_proj,0.0007892898,0.05000,2.884
59,mlp.down_proj,0.0001530582,0.05000,9.259
60,linear_attn.in_proj_qkv,0.0008737440,0.05000,2.028
60,linear_attn.in_proj_z,0.0004170820,0.05000,1.903
60,linear_attn.out_proj,0.0001280804,0.05000,2.311
60,mlp.up_proj,0.0008007702,0.05000,2.858
60,mlp.gate_proj,0.0008728313,0.05000,3.094
60,mlp.down_proj,0.0001893005,0.05000,9.602
61,linear_attn.in_proj_qkv,0.0006409717,0.05000,1.929
61,linear_attn.in_proj_z,0.0003683010,0.05000,1.809
61,linear_attn.out_proj,0.0001176246,0.05000,2.279
61,mlp.up_proj,0.0008527413,0.05000,2.939
61,mlp.gate_proj,0.0009217091,0.05000,2.946
61,mlp.down_proj,0.0002487219,0.05000,9.279
62,linear_attn.in_proj_qkv,0.0007001918,0.05000,2.108
62,linear_attn.in_proj_z,0.0003716715,0.05000,1.928
62,linear_attn.out_proj,0.0002993706,0.05000,2.350
62,mlp.up_proj,0.0007859706,0.05000,2.771
62,mlp.gate_proj,0.0008442585,0.05000,2.820
62,mlp.down_proj,0.0003750678,0.05000,9.371
63,self_attn.k_proj,0.0000722345,0.05000,3.880
63,self_attn.v_proj,0.0002261060,0.05000,3.961
63,self_attn.q_proj,0.0006227864,0.05000,4.029
63,self_attn.o_proj,0.0002908754,0.05000,2.263
63,mlp.gate_proj,0.0007084487,0.05000,2.730
63,mlp.up_proj,0.0006370316,0.05000,2.834
63,mlp.down_proj,0.0007829439,0.05000,9.542

	layer,module,loss,samples,damp,time
	0,linear_attn.in_proj_qkv,0.0003368612,0.05000,2.241
	0,linear_attn.in_proj_z,0.0002161865,0.05000,1.935
	0,linear_attn.out_proj,0.0000001807,0.05000,2.306
	0,mlp.up_proj,0.0000019150,0.05000,2.592
	0,mlp.gate_proj,0.0000020682,0.05000,2.739
	0,mlp.down_proj,0.0000000214,0.05000,9.331
	1,linear_attn.in_proj_qkv,0.0000155794,0.05000,1.837
	1,linear_attn.in_proj_z,0.0000095438,0.05000,1.921
	1,linear_attn.out_proj,0.0000000712,0.05000,2.304
	1,mlp.gate_proj,0.0000045172,0.05000,2.787
	1,mlp.up_proj,0.0000043109,0.05000,2.766
	1,mlp.down_proj,0.0000000344,0.05000,9.505
	2,linear_attn.in_proj_qkv,0.0000233371,0.05000,2.025
	2,linear_attn.in_proj_z,0.0000140999,0.05000,1.871
	2,linear_attn.out_proj,0.0000001037,0.05000,2.299
	2,mlp.up_proj,0.0000079759,0.05000,2.765
	2,mlp.gate_proj,0.0000084705,0.05000,2.862
	2,mlp.down_proj,0.0000001229,0.05000,9.616
	3,self_attn.k_proj,0.0000173189,0.05000,3.799
	3,self_attn.v_proj,0.0000173191,0.05000,3.854
	3,self_attn.q_proj,0.0002429218,0.05000,3.987
	3,self_attn.o_proj,0.0000001062,0.05000,2.506
	3,mlp.gate_proj,0.0000102965,0.05000,2.686
	3,mlp.up_proj,0.0000099161,0.05000,2.798
	3,mlp.down_proj,0.0000000954,0.05000,9.555
	4,linear_attn.in_proj_qkv,0.0000326377,0.05000,1.977
	4,linear_attn.in_proj_z,0.0000207677,0.05000,1.802
	4,linear_attn.out_proj,0.0000001566,0.05000,2.534
	4,mlp.up_proj,0.0000133052,0.05000,2.883
	4,mlp.gate_proj,0.0000137436,0.05000,2.941
	4,mlp.down_proj,0.0000001336,0.05000,9.328
	5,linear_attn.in_proj_qkv,0.0000416407,0.05000,2.109
	5,linear_attn.in_proj_z,0.0000270064,0.05000,1.975
	5,linear_attn.out_proj,0.0000002095,0.05000,2.415
	5,mlp.gate_proj,0.0000176608,0.05000,2.717
	5,mlp.up_proj,0.0000166580,0.05000,2.812
	5,mlp.down_proj,0.0000002073,0.05000,9.511
	6,linear_attn.in_proj_qkv,0.0000611388,0.05000,2.120
	6,linear_attn.in_proj_z,0.0000364240,0.05000,1.807
	6,linear_attn.out_proj,0.0000002775,0.05000,2.352
	6,mlp.up_proj,0.0000216415,0.05000,3.333
	6,mlp.gate_proj,0.0000231893,0.05000,3.383
	6,mlp.down_proj,0.0000003309,0.05000,9.251
	7,self_attn.v_proj,0.0000148674,0.05000,3.699
	7,self_attn.k_proj,0.0000158531,0.05000,3.816
	7,self_attn.q_proj,0.0002016444,0.05000,3.953
	7,self_attn.o_proj,0.0000001799,0.05000,2.435
	7,mlp.gate_proj,0.0000262628,0.05000,2.759
	7,mlp.up_proj,0.0000245137,0.05000,2.858
	7,mlp.down_proj,0.0000003872,0.05000,9.828
	8,linear_attn.in_proj_qkv,0.0000634480,0.05000,1.994
	8,linear_attn.in_proj_z,0.0000399709,0.05000,1.994
	8,linear_attn.out_proj,0.0000003959,0.05000,2.335
	8,mlp.up_proj,0.0000273954,0.05000,2.748
	8,mlp.gate_proj,0.0000291099,0.05000,2.848
	8,mlp.down_proj,0.0000004538,0.05000,9.649
	9,linear_attn.in_proj_qkv,0.0000653086,0.05000,1.925
	9,linear_attn.in_proj_z,0.0000397026,0.05000,2.071
	9,linear_attn.out_proj,0.0000004228,0.05000,2.493
	9,mlp.up_proj,0.0000296803,0.05000,2.737
	9,mlp.gate_proj,0.0000313414,0.05000,2.830
	9,mlp.down_proj,0.0000005040,0.05000,9.339
	10,linear_attn.in_proj_qkv,0.0000675400,0.05000,2.034
	10,linear_attn.in_proj_z,0.0000399367,0.05000,1.949
	10,linear_attn.out_proj,0.0000004706,0.05000,2.441
	10,mlp.gate_proj,0.0000315122,0.05000,2.953
	10,mlp.up_proj,0.0000300201,0.05000,2.959
	10,mlp.down_proj,0.0000005424,0.05000,9.450
	11,self_attn.v_proj,0.0000137915,0.05000,3.864
	11,self_attn.k_proj,0.0000148768,0.05000,3.911
	11,self_attn.q_proj,0.0001777447,0.05000,4.005
	11,self_attn.o_proj,0.0000002979,0.05000,2.480
	11,mlp.up_proj,0.0000317410,0.05000,2.748
	11,mlp.gate_proj,0.0000331847,0.05000,2.749
	11,mlp.down_proj,0.0000006230,0.05000,9.330
	12,linear_attn.in_proj_qkv,0.0000752975,0.05000,1.996
	12,linear_attn.in_proj_z,0.0000445512,0.05000,1.923
	12,linear_attn.out_proj,0.0000006823,0.05000,2.228
	12,mlp.up_proj,0.0000328472,0.05000,2.789
	12,mlp.gate_proj,0.0000343769,0.05000,2.839
	12,mlp.down_proj,0.0000007140,0.05000,9.529
	13,linear_attn.in_proj_qkv,0.0000792920,0.05000,2.077
	13,linear_attn.in_proj_z,0.0000446594,0.05000,1.928
	13,linear_attn.out_proj,0.0000007346,0.05000,2.322
	13,mlp.gate_proj,0.0000368082,0.05000,3.155
	13,mlp.up_proj,0.0000351891,0.05000,3.251
	13,mlp.down_proj,0.0000008116,0.05000,9.204
	14,linear_attn.in_proj_qkv,0.0000863997,0.05000,2.020
	14,linear_attn.in_proj_z,0.0000501994,0.05000,1.914
	14,linear_attn.out_proj,0.0000008711,0.05000,2.414
	14,mlp.up_proj,0.0000371291,0.05000,2.753
	14,mlp.gate_proj,0.0000387172,0.05000,2.797
	14,mlp.down_proj,0.0000008954,0.05000,9.483
	15,self_attn.k_proj,0.0000132157,0.05000,3.811
	15,self_attn.v_proj,0.0000130554,0.05000,3.910
	15,self_attn.q_proj,0.0001511661,0.05000,4.006
	15,self_attn.o_proj,0.0000004878,0.05000,2.336
	15,mlp.gate_proj,0.0000436905,0.05000,2.710
	15,mlp.up_proj,0.0000422585,0.05000,2.713
	15,mlp.down_proj,0.0000010513,0.05000,9.351
	16,linear_attn.in_proj_qkv,0.0000917909,0.05000,1.968
	16,linear_attn.in_proj_z,0.0000523076,0.05000,2.030
	16,linear_attn.out_proj,0.0000009318,0.05000,2.450
	16,mlp.gate_proj,0.0000481896,0.05000,2.843
	16,mlp.up_proj,0.0000463708,0.05000,2.898
	16,mlp.down_proj,0.0000012755,0.05000,9.592
	17,linear_attn.in_proj_qkv,0.0001043242,0.05000,1.991
	17,linear_attn.in_proj_z,0.0000535001,0.05000,1.919
	17,linear_attn.out_proj,0.0000012644,0.05000,2.200
	17,mlp.up_proj,0.0000533127,0.05000,2.742
	17,mlp.gate_proj,0.0000554380,0.05000,2.888
	17,mlp.down_proj,0.0000017082,0.05000,9.680
	18,linear_attn.in_proj_qkv,0.0001136891,0.05000,1.993
	18,linear_attn.in_proj_z,0.0000628993,0.05000,1.917
	18,linear_attn.out_proj,0.0000015056,0.05000,2.480
	18,mlp.up_proj,0.0000634088,0.05000,2.813
	18,mlp.gate_proj,0.0000674569,0.05000,2.868
	18,mlp.down_proj,0.0000027068,0.05000,9.709
	19,self_attn.k_proj,0.0000166268,0.05000,3.937
	19,self_attn.v_proj,0.0000189011,0.05000,3.965
	19,self_attn.q_proj,0.0001865257,0.05000,3.992
	19,self_attn.o_proj,0.0000019393,0.05000,2.402
	19,mlp.up_proj,0.0000810440,0.05000,2.662
	19,mlp.gate_proj,0.0000855590,0.05000,2.747
	19,mlp.down_proj,0.0000037328,0.05000,9.433
	20,linear_attn.in_proj_qkv,0.0002039232,0.05000,2.144
	20,linear_attn.in_proj_z,0.0001151243,0.05000,1.996
	20,linear_attn.out_proj,0.0000025767,0.05000,2.331
	20,mlp.up_proj,0.0000985494,0.05000,2.907
	20,mlp.gate_proj,0.0001057696,0.05000,3.172
	20,mlp.down_proj,0.0000044256,0.05000,9.291
	21,linear_attn.in_proj_qkv,0.0002524438,0.05000,1.994
	21,linear_attn.in_proj_z,0.0001616186,0.05000,1.874
	21,linear_attn.out_proj,0.0000033764,0.05000,2.406
	21,mlp.up_proj,0.0001090591,0.05000,2.887
	21,mlp.gate_proj,0.0001163682,0.05000,2.889
	21,mlp.down_proj,0.0000048190,0.05000,9.750
	22,linear_attn.in_proj_qkv,0.0002218586,0.05000,2.049
	22,linear_attn.in_proj_z,0.0001359311,0.05000,1.984
	22,linear_attn.out_proj,0.0000031948,0.05000,2.406
	22,mlp.gate_proj,0.0001366316,0.05000,2.844
	22,mlp.up_proj,0.0001217416,0.05000,2.862
	22,mlp.down_proj,0.0000054926,0.05000,9.431
	23,self_attn.v_proj,0.0000261538,0.05000,3.910
	23,self_attn.k_proj,0.0000240889,0.05000,3.960
	23,self_attn.q_proj,0.0002466288,0.05000,4.048
	23,self_attn.o_proj,0.0000025475,0.05000,2.350
	23,mlp.gate_proj,0.0001418136,0.05000,2.799
	23,mlp.up_proj,0.0001317739,0.05000,2.805
	23,mlp.down_proj,0.0000056953,0.05000,9.682
	24,linear_attn.in_proj_qkv,0.0002171155,0.05000,1.960
	24,linear_attn.in_proj_z,0.0001230112,0.05000,1.877
	24,linear_attn.out_proj,0.0000038757,0.05000,2.360
	24,mlp.gate_proj,0.0001553255,0.05000,2.868
	24,mlp.up_proj,0.0001441100,0.05000,2.875
	24,mlp.down_proj,0.0000068346,0.05000,9.680
	25,linear_attn.in_proj_qkv,0.0002227245,0.05000,2.002
	25,linear_attn.in_proj_z,0.0001177830,0.05000,1.969
	25,linear_attn.out_proj,0.0000040613,0.05000,2.318
	25,mlp.gate_proj,0.0001498546,0.05000,2.806
	25,mlp.up_proj,0.0001460494,0.05000,2.839
	25,mlp.down_proj,0.0000068895,0.05000,9.662
	26,linear_attn.in_proj_qkv,0.0002012057,0.05000,2.061
	26,linear_attn.in_proj_z,0.0001023811,0.05000,2.082
	26,linear_attn.out_proj,0.0000043126,0.05000,2.332
	26,mlp.up_proj,0.0001428793,0.05000,2.798
	26,mlp.gate_proj,0.0001395380,0.05000,2.837
	26,mlp.down_proj,0.0000072643,0.05000,9.651
	27,self_attn.v_proj,0.0000215405,0.05000,3.914
	27,self_attn.k_proj,0.0000227641,0.05000,3.969
	27,self_attn.q_proj,0.0002286453,0.05000,4.034
	27,self_attn.o_proj,0.0000055269,0.05000,2.383
	27,mlp.up_proj,0.0001534065,0.05000,2.776
	27,mlp.gate_proj,0.0001449911,0.05000,2.807
	27,mlp.down_proj,0.0000078070,0.05000,9.142
	28,linear_attn.in_proj_qkv,0.0001981716,0.05000,1.882
	28,linear_attn.in_proj_z,0.0000941573,0.05000,2.102
	28,linear_attn.out_proj,0.0000055053,0.05000,2.342
	28,mlp.up_proj,0.0001502735,0.05000,2.673
	28,mlp.gate_proj,0.0001394303,0.05000,3.035
	28,mlp.down_proj,0.0000079834,0.05000,9.409
	29,linear_attn.in_proj_qkv,0.0002155416,0.05000,1.972
	29,linear_attn.in_proj_z,0.0000993961,0.05000,1.883
	29,linear_attn.out_proj,0.0000050188,0.05000,2.338
	29,mlp.up_proj,0.0001521553,0.05000,2.986
	29,mlp.gate_proj,0.0001403565,0.05000,3.014
	29,mlp.down_proj,0.0000080845,0.05000,9.294
	30,linear_attn.in_proj_qkv,0.0002148425,0.05000,2.097
	30,linear_attn.in_proj_z,0.0001015619,0.05000,2.084
	30,linear_attn.out_proj,0.0000061994,0.05000,2.395
	30,mlp.gate_proj,0.0001370147,0.05000,2.795
	30,mlp.up_proj,0.0001523059,0.05000,2.857
	30,mlp.down_proj,0.0000082668,0.05000,9.676
	31,self_attn.k_proj,0.0000231682,0.05000,3.895
	31,self_attn.v_proj,0.0000273828,0.05000,3.996
	31,self_attn.q_proj,0.0002229610,0.05000,4.106
	31,self_attn.o_proj,0.0000073854,0.05000,2.383
	31,mlp.up_proj,0.0001673358,0.05000,2.796
	31,mlp.gate_proj,0.0001468095,0.05000,2.854
	31,mlp.down_proj,0.0000091737,0.05000,9.578
	32,linear_attn.in_proj_qkv,0.0002296537,0.05000,2.166
	32,linear_attn.in_proj_z,0.0001012839,0.05000,1.941
	32,linear_attn.out_proj,0.0000065890,0.05000,2.337
	32,mlp.up_proj,0.0001791505,0.05000,2.758
	32,mlp.gate_proj,0.0001558562,0.05000,2.892
	32,mlp.down_proj,0.0000099460,0.05000,9.649
	33,linear_attn.in_proj_qkv,0.0002735911,0.05000,2.182
	33,linear_attn.in_proj_z,0.0001131769,0.05000,1.949
	33,linear_attn.out_proj,0.0000076368,0.05000,2.344
	33,mlp.up_proj,0.0001883489,0.05000,2.766
	33,mlp.gate_proj,0.0001620953,0.05000,2.871
	33,mlp.down_proj,0.0000102629,0.05000,9.698
	34,linear_attn.in_proj_qkv,0.0002683308,0.05000,2.014
	34,linear_attn.in_proj_z,0.0001216698,0.05000,1.799
	34,linear_attn.out_proj,0.0000077787,0.05000,2.232
	34,mlp.up_proj,0.0002016926,0.05000,3.033
	34,mlp.gate_proj,0.0001748909,0.05000,3.043
	34,mlp.down_proj,0.0000134868,0.05000,9.760
	35,self_attn.k_proj,0.0000294353,0.05000,3.739
	35,self_attn.v_proj,0.0000423436,0.05000,3.832
	35,self_attn.q_proj,0.0002818952,0.05000,3.964
	35,self_attn.o_proj,0.0000129746,0.05000,2.268
	35,mlp.up_proj,0.0002112278,0.05000,2.812
	35,mlp.gate_proj,0.0001868287,0.05000,2.878
	35,mlp.down_proj,0.0000149566,0.05000,9.501
	36,linear_attn.in_proj_qkv,0.0003793859,0.05000,1.995
	36,linear_attn.in_proj_z,0.0001906052,0.05000,2.056
	36,linear_attn.out_proj,0.0000076288,0.05000,2.398
	36,mlp.gate_proj,0.0002082069,0.05000,2.823
	36,mlp.up_proj,0.0002087939,0.05000,2.846
	36,mlp.down_proj,0.0000134904,0.05000,9.323
	37,linear_attn.in_proj_qkv,0.0004087727,0.05000,2.040
	37,linear_attn.in_proj_z,0.0002336417,0.05000,1.937
	37,linear_attn.out_proj,0.0000079233,0.05000,2.371
	37,mlp.up_proj,0.0002071794,0.05000,2.924
	37,mlp.gate_proj,0.0002113439,0.05000,2.940
	37,mlp.down_proj,0.0000126032,0.05000,9.261
	38,linear_attn.in_proj_qkv,0.0003641806,0.05000,1.933
	38,linear_attn.in_proj_z,0.0002126180,0.05000,1.927
	38,linear_attn.out_proj,0.0000062375,0.05000,2.351
	38,mlp.gate_proj,0.0002730919,0.05000,2.693
	38,mlp.up_proj,0.0002319409,0.05000,3.058
	38,mlp.down_proj,0.0000126153,0.05000,9.272
	39,self_attn.q_proj,0.0003205566,0.05000,3.990
	39,self_attn.v_proj,0.0000426120,0.05000,3.989
	39,self_attn.k_proj,0.0000334579,0.05000,3.999
	39,self_attn.o_proj,0.0000080916,0.05000,2.274
	39,mlp.up_proj,0.0002231741,0.05000,2.852
	39,mlp.gate_proj,0.0002467629,0.05000,2.966
	39,mlp.down_proj,0.0000115561,0.05000,9.532
	40,linear_attn.in_proj_qkv,0.0003428894,0.05000,1.878
	40,linear_attn.in_proj_z,0.0001902024,0.05000,1.974
	40,linear_attn.out_proj,0.0000068595,0.05000,2.359
	40,mlp.up_proj,0.0002364197,0.05000,2.828
	40,mlp.gate_proj,0.0002664747,0.05000,2.845
	40,mlp.down_proj,0.0000120810,0.05000,9.456
	41,linear_attn.in_proj_qkv,0.0003137499,0.05000,2.138
	41,linear_attn.in_proj_z,0.0001716255,0.05000,1.880
	41,linear_attn.out_proj,0.0000059280,0.05000,2.473
	41,mlp.up_proj,0.0002241652,0.05000,2.721
	41,mlp.gate_proj,0.0002274599,0.05000,2.739
	41,mlp.down_proj,0.0000115259,0.05000,9.483
	42,linear_attn.in_proj_qkv,0.0002912149,0.05000,1.879
	42,linear_attn.in_proj_z,0.0001538357,0.05000,2.045
	42,linear_attn.out_proj,0.0000066113,0.05000,2.447
	42,mlp.gate_proj,0.0002051962,0.05000,2.746
	42,mlp.up_proj,0.0002163007,0.05000,2.847
	42,mlp.down_proj,0.0000123591,0.05000,9.604
	43,self_attn.k_proj,0.0000328653,0.05000,3.945
	43,self_attn.v_proj,0.0000419066,0.05000,3.974
	43,self_attn.q_proj,0.0003056384,0.05000,4.065
	43,self_attn.o_proj,0.0000111928,0.05000,2.405
	43,mlp.up_proj,0.0002239182,0.05000,2.926
	43,mlp.gate_proj,0.0002035338,0.05000,2.931
	43,mlp.down_proj,0.0000135572,0.05000,9.248
	44,linear_attn.in_proj_qkv,0.0002801879,0.05000,2.008
	44,linear_attn.in_proj_z,0.0001365148,0.05000,1.790
	44,linear_attn.out_proj,0.0000092356,0.05000,2.407
	44,mlp.up_proj,0.0002258869,0.05000,2.809
	44,mlp.gate_proj,0.0002031547,0.05000,2.878
	44,mlp.down_proj,0.0000151829,0.05000,9.601
	45,linear_attn.in_proj_qkv,0.0002850569,0.05000,1.954
	45,linear_attn.in_proj_z,0.0001386788,0.05000,1.806
	45,linear_attn.out_proj,0.0000080556,0.05000,2.429
	45,mlp.gate_proj,0.0002044231,0.05000,2.805
	45,mlp.up_proj,0.0002285339,0.05000,3.046
	45,mlp.down_proj,0.0000157144,0.05000,9.286
	46,linear_attn.in_proj_qkv,0.0002990598,0.05000,1.973
	46,linear_attn.in_proj_z,0.0001469002,0.05000,1.927
	46,linear_attn.out_proj,0.0000117627,0.05000,2.395
	46,mlp.gate_proj,0.0002012278,0.05000,2.789
	46,mlp.up_proj,0.0002300471,0.05000,2.827
	46,mlp.down_proj,0.0000171396,0.05000,9.654
	47,self_attn.k_proj,0.0000347433,0.05000,3.852
	47,self_attn.v_proj,0.0000558419,0.05000,3.955
	47,self_attn.q_proj,0.0003292233,0.05000,4.001
	47,self_attn.o_proj,0.0000132204,0.05000,2.237
	47,mlp.up_proj,0.0002564880,0.05000,2.770
	47,mlp.gate_proj,0.0002245832,0.05000,2.791
	47,mlp.down_proj,0.0000213101,0.05000,9.215
	48,linear_attn.in_proj_qkv,0.0003306475,0.05000,2.080
	48,linear_attn.in_proj_z,0.0001571578,0.05000,1.962
	48,linear_attn.out_proj,0.0000139544,0.05000,2.476
	48,mlp.up_proj,0.0002781727,0.05000,2.887
	48,mlp.gate_proj,0.0002438090,0.05000,2.900
	48,mlp.down_proj,0.0000261321,0.05000,9.646
	49,linear_attn.in_proj_qkv,0.0003951708,0.05000,2.102
	49,linear_attn.in_proj_z,0.0001682103,0.05000,2.024
	49,linear_attn.out_proj,0.0000192123,0.05000,2.427
	49,mlp.gate_proj,0.0002650328,0.05000,2.760
	49,mlp.up_proj,0.0003012208,0.05000,2.796
	49,mlp.down_proj,0.0000339468,0.05000,9.749
	50,linear_attn.in_proj_qkv,0.0004090191,0.05000,1.981
	50,linear_attn.in_proj_z,0.0001887876,0.05000,1.814
	50,linear_attn.out_proj,0.0000276691,0.05000,2.365
	50,mlp.up_proj,0.0003447875,0.05000,2.891
	50,mlp.gate_proj,0.0003162317,0.05000,2.927
	50,mlp.down_proj,0.0000570232,0.05000,9.744
	51,self_attn.q_proj,0.0004390058,0.05000,3.946
	51,self_attn.v_proj,0.0001044389,0.05000,3.971
	51,self_attn.k_proj,0.0000458526,0.05000,3.982
	51,self_attn.o_proj,0.0000393520,0.05000,2.430
	51,mlp.up_proj,0.0003995472,0.05000,2.695
	51,mlp.gate_proj,0.0003556540,0.05000,2.800
	51,mlp.down_proj,0.0000732886,0.05000,9.655
	52,linear_attn.in_proj_qkv,0.0006285047,0.05000,2.038
	52,linear_attn.in_proj_z,0.0002888460,0.05000,1.926
	52,linear_attn.out_proj,0.0000387001,0.05000,2.393
	52,mlp.up_proj,0.0004349370,0.05000,2.843
	52,mlp.gate_proj,0.0004419015,0.05000,2.924
	52,mlp.down_proj,0.0000731234,0.05000,9.152
	53,linear_attn.in_proj_qkv,0.0006293647,0.05000,2.033
	53,linear_attn.in_proj_z,0.0003074905,0.05000,1.847
	53,linear_attn.out_proj,0.0000427141,0.05000,2.380
	53,mlp.gate_proj,0.0004943917,0.05000,2.842
	53,mlp.up_proj,0.0004622656,0.05000,2.919
	53,mlp.down_proj,0.0000799335,0.05000,9.762
	54,linear_attn.in_proj_qkv,0.0006097221,0.05000,2.126
	54,linear_attn.in_proj_z,0.0003273230,0.05000,1.937
	54,linear_attn.out_proj,0.0000440077,0.05000,2.526
	54,mlp.up_proj,0.0005474443,0.05000,2.852
	54,mlp.gate_proj,0.0006205177,0.05000,2.870
	54,mlp.down_proj,0.0001119038,0.05000,9.327
	55,self_attn.q_proj,0.0005899858,0.05000,3.978
	55,self_attn.k_proj,0.0000637752,0.05000,4.002
	55,self_attn.v_proj,0.0001395592,0.05000,4.035
	55,self_attn.o_proj,0.0000379650,0.05000,2.496
	55,mlp.up_proj,0.0005508717,0.05000,2.840
	55,mlp.gate_proj,0.0005990099,0.05000,2.841
	55,mlp.down_proj,0.0000952430,0.05000,9.324
	56,linear_attn.in_proj_qkv,0.0007110504,0.05000,2.040
	56,linear_attn.in_proj_z,0.0003553513,0.05000,1.851
	56,linear_attn.out_proj,0.0000582206,0.05000,2.566
	56,mlp.gate_proj,0.0006629710,0.05000,2.838
	56,mlp.up_proj,0.0005916605,0.05000,2.924
	56,mlp.down_proj,0.0000945713,0.05000,9.728
	57,linear_attn.in_proj_qkv,0.0007448315,0.05000,1.959
	57,linear_attn.in_proj_z,0.0003739022,0.05000,1.902
	57,linear_attn.out_proj,0.0000515008,0.05000,2.429
	57,mlp.gate_proj,0.0007721504,0.05000,2.871
	57,mlp.up_proj,0.0006770504,0.05000,2.884
	57,mlp.down_proj,0.0000980713,0.05000,9.658
	58,linear_attn.in_proj_qkv,0.0007159035,0.05000,2.370
	58,linear_attn.in_proj_z,0.0003891521,0.05000,2.025
	58,linear_attn.out_proj,0.0000555304,0.05000,2.422
	58,mlp.gate_proj,0.0008880539,0.05000,2.742
	58,mlp.up_proj,0.0007787612,0.05000,2.753
	58,mlp.down_proj,0.0001309379,0.05000,9.574
	59,self_attn.k_proj,0.0000801485,0.05000,3.907
	59,self_attn.v_proj,0.0002913585,0.05000,3.955
	59,self_attn.q_proj,0.0007040162,0.05000,4.042
	59,self_attn.o_proj,0.0001141125,0.05000,2.312
	59,mlp.gate_proj,0.0008777454,0.05000,2.820
	59,mlp.up_proj,0.0007892898,0.05000,2.884
	59,mlp.down_proj,0.0001530582,0.05000,9.259
	60,linear_attn.in_proj_qkv,0.0008737440,0.05000,2.028
	60,linear_attn.in_proj_z,0.0004170820,0.05000,1.903
	60,linear_attn.out_proj,0.0001280804,0.05000,2.311
	60,mlp.up_proj,0.0008007702,0.05000,2.858
	60,mlp.gate_proj,0.0008728313,0.05000,3.094
	60,mlp.down_proj,0.0001893005,0.05000,9.602
	61,linear_attn.in_proj_qkv,0.0006409717,0.05000,1.929
	61,linear_attn.in_proj_z,0.0003683010,0.05000,1.809
	61,linear_attn.out_proj,0.0001176246,0.05000,2.279
	61,mlp.up_proj,0.0008527413,0.05000,2.939
	61,mlp.gate_proj,0.0009217091,0.05000,2.946
	61,mlp.down_proj,0.0002487219,0.05000,9.279
	62,linear_attn.in_proj_qkv,0.0007001918,0.05000,2.108
	62,linear_attn.in_proj_z,0.0003716715,0.05000,1.928
	62,linear_attn.out_proj,0.0002993706,0.05000,2.350
	62,mlp.up_proj,0.0007859706,0.05000,2.771
	62,mlp.gate_proj,0.0008442585,0.05000,2.820
	62,mlp.down_proj,0.0003750678,0.05000,9.371
	63,self_attn.k_proj,0.0000722345,0.05000,3.880
	63,self_attn.v_proj,0.0002261060,0.05000,3.961
	63,self_attn.q_proj,0.0006227864,0.05000,4.029
	63,self_attn.o_proj,0.0002908754,0.05000,2.263
	63,mlp.gate_proj,0.0007084487,0.05000,2.730
	63,mlp.up_proj,0.0006370316,0.05000,2.834
	63,mlp.down_proj,0.0007829439,0.05000,9.542