hari-krishna-ai commited on
Commit
6b96926
Β·
verified Β·
1 Parent(s): 98a0ec4

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +365 -0
README.md ADDED
@@ -0,0 +1,365 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen3-8B
3
+ library_name: peft
4
+ license: apache-2.0
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - text-to-sql
8
+ - sql
9
+ - postgresql
10
+ - qlora
11
+ - lora
12
+ - peft
13
+ - qwen3
14
+ language:
15
+ - en
16
+ ---
17
+
18
+ # Qwen3-8B Text-to-SQL (QLoRA adapter)
19
+
20
+ A LoRA adapter that turns **Qwen3-8B** into a PostgreSQL text-to-SQL model for a
21
+ 12-table enterprise schema (sales, catalogue, logistics, HR).
22
+
23
+ **Strict execution accuracy went from 10.82 % to 50.99 %** on 453 held-out
24
+ questions β€” same prompt, same schema, same database, same executor. Only the
25
+ weights changed. With a one-shot self-correction loop on top, **52.10 %**.
26
+
27
+ The evaluation is execution-based: every prediction is run against a real
28
+ PostgreSQL database and its result set compared to the gold query's. No string
29
+ similarity anywhere.
30
+
31
+ > **Read the [Where the gain came from](#where-the-gain-actually-came-from)
32
+ > section before quoting the headline number.** Most of the improvement is the
33
+ > model learning this database's column conventions, not becoming dramatically
34
+ > better at SQL logic. That distinction is measurable, and it is reported.
35
+
36
+ ---
37
+
38
+ ## Quick start
39
+
40
+ ```python
41
+ import torch
42
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
43
+ from peft import PeftModel
44
+
45
+ ADAPTER = "hari-krishna-ai/qwen3-8b-text2sql-qlora"
46
+
47
+ bnb = BitsAndBytesConfig(
48
+ load_in_4bit=True,
49
+ bnb_4bit_quant_type="nf4",
50
+ bnb_4bit_use_double_quant=True,
51
+ bnb_4bit_compute_dtype=torch.float16,
52
+ )
53
+
54
+ tok = AutoTokenizer.from_pretrained(ADAPTER)
55
+ base = AutoModelForCausalLM.from_pretrained(
56
+ "Qwen/Qwen3-8B", revision="b968826d9c46", # pin: the exact weights trained on
57
+ quantization_config=bnb, device_map={"": 0}, torch_dtype=torch.float16,
58
+ )
59
+ model = PeftModel.from_pretrained(base, ADAPTER).eval()
60
+
61
+ SYSTEM = """You are an expert PostgreSQL analyst. You convert business \
62
+ questions into correct, executable PostgreSQL queries.
63
+
64
+ Rules:
65
+ - Output ONLY the SQL query. No explanation, no commentary.
66
+ - Use only the tables and columns given in the schema.
67
+ - Write a single SELECT statement. Never modify data.
68
+ - Use PostgreSQL syntax.
69
+ - When a foreign key is nullable, consider whether LEFT JOIN is needed to \
70
+ avoid silently dropping rows."""
71
+
72
+ USER = """Database schema:
73
+
74
+ {schema}
75
+
76
+ Question: {question}
77
+
78
+ PostgreSQL query:"""
79
+
80
+
81
+ def to_sql(question: str, schema: str) -> str:
82
+ messages = [
83
+ {"role": "system", "content": SYSTEM},
84
+ {"role": "user",
85
+ "content": USER.format(schema=schema, question=question) + " /no_think"},
86
+ ]
87
+ text = tok.apply_chat_template(
88
+ messages, tokenize=False,
89
+ add_generation_prompt=True,
90
+ enable_thinking=False, # <-- REQUIRED, see below
91
+ )
92
+ enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
93
+ out = model.generate(**enc, max_new_tokens=512, do_sample=False,
94
+ pad_token_id=tok.pad_token_id)
95
+ return tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
96
+ ```
97
+
98
+ ### ⚠️ `enable_thinking=False` is not optional
99
+
100
+ Qwen3's chat template renders a *finished* assistant turn as:
101
+
102
+ ```
103
+ <|im_start|>assistant\n<think>\n\n</think>\n\nSELECT ...
104
+ ```
105
+
106
+ Every training target was therefore preceded by an **empty think block**. With
107
+ `add_generation_prompt=True` and no `enable_thinking` argument, the template
108
+ stops at `<|im_start|>assistant\n` and leaves the model to produce `<think>`
109
+ itself β€” a starting point it never saw in training.
110
+
111
+ Omitting it produces reasoning prose, stray `</think>` tags, junk lead tokens
112
+ and repetition loops. This cost one full GPU evaluation run to discover.
113
+
114
+ ---
115
+
116
+ ## Prompt format
117
+
118
+ The adapter expects **the exact prompt above**, with the schema rendered as
119
+ `CREATE TABLE` statements plus `--` foreign-key comments, and the user turn
120
+ ending in `/no_think`.
121
+
122
+ | | |
123
+ |---|---|
124
+ | prompt template fingerprint | `8288e41a496531a9` |
125
+ | schema rendering fingerprint | `d03619e711661bc5` |
126
+ | schema size | 5,455 characters, 12 tables |
127
+
128
+ A different prompt will not reproduce these numbers. The fingerprints exist so
129
+ that can be checked rather than assumed.
130
+
131
+ ---
132
+
133
+ ## Training
134
+
135
+ QLoRA on a single **Tesla T4** (free Kaggle tier), **one epoch, 4 h 53 m**.
136
+
137
+ | | |
138
+ |---|---|
139
+ | base model | `Qwen/Qwen3-8B` @ `b968826d9c46` |
140
+ | quantisation | 4-bit NF4 + double quantisation, fp16 compute |
141
+ | LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
142
+ | target modules | `q,k,v,o,gate,up,down` β€” all attention and MLP projections |
143
+ | trainable parameters | **43,646,976 of 4,761,498,624 (0.917 %)** |
144
+ | sequence length | 2048 (longest example 1,602 tokens; nothing truncated) |
145
+ | effective batch / optimiser steps | 16 / 134 |
146
+ | learning rate | 2e-4, cosine, 3 % warmup |
147
+ | optimiser | paged 8-bit AdamW |
148
+ | loss | **completion-only** β€” prompt masked to `-100` |
149
+ | supervised token share | **3.01 %** |
150
+ | train loss | 4.10 β†’ 0.006 |
151
+ | eval loss | 0.475 β†’ 0.340 β†’ **0.335** |
152
+ | seed | 20260808 |
153
+
154
+ ### Why completion-only loss matters here
155
+
156
+ Each example is ~1,490 prompt tokens and ~46 completion tokens, and the prompt
157
+ is **~97 % database schema, byte-identical across all 2,133 examples**.
158
+
159
+ Training on the full sequence would send ~97 % of the gradient into memorising a
160
+ schema the model is *handed* at inference time. The loss curve would look
161
+ excellent throughout β€” predicting a constant is easy β€” while the ability you
162
+ actually care about barely moved.
163
+
164
+ Masking the prompt is what makes the remaining 3 % of tokens the thing being
165
+ learned.
166
+
167
+ ---
168
+
169
+ ## Dataset
170
+
171
+ Template-generated from a 12-table PostgreSQL database (~328,000 synthetic rows,
172
+ fixed seed). All data is synthetic; no real personal information.
173
+
174
+ | split | records | purpose |
175
+ |---|---:|---|
176
+ | train | 2,133 | fine-tuning |
177
+ | validation | 459 | monitoring, hyperparameter choices |
178
+ | test | 453 | **held out, measured once per configuration** |
179
+
180
+ **Splits are leakage-free by template equivalence group, not by row.**
181
+ Verified: 74 training templates, 48 test templates, **zero overlap**, and no
182
+ shared question strings. So the reported accuracy is generalisation to query
183
+ *patterns the model never saw*, not to new slot values in familiar patterns.
184
+
185
+ Difficulty tiers: `easy` (SELECT/WHERE/ORDER BY) β†’ `medium` (GROUP BY, HAVING,
186
+ aggregates) β†’ `hard` (joins, subqueries, CTEs) β†’ `very_hard` (window functions,
187
+ nested aggregation) β†’ `enterprise` (ambiguous terminology, NULL handling,
188
+ business rules).
189
+
190
+ ---
191
+
192
+ ## Evaluation
193
+
194
+ Execution-based. Every prediction runs against PostgreSQL and its result set is
195
+ fingerprinted (MD5 over sorted, stringified rows). These two are different
196
+ strings and both count as correct:
197
+
198
+ ```sql
199
+ SELECT COUNT(*) FROM customers WHERE country = 'India';
200
+ SELECT COUNT(customer_id) FROM customers WHERE country = 'India';
201
+ ```
202
+
203
+ Order is compared only where the gold query has an `ORDER BY`.
204
+
205
+ ### Ablation β€” 453 held-out questions
206
+
207
+ | metric | base | + retrieval | **this adapter** | adapter + retrieval | **adapter + repair** |
208
+ |---|---:|---:|---:|---:|---:|
209
+ | **strict execution accuracy** | 10.82 % | 9.27 % | **50.99 %** | 41.72 % | **52.10 %** |
210
+ | projection-tolerant accuracy | 45.92 % | 43.93 % | 50.99 % | 41.72 % | **52.10 %** |
211
+ | executable SQL | 98.90 % | 92.27 % | 95.81 % | 94.92 % | **98.23 %** |
212
+ | schema hallucination | 0.66 % | 3.31 % | 1.99 % | 3.53 % | **0.66 %** |
213
+ | syntax errors | 0.00 % | 0.00 % | 0.22 % | 0.00 % | 0.22 % |
214
+
215
+ All five configurations measured on the same 453 held-out questions.
216
+
217
+ ### Self-correction recovers what fine-tuning broke
218
+
219
+ Feeding the database's own error back and retrying **once** returns executable
220
+ SQL and hallucination to the *base model's* level while keeping the accuracy
221
+ gain β€” 95.81 % β†’ 98.23 % executable, 1.99 % β†’ 0.66 % hallucination.
222
+
223
+ Repair fires only on **detectable** failures: SQL that fails to validate or
224
+ fails to execute. A query that runs and returns the wrong rows is
225
+ indistinguishable from a correct one without a gold answer, so it is left alone.
226
+
227
+ | | | |
228
+ |---|---:|---|
229
+ | repair **success** β€” now executes | 11 / 19 | 57.9 % |
230
+ | repair **correctness** β€” returns the gold rows | 5 / 19 | 26.3 % |
231
+ | returned identical SQL | 4 / 19 | |
232
+
233
+ **Six of the eleven "successes" turned a crash into a confident wrong answer.**
234
+ Reporting one rate would have claimed 57.9 % and hidden that.
235
+
236
+ Repair works when the error *names the fix* and fails when it names only the
237
+ *symptom*. Every correct repair was `column "method" does not exist` β€” the
238
+ schema shows `payment_method`, and the model renames it. The failures were
239
+ `datediff does not exist` and `column reference is ambiguous`: both state a
240
+ symptom without implying the correction.
241
+
242
+ ### By difficulty
243
+
244
+ | tier | n | base | **adapter** | Ξ” |
245
+ |---|---:|---:|---:|---:|
246
+ | easy | 126 | 0.0 % | 23.0 % | +23.0 |
247
+ | medium | 63 | 52.4 % | 52.4 % | **+0.0** |
248
+ | hard | 144 | 3.5 % | **84.7 %** | **+81.2** |
249
+ | very_hard | 36 | β€” | 8.3 % | β€” |
250
+ | enterprise | 84 | 13.1 % | 52.4 % | +39.3 |
251
+
252
+ ---
253
+
254
+ ## Where the gain actually came from
255
+
256
+ A 4.7Γ— improvement deserves scrutiny, not a victory lap.
257
+
258
+ | failure mode | base | adapter |
259
+ |---|---:|---:|
260
+ | right rows, **wrong columns** | 159 | **0** |
261
+ | genuinely wrong rows | 240 | 203 |
262
+
263
+ The benchmark's single largest failure was returning **correct data under a
264
+ different column projection** β€” *"show orders in 2023"* never says which columns
265
+ to return. That went to zero. Genuinely wrong answers fell far less.
266
+
267
+ So: **most of the +40 points is the model learning this project's column
268
+ conventions from the gold SQL.** That is a real and useful thing to learn, and
269
+ it is not the same as becoming much better at SQL logic. The `hard` tier
270
+ (+81.2 pp) is where reasoning genuinely improved.
271
+
272
+ A concrete example of what was learned β€” the base model wrote:
273
+
274
+ ```sql
275
+ WHERE o.status = 'Completed' -- returns 0 rows: no such value exists
276
+ ```
277
+
278
+ The adapter writes:
279
+
280
+ ```sql
281
+ WHERE status = 'completed' -- the actual value in this database
282
+ ```
283
+
284
+ The schema declares `status VARCHAR(20)` and never enumerates the allowed
285
+ values. The base model guessed a plausible enum; the adapter learned the real
286
+ one.
287
+
288
+ ---
289
+
290
+ ## Example outputs
291
+
292
+ All produced by this adapter, all verified correct by execution.
293
+
294
+ **easy** β€” *"Show payments with status completed."*
295
+ ```sql
296
+ SELECT payment_id, order_id, amount, status FROM payments WHERE status = 'completed'
297
+ ```
298
+
299
+ **medium** β€” *"How many orders are there in each status?"*
300
+ ```sql
301
+ SELECT status, COUNT(*) AS order_count FROM orders GROUP BY status ORDER BY order_count DESC
302
+ ```
303
+
304
+ **hard** β€” *"Who are the top 15 customers by revenue?"*
305
+ ```sql
306
+ SELECT c.customer_name,
307
+ SUM(oi.quantity * oi.unit_price * (1 - oi.discount_pct / 100)) AS revenue
308
+ FROM customers c
309
+ JOIN orders o ON o.customer_id = c.customer_id
310
+ JOIN order_items oi ON oi.order_id = o.order_id
311
+ GROUP BY c.customer_id, c.customer_name
312
+ ORDER BY revenue DESC LIMIT 15
313
+ ```
314
+
315
+ **enterprise** β€” *"What is our actual revenue, excluding cancelled and returned orders?"*
316
+ ```sql
317
+ SELECT SUM(o.total_amount) AS revenue FROM orders o
318
+ WHERE o.status NOT IN ('cancelled', 'returned')
319
+ ```
320
+
321
+ ---
322
+
323
+ ## Limitations
324
+
325
+ Stated plainly, because they bound what the numbers mean.
326
+
327
+ - **Schema-specific.** Trained on one 12-table schema. It has learned *this*
328
+ database's conventions β€” that is most of the measured gain β€” so accuracy on a
329
+ different schema will be far lower.
330
+ - **Half its answers are still wrong.** 50.99 % alone, 52.10 % with repair, is a
331
+ large improvement over 10.82 %; it is not production-grade unaided.
332
+ - **It broke things too.** Executable SQL fell 3.09 pp and schema hallucination
333
+ rose 1.33 pp: 19 queries fail where 5 did before. A final training loss of
334
+ 0.006 against an eval loss of 0.335 means it fit 2,133 examples hard, and a
335
+ tightly-fit model emits confident wrong identifiers more readily.
336
+ - **`medium` did not improve at all** (52.4 % β†’ 52.4 %). Unexplained.
337
+ - **Schema retrieval makes it worse**, and worse than it made the base model
338
+ (βˆ’9.27 pp vs βˆ’1.55 pp): the adapter only ever saw full-schema prompts, so
339
+ subsets are out of distribution for it. Give it the whole schema.
340
+ - **Test questions are template-generated**, from the same generator as
341
+ training. Real user phrasing β€” abbreviations, typos, ambiguity β€” is untested.
342
+ This is the biggest caveat on the headline number.
343
+ - **One epoch, one seed, one run.** No variance estimate.
344
+ - **Silent wrong answers are the real risk**, and repair cannot help with them:
345
+ a query that runs and returns the wrong rows raises no error to feed back.
346
+ Repair itself creates six more of them out of nineteen attempts. Never put
347
+ this in front of users without showing them the SQL.
348
+
349
+ ## Intended use
350
+
351
+ Assisted SQL authoring against this schema, with a human reading the generated
352
+ query. **Not** an autonomous agent over a production database.
353
+
354
+ If you deploy it, do what the source project does and do not rely on the model
355
+ behaving: a non-superuser role, read-only sessions with statement timeouts, and
356
+ static rejection of anything that is not a single read-only statement over
357
+ tables that exist.
358
+
359
+ ## Citation / source
360
+
361
+ Full pipeline, evaluation harness, ablation study and reproduction instructions:
362
+ the source repository accompanying this adapter.
363
+
364
+ Every number here comes from `experiments/` in that repository and is
365
+ reproducible from the frozen fingerprints listed above.