Spider: text to SQL on unseen databases
This example trains a model on 10,181 human-written questions across 200 databases. Each input contains a question and its database schema, and the model returns one SQLite query. Running that query determines its score. Use this example to compare supervised tuning with reinforcement learning. Supervised tuning learns the wording of the gold queries, while the execution reward scores the returned rows.
The example directory contains the data preparation script, scoring bundle, pipeline descriptor, and walkthrough.sh. The walkthrough runs each step on this page against the trainer at TRAINER_URL. To download the directory, see the example archive. The example archive doesn’t contain the Spider databases; see The data.
Before you begin
-
A trainer deployed in your Akka project, with its address in
TRAINER_URL.akka service get trainershows the route hostname. See Akka Optimize CLI. -
An authenticated CLI session. Run
akka auth loginto authenticateakka. -
Python 3. The data script uses the standard library only.
-
Network access to
datasets-server.huggingface.co, where the data script reads the rows. The API limits the request rate, so fetching a largeROWSvalue takes several minutes. The script caches rows, so subsequent runs use the cached data. -
The Spider databases, a 206 MB download; see The data.
-
A base model that your trainer allows. Run
akka training base-models listto list the models, andakka training base-models select <repository>to select one. See Base models.
The walkthrough reads ROWS, RL_ROWS, and EVAL_ROWS from the environment. Start with ROWS=200 RL_ROWS=100 EVAL_ROWS=50. Data preparation takes seconds, and the supervised score becomes available in about 15 minutes. Most of that time is spent waiting for a GPU. The defaults are 2,000, 600, and 300.
Why this dataset
Spider’s dev split uses databases that don’t appear in any training question. The held-out score therefore tests composition rather than recall. The model must query an unseen database, rather than answer a new question about a known schema. A model can memorize question templates for one schema, score well on similar data, and fail outside that distribution. Human-written questions across 200 databases don’t provide a fixed enumeration to learn.
Every prompt includes a schema read from the database file itself. The prompt and query therefore use the same schema.
The data
Questions come from xlangai/spider through the Hugging Face datasets-server API. Download the SQLite databases separately because that repository contains only the questions, and execution grading must run each query. The Spider dataset archive is about 206 MB and extracts 166 databases into spider_data/database. Download it from a mirror of the Yale archive:
curl -sL -o spider_data.zip \
https://huggingface.co/datasets/HAL-9001/spider-databases/resolve/main/spider_data.zip
unzip -q spider_data.zip
export SPIDER_DATABASES="$PWD/spider_data/database"To prepare the data, run:
python3 prepare_dataset.py --databases "$SPIDER_DATABASES" \
--rows "$ROWS" --rl-rows "$RL_ROWS" --eval-rows "$EVAL_ROWS"The script writes three splits. train.jsonl is for supervised tuning. rl.jsonl contains questions that the supervised model hasn’t seen and is used for the reinforcement run. eval.jsonl contains held-out questions about the unseen databases. The script skips a database when its schema exceeds --max-schema characters, which keeps prompts within the sequence budget. It also skips a row whose gold query doesn’t run.
Every prompt states its database’s schema as one line per table:
You answer questions about a database with one SQLite query.
Reply with the query and nothing else.
stadium(Stadium_ID, Location, Name, Capacity, Highest, Lowest, Average)
singer(Singer_ID, Name, Country, Song_Name, Song_release_year, Age, Is_male)
concert(concert_ID, concert_Name, Theme, Stadium_ID, Year)
singer_in_concert(concert_ID, Singer_ID)The schema accounts for most of the prompt length, so maxSeqLength in the descriptor must accommodate it. The dev split asks each question twice with different wording, so consecutive rows in eval.jsonl share a gold query.
The script also fills scoring/data/ with the databases used by the splits and an index that maps each row to its database. The default splits use 63 Spider databases, totaling 8 MB of SQLite data from the 206 MB archive. The bundle therefore includes only the required databases. Prepare the data and push the bundle in the same run. The reward rejects a row if its gold query doesn’t match the query stored in the bundle.
The scoring bundle
Scoring executes the query, so the trainer needs access to the databases. The bundle includes them, along with verify.py for the execution check, reward.py for the reward term, grader.py for evaluations, and data/ for the databases and index. A query is correct if it returns the same rows as the gold query, regardless of its wording.
To register the datasets and push the bundle, run:
akka-optimize datasets create -f data/train.jsonl --name spider-train
akka-optimize datasets create -f data/rl.jsonl --name spider-rl
akka-optimize datasets create -f data/eval.jsonl --name spider-eval
akka-optimize training bundles push scoring/ --name spider-scoringTo create the workload and set its evaluation settings, run:
akka-optimize training workloads get spider-sql >/dev/null 2>&1 ||
akka-optimize training workloads create spider-sql
akka-optimize training workloads set-evaluation spider-sql --dataset spider-eval --grader bundle:spider-scoring/graderThe trainer rejects create when the name exists, so the walkthrough runs it only when the workload is missing. The walkthrough runs set-evaluation every time. As a result, it applies this dataset and grader to a workload left partially configured by an earlier pass.
The reinforcement job imports reward from the bundle. An evaluation specifies bundle:spider-scoring/grader, which the trainer serves as a job for the duration of the evaluation. Scoring doesn’t run on the machine that submitted the run.
Select the base model for every stage. A stage can specify only a model that the deployment allows and that someone has selected:
akka-optimize training base-models select Qwen/Qwen2.5-0.5B-Instruct --wait--wait returns when the weights are cached. Without it, the first stage waits for the weights. Selecting an already selected model succeeds, so you can repeat the walkthrough.
The pipeline
One descriptor submits the complete process: the untuned base model’s held-out score, the supervised run and its held-out score, and the GRPO run and its held-out score. The first stage specifies baseModel because no earlier stage produced its model. This includes the measurement before training in the same pipeline as the later measurements. Every later stage refers to an earlier stage as @STAGE_NAME, and rl warm-starts from @sft:
{
"kind": "pipeline",
"description": "Tune Qwen2.5-0.5B-Instruct on Spider text-to-SQL, then improve it with GRPO against a reward that runs the query. Five pipeline stages: the base model's held-out score, the supervised tune, its held-out score, the GRPO run, and its held-out score. The base, sft and rl scores are the three points this pipeline is for comparing.",
"stages": [
{
"type": "evaluate",
"name": "base-scored",
"description": "Execution accuracy of the untuned base model on the held-out databases, before anything trains. This is the pipeline's before measurement: sft-scored and rl-scored are only worth reading next to what the model already did without them. It gates on nothing, because a low score here is the reason the rest of the pipeline exists.",
"baseModel": "Qwen/Qwen2.5-0.5B-Instruct",
"dataset": "spider-eval",
"grader": "bundle:spider-scoring/grader",
"maxCompletionTokens": 128
},
{
"type": "train",
"name": "sft",
"configName": "sft-r16-e1",
"description": "The real supervised tune: fit the base model to the training split's gold SQL, producing the candidate that sft-scored measures and that rl warm-starts from.",
"workload": "spider-sql",
"baseModel": "Qwen/Qwen2.5-0.5B-Instruct",
"config": {
"dataset": "spider-train",
"hyperparams": {
"lora": {
"rank": 16
},
"optimizer": {
"learningRate": 0.0001
},
"training": {
"epochs": 1
}
},
"maxSeqLength": 768,
"model": {"precision": "BF16"}
},
"execution": {
"gpus": 1
}
},
{
"type": "evaluate",
"name": "sft-scored",
"description": "Execution accuracy of the sft model on the held-out databases: the baseline rl must beat for the GRPO stage below to have been worth running.",
"model": "@sft",
"dataset": "spider-eval",
"grader": "bundle:spider-scoring/grader",
"maxCompletionTokens": 128
},
{
"type": "train",
"name": "rl",
"configName": "grpo-40-steps-off-sft",
"description": "The full GRPO run off @sft: reward is the grader running each generated query against its own database, so the model learns directly from execution rather than from imitating gold SQL.",
"workload": "spider-sql",
"baseModel": "Qwen/Qwen2.5-0.5B-Instruct",
"warmStartModelId": "@sft",
"config": {
"dataset": "spider-rl",
"method": "grpo",
"rlHyperparams": {
"epochs": 1,
"maxSteps": 40,
"groupSize": 8,
"promptsPerStep": 8,
"klBeta": 0.0,
"temperature": 1.0,
"maxCompletionLength": 64,
"stalenessThreshold": 0.0,
"lora": {
"rank": 16
},
"optimizer": {
"learningRate": 1e-05
}
},
"reward": {
"terms": [
{
"kind": "rule",
"weight": 1.0,
"bundle": "spider-scoring",
"module": "reward"
}
]
},
"maxSeqLength": 384,
"model": {"precision": "BF16"}
},
"execution": {
"gpus": 1,
"rolloutGpus": 0,
"microBatchSequences": 32,
"saveEvery": 4,
"keepAdapters": 10
}
},
{
"type": "evaluate",
"name": "rl-scored",
"description": "Execution accuracy of the rl model on the same held-out databases as sft-scored: the pipeline's answer to whether GRPO moved the needle over supervised tuning alone.",
"model": "@rl",
"dataset": "spider-eval",
"grader": "bundle:spider-scoring/grader",
"maxCompletionTokens": 128
}
]
}The base model must be one that your deployment allows and that someone has selected; see Base models. If the deployment doesn’t allow Qwen/Qwen2.5-0.5B-Instruct, replace the model in every stage of pipeline.json. The results will differ from the following values.
The reinforcement stage uses maxSteps of 40 and promptsPerStep of 8, for a total of 320 prompts. maxSteps takes precedence over epochs. With RL_ROWS=100, the stage makes about three passes over the reinforcement split. With the default 600, it processes about half of the split once. When you change either setting, set RL_ROWS to the number of prompts per step multiplied by the number of steps.
To start the pipeline, run:
PIPELINE=$(akka-optimize training pipelines start -f pipeline.json -o json --jq .pipelineId)
akka-optimize training pipelines wait "$PIPELINE" --exit-status
akka-optimize training pipelines get "$PIPELINE"pipelines get reports each stage and shows its description as its purpose. The pipeline compares the three stages base-scored, sft-scored, and rl-scored. Each train stage contains a configName, which labels its row in training workloads scores.
What the loop produced
The following results use supervised tuning on 2,000 questions, followed by GRPO warm-started from that adapter on 424 unseen questions. Execution grading scored both models on 300 held-out questions across six databases that don’t appear in any training question:
| Model | Execution accuracy | Gained | Lost | Paired |
|---|---|---|---|---|
|
0.325 |
|||
After supervised tuning |
0.393 |
baseline |
||
GRPO, 4 steps |
0.407 |
7 |
3 |
p 0.34, inside noise |
GRPO, 12 steps |
0.430 |
17 |
6 |
p 0.037 |
GRPO, 24 steps |
0.453 |
26 |
8 |
p 0.0036 |
GRPO improved execution accuracy by six percentage points over supervised tuning after 24 steps on 192 prompts. The result after four steps is within the noise, and the gain becomes significant as the number of steps increases. The changed rows show what the reward promotes. Supervised tuning learns Spider’s frequent use of aliases and begins to add joins that a question doesn’t require. The execution reward reverses that behavior because it scores the returned rows:
Q What are all distinct countries where singers above age 20 are from?
gold SELECT DISTINCT country FROM singer WHERE age > 20
sft SELECT DISTINCT T1.Country FROM singer AS T1 JOIN singer_in_concert AS T2 ON ...
rl SELECT DISTINCT country FROM singer WHERE age > 20The losses show the same change taken too far: the model removes a join that the question requires.
The run’s reward curve doesn’t show this improvement. Over the same steps, the reward line rises by less than its residual spread. The reward therefore appears to have no trend even though the evaluation improves by six percentage points. Judge the run by its held-out evaluation instead of its reward curve.
Cost
A Spider prompt includes a complete database schema, and gold queries average 110 characters. Both sequence length and generation cost are therefore higher than for a classification task. The untuned base model’s held-out score, base-scored, shows whether a model of this size can write a query for an unfamiliar schema. Use that result to choose the model size.