MortgageTax

Proprietary

Vals AI original benchmark crafted with industry experts on non-public datasets

Updated: 4/16/2026

Evaluating reading and understanding tax certificates as images

Systems (69)

Accuracy

Key Takeaways

• Claude Opus 4.7 is the best performing model overall with 70.27% accuracy, excelling in both semantic and numerical extraction tasks, followed closely by Gemini 3.1 Pro Preview (02/26) at 69.40%.
• The benchmark tests the multimodal capabilities of models, as the documents are provided as images, including both computer-written and handwritten parts.
• The best models perform similarly on the Semantic Extraction task, while the gaps are more pronounced in the Numerical Extraction task.

Dataset and Context

The MortgageTax benchmark evaluates the ability of language models to extract information from mortgage tax certificates. Vontive provided both the mortgage documents and the labeled queries, which we used as a foundation for the dataset.

When prompting the models, we provide the documents as images, testing their multimodal capabilities. We test them on two tasks:

• Semantic Extraction: We ask the models to extract the year, the parcel number, and the county from the tax certificate.
• Numerical Extraction: We ask the models to calculate the annualized amount due based on the tax certificate.

For both tasks, the models are asked to reply in a JSON format containing both an explanation and the final answer for each question. Not all models follow this format precisely, so some regex pattern matching is used to extract the final answers.

The dataset consists of 1258 documents, divided into three sets:

• Public Validation (20 samples): Publicly accessible, available on request.
• Private Validation (300 samples): Available for purchase to evaluate your own models/agents on.
• Test (938 samples): A hold-out set that we never share externally. Only this set is used for the numbers reported on this page.

To receive access to the Public or Private Validation sets, please reach out to us at contact@vals.ai.

MortgageTax Input Example

Below is a typical example of the images passed to the models. For confidentiality reasons, we redacted the names and addresses of the owners and properties.

Here are examples of the questions asked of the models.

Numerical Extraction:

{
    "annualized_amount_due": {
        "reasoning": "Reasoning for annualized_amount_due",
        "answer": "Answer to annualized_amount_due"
    }
}

Semantic Extraction:

{
    "tax_year": {
        "reasoning": "Reasoning for tax_year",
        "answer": "Answer to tax_year question"
    },
    "county": {
        "reasoning": "Reasoning for county",
        "answer": "Answer to county question"
    },
    "parcel_number": {
        "reasoning": "Reasoning for parcel_number",
        "answer": "Answer to parcel_number question"
    }
}