In the competition, you need to use LLMs to solve high school mathematics problems. Your task is to develop models that can accurately solve these problems and submit your predictions.
For more details, visit the competition page.
Getting started code: starter_notebook.ipynb
You will need to install langchain for this code:
pip install -qU langchain-openai langchainThanks Blaq for contibuting the notebook!
Note that the solution is not deterministic: when you run it again on the same record, sometimes the solution is different from the initial run.
Which might be more a feature than a bug: what if you run it multuple times and get the majority vote?
We use accuracy as the evaluation metric. Sometimes multiple answers are correct. In this case, a solution is correct if it matches at least one of the possible answers.
You can find the code for evaluation in scorer.py - it's taken from kaggle and this is exactly the code we use in the competition.