GPTParse is a powerful and versatile document parser designed specifically for Retrieval-Augmented Generation (RAG) systems. It enables seamless conversion of PDF documents into Markdown format using advanced vision language models (VLMs), facilitating easy integration into text-based workflows and applications.
With GPTParse, you can:
- Convert complex PDFs, including those with tables, lists, and images, into well-structured Markdown.
- Choose from multiple AI providers like OpenAI, Anthropic, and Google, leveraging their state-of-the-art models.
- Use GPTParse as a Python library or via a command-line interface (CLI), offering flexibility in how you integrate it into your projects.
It's as simple as:
gptparse vision example.pdf --output_file output.md
- Convert PDFs to Markdown: Transform PDF documents into Markdown format, preserving the structure and content, including tables, lists, and images.
- Multiple Parsing Methods: Choose between using Vision Language Models (VLMs) for high-fidelity conversion or traditional OCR methods (coming soon).
- Support for Multiple AI Providers: Seamlessly integrate with OpenAI, Anthropic, and Google AI models, selecting the one that best fits your needs.
- Python Library and CLI Application: Use GPTParse within your Python applications or interact with it through the command line.
- Customizable Processing Options: Configure concurrency levels, select specific pages to process, and customize system prompts to tailor the output.
- Page Selection: Process entire documents or specify individual pages or ranges of pages.
- Detailed Statistics: Optionally display detailed processing statistics, including token usage and processing times.
- Installation
- Quick Start
- Usage
- Available Models and Providers
- Examples
- Contributing
- License
- Acknowledgments
Install GPTParse using pip:
pip install gptparse
Ensure you have the following installed:
- Python 3.9 or higher
- Poppler: For PDF to image conversion
Poppler is the underlying project that handles PDF processing. You can check if you already have it installed by running pdftoppm -h
in your terminal/command prompt.
-
Ubuntu/Debian:
sudo apt-get install poppler-utils
-
Arch Linux:
sudo pacman -S poppler
-
macOS (with Homebrew):
brew install poppler
-
Windows:
- Download the latest poppler package from oschwartz10612's version, which is the most up-to-date.
- Extract the downloaded package and move the extracted directory to your desired location.
- Add the
bin/
directory from the extracted folder to your system PATH. - Verify the installation by opening a new command prompt and running
pdftoppm -h
.
After installing Poppler, you should be ready to use GPTParse.
Here's how you can quickly get started with GPTParse:
# Set your API key
export OPENAI_API_KEY="your-openai-api-key"
# Convert a PDF to Markdown
gptparse vision example.pdf --output_file output.md
Before using GPTParse, set up the API keys for the AI providers you plan to use by setting the appropriate environment variables:
-
OpenAI:
export OPENAI_API_KEY="your-openai-api-key"
-
Anthropic:
export ANTHROPIC_API_KEY="your-anthropic-api-key"
-
Google:
export GOOGLE_API_KEY="your-google-api-key"
You can set these variables in your shell profile (~/.bashrc
, ~/.zshrc
, etc.) or include them in your Python script before importing GPTParse.
Note: Keep your API keys secure and do not expose them in code repositories.
GPTParse allows you to set default configurations for ease of use. Use the configure
command to set default values for the AI provider, model, and concurrency:
gptparse configure
You will be prompted to enter the desired provider, model, and concurrency level. The configuration is saved in ~/.gptparse_config.json
.
Example:
$ gptparse configure
GPTParse Configuration
Enter new values or press Enter to keep the current values.
Current values are shown in [brackets].
AI Provider [openai]: anthropic
Default Model for anthropic [claude-3-5-sonnet-latest]: claude-3-opus-latest
Default Concurrency [10]: 5
Configuration updated successfully.
Current configuration:
provider: anthropic
model: claude-3-opus-latest
concurrency: 5
Below is an example of how to use GPTParse in your Python code:
import os
# Set the appropriate API key
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
from gptparse.modes.vision import vision
result = vision(
concurrency=10,
file_path="example.pdf",
model="gpt-4o",
output_file="output.md",
custom_system_prompt=None,
select_pages=None,
provider="openai",
)
# Access the result
print(f"Processed {len(result.pages)} pages in {result.completion_time:.2f} seconds.")
for page in result.pages:
print(f"Page {page.page}:")
print(page.content)
When using the command-line interface, ensure you've set the appropriate environment variables.
export OPENAI_API_KEY="your-openai-api-key"
To convert a PDF file to Markdown:
gptparse vision example.pdf --output_file output.md --provider openai
This command will process example.pdf
using the OpenAI provider and save the output to output.md
.
--concurrency
: Number of concurrent processes (default: value set in configuration or 10).--model
: Vision language model to use (overrides configured default).--output_file
: Output file name (must have a.md
or.txt
extension). If not specified, output will be printed to the console.--custom_system_prompt
: Custom system prompt for the language model.--select_pages
: Pages to process (e.g.,"1,3-5,10"
).--provider
: AI provider to use (openai
,anthropic
,google
) (overrides configured default).--stats
: Display detailed statistics after processing.
Example with additional options:
gptparse vision example.pdf --select_pages "1,3-5" --stats --output_file result.md
GPTParse supports multiple models from different AI providers.
gpt-4o
(Default)gpt-4o-mini
claude-3-5-sonnet-latest
(Default)claude-3-opus-latest
claude-3-sonnet-20240229
claude-3-haiku-20240307
gemini-1.5-pro-002
(Default)gemini-1.5-flash-002
gemini-1.5-flash-8b
To list available models for a provider in your code, you can use:
from gptparse.models.model_interface import list_available_models
# List models for a specific provider
models = list_available_models(provider='openai')
print("OpenAI models:", models)
# List all available models from all providers
all_models = list_available_models()
print("All available models:", all_models)
To process only specific pages from a PDF document, use the --select_pages
option:
gptparse vision example.pdf --select_pages "2,4,6-8"
This command will process pages 2, 4, 6, 7, and 8 of example.pdf
.
Provide a custom system prompt to influence the model's output:
gptparse vision example.pdf --custom_system_prompt "Please extract all text in bullet points."
To display detailed processing statistics, use the --stats
flag:
gptparse vision example.pdf --stats
Sample output:
Detailed Statistics:
File Path: example.pdf
Completion Time: 12.34 seconds
Total Pages Processed: 5
Total Input Tokens: 2500
Total Output Tokens: 3000
Total Tokens: 5500
Average Tokens per Page: 1100.00
Page-wise Statistics:
Page 1: 600 tokens
Page 2: 500 tokens
Page 3: 700 tokens
Page 4: 800 tokens
Page 5: 400 tokens
Contributions are welcome! If you'd like to contribute to GPTParse, please follow these steps:
- Fork the repository on GitHub.
- Create a new branch for your feature or bugfix.
- Make your changes and ensure tests pass.
- Submit a pull request with a clear description of your changes.
Please ensure that your code adheres to the existing style conventions and passes all tests.
GPTParse is licensed under the Apache-2.0 License. See LICENSE for more information.