ProofHound

ProofHound is a self-hosted Prompt full lifecycle management platform, suitable for scenarios where you need to continuously tune prompt words, verify model output quality, track experimental results, and manage online versions. It puts model configuration, data sets, prompt version, experiments, automatic optimization, publishing, run results, manual annotations, Webhooks and MCP tool entries in the same workbench. Users can use real data sets to verify Prompt performance, locate failed samples based on indicators, generate better versions through experiments or automatic optimization, and finally publish the verified version to the online call link. ##Who is suitable for use - The AI application team that requires long-term maintenance of Prompt. - Product, operations, risk control, customer service, financial analysis and other teams that need to use dataset regression to verify the Prompt effect. - Teams that want to migrate Prompt tuning from scripts and forms to visual processes. - Users who need to self-host and manage model API keys and data themselves. - Users who need to connect Prompt management capabilities to external systems or Agents through Webhooks or MCP. ##Basic usage method ###Configuration Model Enter "Model" and click New Model. Need to fill in: - Name: For example,`gpt-4o-prod`,`deepseek-chat`. - Vendor type: Select OpenAI, Anthropic, DeepSeek, Qwen, etc. - Model ID: Fill in the actual model name of the supplier. - Endpoint: Fill in the model API address. - API Key: Fill in your own model call credentials. - Context window: Maximum context length of the model. - RPM, TPM, concurrency upper limit: Used to control call speed and avoid triggering vendor current limit. - Input/output Token unit price: Used to estimate experimental and production call costs. You can use Test Connectivity to confirm that the model is available before saving. After the model configuration is complete, the model can be selected for experimentation, optimization, and release. ###Importing data sets Enter "Data Set" and click New or Upload Data Set. Supported data formats include: - CSV - TSV - JSONL - JSON array - ZIP Field roles need to be confirmed when importing: - ID field: Unique identification of the sample. - Input fields: will be mapped to Prompt variables, such as `text`,`question`,`content`. - Picture field: Multimodal models are available. - Expected output: Used to determine whether the model output is correct. - Metadata fields: Used to analyze, filter, or assist in locating problems. It is recommended to prepare a small data set to complete the first verification before importing a larger formal regression set. ###Create a prompt word Enter "Prompt Words" and click New Prompt Words. Once created, an editable version of v1 is generated. You can continue to configure: - Prompt text. - Enter variables such as `{{text}}`,`{{question}}`. - Output fields, such as label, reason. - Decision rules used to compare the model output to the expected output in the data set. Once the prompt word version is experimented, optimized, or published, it will be automatically frozen. Subsequent modifications will form a new version, and the old version can still be traced back. ###Run the experiment Enter "Experiment" and click New Experiment. Choose: - Prompt words and versions. - Data sets. - model. - Operating parameters such as temperature, concurrency, RPM, TPM, timeout, and number of retries. The system checks whether variables and dataset fields are aligned before committing. After the experiment is run, you can view: - Total number of samples, processed samples, failed samples. - Quality indicators such as Accuracy, Precision, Recall, and F1. - Indicators and confusion for each category. - Input, output, expected result, determination result and failure reason for each sample. - Token consumption, cost estimates and delays. Experimental results are an important basis for subsequent optimization and release. ###Using automatic optimization Enter "Optimization" and click New Optimization. There are three types of starting points to start: - Start with existing experiments: It is suitable for continuous optimization based on an experiment with many failures. - Start with the prompt: Appropriate Prompt exists, but there is no clear baseline experiment. - Start with the data set: It is suitable to build optimization tasks around the data set first. ###Create and publish Enter "publish", you can create a production release or add grayscale candidates to existing release lines. Typical process: 1. Select the verified prompt word version. 2. Select a model. 3. Select an input connector, such as Webhook, Redis, or Kafka. 4. Configure field mapping to map external message fields to Prompt variables. 5. Configure traffic modes, such as production 100%, gray scale cut or dual run observation. 6. Set operating parameters and output connectors. After release, online calls will write the unified operation results to facilitate viewing of the actual production results. When a problem occurs, you can roll back or stop in the release details.