344

Prompt

Run a comparative MicroEval across the following low-cost / budget LLM APIs: - DeepSeek V4.1 Flash (Non-reasoning mode) - Gemini 3.5 Flash-Lite - Llama 3.3 70B (Groq deployment) - GPT-4o-mini / GPT-5.4-mini Evaluate these models specifically for a backend System Agent / Orchestration role with the following task metrics: 1. JSON & SCHEMA RIGIDITY: - Evaluates strict JSON mode output without markdown wrapping or conversational fluff. - Measures parsing error rate on complex nested JSON schemas (specifically translation bounding-box mappings, timing intervals, and state flags). 2. DECISION REASONING & ACCURACY: - Evaluates system state routing decisions: deterministically choosing when to trigger fallback OCR paths, handling rate-limit backoff rules, and selecting optimal target languages. - Micro-benchmark: 100 system orchestration edge cases containing ambiguous context or interrupted API signals. 3. LATENCY & PRICE PERFORMANCE: - Measure Time to First Token (TTFT) and throughput (tokens/sec). - Compute total cost per 1M execution tokens (Input vs Output) based on current live pricing. Output a ranked scorecard summarizing Accuracy %, Schema Adherence %, TTFT (ms), and Price per 1M tokens to help select the best budget replacement model for system orchestrations.