I Trained My Own AI... It beat ChatGPT

Summary by Krater

The creator trains their own AI model using a dataset of code, running benchmarks against GPT-4o, DeepSeek, and Qwen models, and concludes that despite various iterations and formatting tweaks, it achieves a lower pass rate than major commercial models.

From the video

Answers: How does a custom-trained AI model perform on coding benchmarks compared to GPT-4o?

• AI model training
• benchmarking AI models
• code generation models
• synthetic data generation
• GPU benchmarking

What it concludes

• The creator's trained model achieved a pass rate of 16% on coding benchmarks, falling short of GPT-4o's 18.2%.
• Adding reasoning steps to the training data improved the model's benchmark performance.
• Fixing cable and hardware issues was necessary to successfully run the AI training without crashing GPUs.