Crush: Command-Line Model Specification Feature Request

Hey guys!

We're super excited to dive into a feature request that's gonna make using Crush even more awesome, especially for those of you who love the command line. Let's break down why this is a big deal, how it'll help, and what we're proposing.

The Need for Model Specification

Understanding the Current Workflow

Right now, Crush is fantastic for interactive sessions, letting you tweak and play with different language models (LLMs) on the fly. But, when it comes to integrating Crush into automated workflows or systems like TerminalBench, things get a little trickier. TerminalBench, which is working on integrating Crush through this pull request, needs to run Crush in a non-interactive mode using the crush run command. This is where the current limitation pops up: we can't directly specify which LLM to use when running Crush in this non-interactive mode. Imagine you're trying to benchmark different models or have a specific model you need for a particular task – without this ability, it's like trying to drive a car without a steering wheel!

Why Non-Interactive Mode Matters

Non-interactive mode is crucial for several reasons. First off, it's essential for automation. Think about setting up scripts or systems that automatically run tests or processes using Crush. If you need to manually select a model each time, that defeats the purpose of automation. Secondly, it's vital for integration with other tools and platforms, like TerminalBench. These integrations often require a seamless, hands-off approach to model selection. Lastly, specifying the model via the command line makes your workflows much more reproducible. You can ensure that the same model is used every time, which is super important for consistent results and benchmarking.

The Core Issue: Lack of Command-Line Argument

The heart of the matter is that we need a way to tell Crush which LLM to use directly from the command line when using crush run. Currently, there's no command-line argument or flag to specify the model. This means that if you have a preferred model or a specific model requirement for a task, you're out of luck in non-interactive mode. This limitation not only adds friction to automated workflows but also hinders the full potential of Crush in various integration scenarios.

The Proposed Solution: A Command-Line Argument for Model Specification

Introducing the --model Flag

Our proposal is straightforward: let's add a command-line argument to the crush run command that allows users to specify the LLM they want to use. We're thinking something like a --model flag would do the trick. For example, you could run:

crush run --model gpt-4 "Your prompt here"

This command would tell Crush to use the GPT-4 model for the given prompt. Easy peasy, right?

Benefits of This Approach

Adding a --model flag brings a whole host of benefits. For starters, it makes non-interactive mode way more useful. You can seamlessly integrate Crush into your automated workflows without having to jump through hoops. It also gives you greater flexibility and control over which models you use, which is a huge win for those of us who like to fine-tune our setups. Plus, it makes your workflows more reproducible, ensuring consistent results every time. This is especially critical for benchmarking and comparing different models.

How This Fits into Existing Workflows

This enhancement fits perfectly into existing workflows, especially those involving automation and integration. Imagine you're setting up a script to run a series of tests using different LLMs. With the --model flag, you can easily specify which model to use for each test, all from the command line. This makes your scripts cleaner, more efficient, and less prone to errors. Similarly, for integrations with tools like TerminalBench, this feature ensures a smooth and seamless experience, allowing you to leverage Crush's capabilities without any manual intervention.

Real-World Use Cases

Automating Benchmarking with TerminalBench

Let's dive deeper into how this feature would rock in a real-world scenario, specifically with TerminalBench. Imagine you're trying to benchmark various LLMs to see which one performs best for a particular task. Currently, without the ability to specify the model via the command line, you'd have to manually configure Crush for each model, which is a total drag. But with the --model flag, you could write a simple script that iterates through different models, running the same benchmark and collecting the results. This not only saves you a ton of time but also ensures a consistent and reproducible benchmarking process.

Streamlining CI/CD Pipelines

Another compelling use case is in CI/CD (Continuous Integration/Continuous Deployment) pipelines. Many software development teams use LLMs to automate tasks like code review, documentation generation, and bug detection. By adding the --model flag, you can easily incorporate Crush into your CI/CD pipeline, specifying the appropriate model for each task. For example, you might use a smaller, faster model for quick code reviews and a more powerful model for generating detailed documentation. This level of control and flexibility can significantly enhance the efficiency and effectiveness of your development process.

Custom Scripting and Automation

Beyond benchmarking and CI/CD, the --model flag opens up a world of possibilities for custom scripting and automation. Think about creating scripts that automatically generate reports, translate text, or even write code snippets using specific LLMs. With the ability to specify the model via the command line, you can tailor your scripts to meet your exact needs, ensuring that you're always using the right tool for the job. This level of customization empowers you to leverage the full potential of Crush in a wide range of applications.

Why This Matters to the Community

Enhancing User Experience

Ultimately, this feature request is all about enhancing the user experience. We want to make Crush as versatile and user-friendly as possible, whether you're using it interactively or in a non-interactive, automated setting. Adding the --model flag is a significant step in that direction, providing you with the control and flexibility you need to get the most out of Crush.

Fostering Integration and Collaboration

By making it easier to integrate Crush into other tools and workflows, we're also fostering collaboration and innovation within the community. When tools work well together, it opens up new possibilities and encourages users to explore creative solutions. This feature will make Crush an even more valuable asset in your toolkit, enabling you to tackle a wider range of challenges with ease.

Driving Innovation in LLM Applications

Finally, this feature request is about driving innovation in the field of LLM applications. By providing a more streamlined and flexible way to use LLMs, we're empowering developers and researchers to push the boundaries of what's possible. Whether you're building cutting-edge AI-powered tools or conducting groundbreaking research, the --model flag will help you achieve your goals more efficiently and effectively.

So, what do you guys think? Let's get this conversation rolling and make Crush even more awesome!