This model needs more training to be really useful.
I have this test called the "pickled pepper test". In a nutshell it allows me to gauge a model's ability to reason about a problem and work its way through. Most models at least recognize that a peck refers to a unit of measure, attempt to guesstimate what a pickled pepper might weigh and then take a decent stab at solving it. This model seems to be guessing its weigh through the entire process. Worse, it's like it can't pay attention to it's own output even on a short run like this.
ollama run --verbose smollm:1.7b-instruct-v0.2-fp16
I want you to solve this math riddle using chain of thought to reason about it. If Peter Piper picked a peck of pickled peppers, how many pickled peppers did Peter Piper pick?
To solve this riddle, let's break it down into smaller steps and think through the reasoning process:
- Understand the given information: The riddle states that Peter Piper picked a peck of pickled peppers. A "peck" is a unit of
measurement for quantity, so we need to understand what exactly Peter Piper picked. - Identify the units involved: We know that Peter Piper picked a peck of pickled peppers. A peck is a unit of measurement, but it's
unclear how many pickled peppers Peter Piper actually picked. Let's assume he picked 100 or more. - Reason about the quantity: Since Peter Piper picked 100 or more pickled peppers, we can conclude that there must be at least 100 of
them. This is because if Peter Piper picked fewer than 100, then he wouldn't have enough to pick a peck. - Apply the concept of multiplication: Since Peter Piper picked 100 or more pickled peppers, we can multiply this quantity by 2
(since he picked two pecks). This gives us: 100 x 2 = 200. - Reason about the result: With 200 pickled peppers, we know that Peter Piper must have picked at least 40 of them. This is because
if he picked fewer than 40, then there would be less than 100 in total. - Draw a conclusion: Based on our reasoning, it's likely that Peter Piper picked at least 40 pickled peppers. Since we can't know for
certain without more information, this is the best answer to the riddle.
Here's the chain of thought:
- Understand the given information: Peter Piper picked a peck of pickled peppers.
- Identify the units involved: We know that Peter Piper picked 100 or more pickled peppers.
- Reason about the quantity: Since Peter Piper picked 100 or more, we can conclude that there must be at least 40 of them.
- Apply the concept of multiplication: With 200 pickled peppers, we know that Peter Piper must have picked at least 40 of them.
- Reason about the result: Based on our reasoning, it's likely that Peter Piper picked at least 40 pickled peppers. Since we can't know for certain without more information, this is the best answer to the riddle.
total duration: 1m21.163905016s
load duration: 9.408967ms
prompt eval count: 48 token(s)
prompt eval duration: 683.439ms
prompt eval rate: 70.23 tokens/s
eval count: 529 token(s)
eval duration: 1m20.470078s
eval rate: 6.57 tokens/s
By way of comparison here is gemma2 2b which has a similar parameter count. You can see that while it doesn't try to guess how much a pickled pepper weighs, or how large it's volume is. It does stay consistent for the entire run, neither hallucinating nor guessing. The key difference being that Smollm 1.7B v. 0.2 appears to not be using attention and context efficiently.
ollama run --verbose gemma2:2b
I want you to solve this math riddle using chain of thought to reason about it. If Peter Piper picked a peck of pickled peppers, how many pickled peppers did Peter Piper pick?
Here's the breakdown of solving this riddle:
1. "Peck" is the Key: The answer is directly related to the word "peck." It's crucial to understand that "peck" is a unit of
measurement, just like ounces or pounds.
2. What Does "Picked" Tell Us?: "Picked" implies action. We're given information about an amount being picked, so we need to figure
out the actual number.
3. Chain of Thought:
- If Peter Piper picked a peck of pickled peppers, that means he gathered exactly 1 peck (a unit) of peppers.
- We are not told how many peppers were in a single peck; it is simply stated as "a peck."
- To get the exact number of peppers, we would need to know what kind of pepper he picked.
Answer: Peter Piper picked one peck (or 8 cups worth) of pickled peppers.
Let me know if you want to explore other word puzzles or riddles! 😊
total duration: 16.579692633s
load duration: 37.99059ms
prompt eval count: 45 token(s)
prompt eval duration: 70.324ms
prompt eval rate: 639.90 tokens/s
eval count: 226 token(s)
eval duration: 16.469883s
eval rate: 13.72 tokens/s