You might be interested in this:

https://github.com/arcee-ai/mergekit/issues/198

I explain 3 different views on why downscaling the W_q and W_k matrices (and thus in turn scale the score matrix) might make sense to do in that thread.

The odd 0.7071067812 constant is just 1/sqrt(2) so the dot-product has the effect of halving the magnitude of all score matrix values.

dtype: float16
merge_method: passthrough
slices:
- sources:
  - layer_range: [0, 10]
    model: miqu-1-70b-sf
- sources:
  - layer_range: [10, 20]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [10, 30]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [20, 40]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [30, 50]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [40, 60]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [50, 70]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [60, 70]
    model: miqu-1-70b-sf
    parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
- sources:
  - layer_range: [70, 80]
    model: miqu-1-70b-sf

Copied from the mergekit thread:

Using temperature = 0, repetition penalty = 1 and no system prompt:

### USER
Two people are in a room alone with a machine that requires two people to operate. One person is operating the machine. The machine is functioning properly. What is the other person doing?

### ASSISTANT
The other person is also operating the machine. The statement "requires two people to operate" implies that both individuals are involved in its operation.

https://old.reddit.com/r/LocalLLaMA/comments/1aqxqfr/interesting_answers_i_got_from_miquliz120b_in_a/

### USER
Sally has 3 sisters. Each of her sisters has one brother. How many sisters does Sally have?

### ASSISTANT
Sally has 3 sisters. The information about their brothers is not relevant to the question. So, Sally still has 3 sister.

(this was a copy and paste fail from me, but he didn't fall for it! :grin:)

### USER
Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?

### ASSISTANT
Sally has 2 sisters.

Here's the reasoning:


1. Sally is a girl and has 3 brothers.
2. Each of her brothers has 2 sisters, which means each brother has one sister (Sally) in common with each other, and one more sister in addition to Sally.
3. Since there are 3 brothers, this accounts for 2 additional sisters (1 from each brother's perspective) + Sally herself, making it a total of 3 girls including Sally herself.

So, Sally has 2 sisters in total (excluding herself).

https://old.reddit.com/r/LocalLLaMA/comments/15kntrq/i_asked_a_simple_riddle_to_30_models/

### USER
Solve this family relationship quiz: Marie is Emma's parent. Emma is Carl's parent. Carl is Timothy's parent. Timothy is male. Who's Timothy to Marie?

### ASSISTANT
 Based on the information provided:

1. Marie is Emma's parent.
2. Emma is Carl's parent.
3. Carl is Timothy's parent.
4. Timothy is male.

From point 1, we know that Marie is either the mother or father of Emma (gender not specified).

From point 2, we learn that Emma is the mother or father of Carl (again, gender not specified).

From point 3, we find out that Carl is Timothy's parent. Since Timothy is male, this means that either Carl is his father or his mother. However, since Timothy has already been established as male in the problem statement, it follows that Carl must be Timothy's father.

Now let's put all these relationships together:

1. Marie -> Emma (parent-child)
2. Emma -> Carl (parent-child)
3. Carl -> Timothy (father-son)

Since we know that Marie is Emma's parent, and Emma is the parent of Carl, this makes Marie either the grandmother or grandfather of Timothy. Therefore, Timothy is Marie's great-grandchild or great-grandson/daughter depending on their gender. However, since Timothy has been established as male in the problem statement, it follows that he is Marie's great-grandson.

In conclusion, Timothy is Marie's great-grandson.

https://old.reddit.com/r/LocalLLaMA/comments/1atr1df/using_family_relationship_quizzes_to_measure_the/

https://pastebin.com/MZ7Aj7ZV

Now for the slightly more problematic puzzles this one is getting wrong. The old miqu-1 and miqu-120b could sometimes solve stuff like this if you added "Think step-by-step as this is a tricky problem:" on to the end:

### USER
Solve this family relationship quiz: Betty is Brenda's parent. Betty is Joe's parent. Joe is Charles' parent. Brenda is female. Who's Brenda to Charles? Think step-by-step as this is a tricky problem:

### ASSISTANT
 Let's break this down step by step:

1. Betty is Brenda's parent. This means either Betty is Brenda's mother or father.
2. Betty is Joe's parent as well, so either Betty is Joe's mother or father too.
3. Joe is Charles' parent, which means Joe is either Charles' mother or father.
4. Brenda and Joe are siblings since they share a common parent (Betty).
5. Since Brenda is female, neither she nor her sibling (Joe) can be Charles' father, so Joe must be Charles' mother.
6. Therefore, Brenda is Charles' aunt (either his aunt by blood or his half-aunt if Joe is his/her step-parent).

So Brenda is Charles' aunt (either biologically or through marriage).

### USER
Solve this family relationship quiz: Willie is Sara's parent. Willie is Zachary's parent. Zachary is Kyle's parent. Sara is female. Who's Sara to Kyle? Think step-by-step as this is a tricky problem:

### ASSISTANT
 Let's break this down step by step:

1. Willie is Sara's parent, so Willie is Sara's father or mother.
2. Willie is also Zachary's parent, so Willie is Zachary's father or mother as well.
3. Since Willie cannot be both the father and mother of Sara and Zachary (as they are not twins), we can conclude that either Sara and Zachary are siblings or cousins.
4. However, since there is no mention of any other siblings or spouses in the question, it's safe to assume that Sara and Zachary are siblings.
5. Now, we know that Zachary is Kyle's parent, so either Kyle is Zachary's son or daughter.
6. Since Sara and Zachary are siblings, this makes Sara either Kyle's aunt or aunt-in-law (if she married into the family).
7. However, since there is no mention of any marriages in the question, we can rule out the possibility of Sara being Kyle's mother-in-law.
8. Therefore, Sara must be Kyle's aunt.

So, Sara is Kyle's aunt.

### USER
Solve this family relationship quiz: Nathan is Timothy's parent. Nathan is Aaron's parent. Aaron is Edward's parent. Timothy is male. Who's Timothy to Edward? Think step-by-step as this is a tricky problem:

### ASSISTANT
Let's break this down step by step:

1. Nathan is Timothy's parent. This means that either Nathan is Timothy's father or mother.
2. Nathan is also Aaron's parent, which means that either Nathan is Aaron's father or mother as well.
3. Aaron is Edward's parent, so Aaron could be Edward's father or mother.
4. We know that Timothy is male, so Nathan cannot be Timothy's mother because same-sex marriage isn't mentioned in the question and we are assuming heterosexual relationships for simplicity here.
5. Since Timothy is male, Nathan must be Timothy's father (not mother).
6. Now, if Aaron is Edward's parent, then Aaron cannot be a woman because Timothy has already been established as male. So Aaron must be Edward's father.
7. Therefore, since Nathan is both Timothy and Aaron's parent, this means that Timothy and Aaron are siblings (either twins or close in age).
8. Since Aaron is Edward's father, then Timothy must be Edward's uncle (either his maternal or paternal uncle).

So to answer the question "Who's Timothy to Edward?" The answer is that Timothy is either Edward's maternal or paternal uncle.

This also doesn't seem to "lose" the information like the original miqu-120b did:

### USER
Can you summarise the ideas behind Pollack's 1990 "Recursive distributed representations" paper?

### ASSISTANT
 Sure, I'd be happy to summarize the main ideas in Pollack's 1990 paper "Recursive Distributed Representations."

In this influential paper, Jordan Pollack introduced the idea of using connectionist networks with recurrent connections to model hierarchical representations of natural language. The key insight was that by recursively composing simpler representations into more complex ones, a network could learn to represent complex structures such as sentences or even entire stories.

The main contributions of this paper include:

1. **Recursive compositionality**: Pollack proposed that language understanding and generation can be modeled using a connectionist architecture that recursively composes simpler representations into more complex ones. This allows the network to learn hierarchical representations of natural language, such as sentences or even entire stories.
2. **Connectionist networks with recurrent connections**: The model uses recurrent connections to allow information from previous time steps to influence current processing, allowing the network to maintain context over time and build up complex representations incrementally.
3. **Distributed representations**: Pollack proposed that distributed representations can capture the meaning of words, phrases, and even entire stories by distributing their meanings across many hidden units in a connectionist network.
4. **Recursive compositionality**: The model recursively composes simpler representations into more complex ones, allowing the network to learn hierarchical structures such as sentences or even entire stories.

Overall, Pollack's paper laid the groundwork for modern deep learning approaches to natural language processing and has had a lasting impact on the field of artificial intelligence research.

This is a question I have definitely asked the miqu-120b merge and he didn't know that Pollack's first name was Jordan and hallucinated something else (even though the original miqu-1 model did know his name).

He doesn't seem to have mentioned the key insight from the paper - compressing sequences to a fixed length vector (but I don't know if the original miqu-1 knew this either).