Hugging Face's logo

Jemnite
/
experiments

Text-to-Image
Safetensors
stable-diffusion
Model card Files Community
experiments / README.md
Jemnite's picture
Jemnite
Update README.md
acfb429 verified
preview code
|
raw
history blame contribute delete
17.3 kB
metadata
license: bigscience-openrail-m
tags:
  - stable-diffusion
  - text-to-image
  - safetensors

Mix Records

Notes

Mix-GEM

MIX-GEM-T1

MyMIX-GEM-T1(CSRB7+INC4):

(U) AnyLoRA + ALunarDream2.0 = AnyLunaLoRA
Weights: 1,1,1,1,1,1,1,1,1,1,1,1,0,0,0.1,0.2,0.3,0.4,0.5,1,1,1,0.8,0.8,0.8

(U) AnyLunaLoRA + AmedirabV2 = ALunamediraExperimentbV2
Weights: 0,0,0,0,0,0.08,0.08,0.19,0.07,0.06,0.09,0.23,0.25,0,0,0,0,0.36,0.73,0.67,0.9,0.62,0.77,0.8,0.75

(C) piromizu-diffusion + SCHExcelsior_v20 * 0.5 =PiromizuExcelsior
(C) ALunamediraExperimentbV2 + PiromizuExcelsior * 0.5 = anyLunaPiromizuExcelsiorbV2

(U) anyLunaPiromizuExcelsiorbV2 + basil-mix = anyLunaPiromizuExcelsiorbV2_basil
Weights: 1,0.9,0.7,0.5,0.3,0.1,1,1,1,1,1,1,0,0,0,0,0,0,0,0.1,0.3,0.5,0.7,0.9,1

(L) anyLunaPiromizuExcelsiorbV2_basil + <thick-coat-cg-style:0.2> + <dpepv5-extracted:0.5> + <cutesexyrobutts_v2:0.7> + <incase2:0.4> = anyLunaPiromizuExcelsiorbV2_basil+LoRA
(C) anyLunaPiromizuExcelsiorbV2_basil+LoRA + ImpastoMix * 0.5 = MIX-GEM-T1ASTE(CSRB7+INC4)
(P) MIX-GEM-T1ASTE(CSRB7+INC4) + kl-f8-anime2 = MIX-GEM-T1

MIX-GEM-T1 failures

(C) mousemix + AnimePastelDream_Hard_bakedVae * 0.5 = mouseyPastelDream
(L) anyLunaPiromizuExcelsiorbV2_basil + <thick-coat-cg-style:0.2> + <dpepv5-extracted:0.5> = anyLunaPiromizuExcelsiorbV2_basil+LoRA  (failed)
(C) anyLunaPiromizuExcelsiorbV2_basil+LoRA  (failed) + mouseyPastelDream * 0.5 = MIX-GEM-T1  (failed)
Notes: Too rounded, no texture retention from Excelsior. CSRB flavor extremely weak. Couldn't do good muscles. Discarded.

(U) GishikiV1.1 + TMND-Mix_VI = MBW-RCOS Gishiki1.1 tmndVI
Weights: REVERSE_COSINE

(U) anyLunaPiromizuExcelsiorbV2_basil+LoRA (failed) + MBW-RCOS Gishiki1.1 tmndVI = MIX-GEM-Pro (failed)
Weights: COSINE
Notes: Style too clean, too anime like. Brightness could not be toned down. Discarded.

(U) GishikiV1.1 + Silicon29-dark = MBW-RCOS Gishiki1.1 Silicon29
Weights: REVERSE_COSINE

(U) MIX-GEM-T1ASTE(CSRB7+INC4) + BW-RCOS Gishiki1.1 Silicon29 = MIX-GEM-T1Pro
Weights: COSINE
Notes: Initial results were good, style was strong, very nice collection on off-set noise. However, mix was too unstable and attempts to use loras kept collapsing. Merging Gishiki into half impasto too much Gishiki? Without loras, style was too flat. Reverted back to T1.

MIX-GEM-T2

MyMIX-GEM-T2(MSTSEV5+GLKO4+CSRB7):

(C) mugen_a1 + snowfall_v1 * 0.5 = InfiniteSnowfall
(C) InfiniteSnowfall + ImpastoMix * 0.3 = InfiniteSnowfalImpastoM3

(U) GishikiV1.1 + Luas Night = MBW-RCOS Gishiki LunarNight
Weights: REVERSE_COSINE
(L) MBW-RCOS Gishiki LunarNight + <VisNoctisLGHT:1> + <JW:1> + <pikas-lighting-isolation:1> + <breakA-lighting-isolation-768:1> + <BeautifulDetailedEyesFN:1> = MBW-RCOS Gishiki LunarNight NRadianced

(U) InfiniteSnowfalImpastoM3 + MBW-RCOS Gishiki LunarNight NRadianced = MBW-COS InfiniteSnowfall GishikiLuaNRadianced
Weights: COSINE
(P) MBW-COS InfiniteSnowfall GishikiLuaNRadianced + kl-f8-anime2 = EternalWinter_WintersEve
(L) EternalWinter_WintersEve + <mstmix_v70:0.5> + <suzuki kenya_768_e8:0.4> + <style_CuteSexyRobutts [MockAi - v1.0]:0.7> = MIX-GEM-T2

MIX-GEM-T2A

MyMIX-GEM-T2(9-AAAAAA)A:

(C) InfiniteSnowfall + ImpastoMix * 0.5 = InfiniteSnowfalImpasto

(U) 9-AAAAAAA + Luas Night = MBW-RCOS 9-AAAAAAA LunarNight
Weights: REVERSE_COSINE
(L) MBW-RCOS Gishiki LunarNight + <VisNoctisLGHT:1> + <JW:1> + <pikas-lighting-isolation:1> + <breakA-lighting-isolation-768:1> + <BeautifulDetailedEyesFN:1> = MBW-RCOS 9-AAAAAAA LunarNight NRadianced

(U) InfiniteSnowfalImpasto + MBW-RCOS 9-AAAAAAA LunarNight NRadianced = EternalWinter_WintersEve2
Weights: COSINE
(L) EternalWinter_WintersEve + <mstmix_v70:0.5> + <suzuki kenya_768_e8:0.4> + <style_CuteSexyRobutts [MockAi - v1.0]:0.7> = MIX-GEM-T2A

MIX-GEM-T2A-Experimentation

MIX-GEM-T2A-LMBon = + <lora:oda-non-828it-novae:1:LIMB>
MIX-GEM-T2A-NFon = + <lora:oda-non-828it-novae:1:NF>
MIX-GEM-T2A-NFon2M = MIX-GEM-T2A-NFon + <WhiteMelancholy:1>
MIX-GEM-T2A-NFon2O = MIX-GEM-T2A-NFon + <OilPaint:1>

None of these are worth it, they screw with the unique quality of T2A too much without a consistent track record in fixing fingers (though they do help, it's not worth the tradeoff)

MIX-GEM-T3

MyMIX-GEM-T3(CSRB7+KRKVNBG3+GALKFN4+MSTSEV55+BG):

(C) mugen_a1 + snowfallSP2 * 0.5 = InfiniteSnowfallSP2
(C) InfiniteSnowfallSP2 + ImpastoMix * 0.5 = InfiniteSnowfalImpastoSP2
(U) 9-AAAAAAA + MBW-F_ANATOMYplusLoRA summerFreeStyleH schExcelsior3 = MBW-RCOS 9-AAAAAAA summerFreeStyleplus
Weights: REVERSE_COSINE

(L) MBW-RCOS 9-AAAAAAA summerFreeStyleplus + <JW:1> + <pikas-lighting-isolation:1> + <breakA-lighting-isolation-768:1> + <BeautifulDetailedEyesFN:1> = MBW-RCOSplusLoRA 9-AAAAAAA summerFreeStyleplus
(U) InfiniteSnowfalImpastoSP2 + MBW-RCOS 9-AAAAAAA summerFreeStyleplus = MBW-COS InfiniteSnowfalImpastoSP2 9-AsummerFreeStyle+
Weights: COSINE

(L) BW-COS InfiniteSnowfalImpastoSP2 9-AsummerFreeStyle+ + <(MockAi) CuteSexyRobutts [147302]:0.7> + <suzuki kenya_768_e8:0.4:FN> +  <mstmix_v70:0.55> + <krekkovV2_e8:0.3:NBG2> + <BG2_dpep4+silicon29dim64:0.8> = MIX-GEM-T3

MIX-GEM-T2_2

MyMIX-GEM-T2_2(CSRB7+KRKVNBG3+GALKFN4+MSTSEV55+BG):

(U) MBW-F_ANATOMYplusLoRA summerFreeStyleH schExcelsior3 + SoushikiV1 = MBW-RCOS summerFreeStyleplus Soushiki
Weights: REVERSE_COSINE

(L) MBW-RCOS summerFreeStyleplus Soushiki + <JW:1> + <pikas-lighting-isolation:1> + <breakA-lighting-isolation-768:1> + <BeautifulDetailedEyesFN:1> = MBW-RCOSplusLoRA summerFreeStyleplus Soushiki
(U) InfiniteSnowfalImpastoSP2 + MBW-RCOSplusLoRA summerFreeStyleplus Soushiki = MBW-COS InfiniteSnowfalImpastoSP2 summerPlusFreeStyleSoushiki
Weights: COSINE

(L) MBW-COS InfiniteSnowfalImpastoSP2 summerPlusFreeStyleSoushiki + <(MockAi) CuteSexyRobutts [147302]:0.7> + <suzuki kenya_768_e8:0.4:FN> +  <mstmix_v70:0.55> + <krekkovV2_e8:0.3:NBG2> + <BG2_dpep4+silicon29dim64:0.8> = MIX-GEM-T2_2

I am mega stupid and merged the A and B wrong in the reverse_cosine merge which caused T3 to be mega-lewd and give the summer-mix too much influence over the IN blocks. Largely fixed.

EternalWinter-Pruned-Branches

(L) InfiniteSnowfalImpasto + <ML:0.2> + <WhiteMelancholy:0.4> = InfiniteSnowfalImpastowithAShadeofMelancholy
(U) InfiniteSnowfalImpastowithAShadeofMelancholy + MBW-RCOS 9-AAAAAAA LunarNight NRadianced = EternalWinter_ShadedMelancholy
Weights: COSINE

Inferior to WintersEve2 in terms of dynamic impact. More consistent though.

(L) MBW-RCOS Gishiki LunarNight + <Anti White LORA:1> + <JW:1> + <pikas-lighting-isolation:1> + <breakA-lighting-isolation-768:1> + <BeautifulDetailedEyesFN:1> = MBW-RCOS 9-AAAAAAA LunarNight aRadianced
Weights: REVERSE_COSINE
(U) InfiniteSnowfalImpastowithAShadeofMelancholy + MBW-RCOS 9-AAAAAAA LunarNight NRadianced = EternalWinter_HolyNight

Inferior to WintersEve2 and ShadedMelancholy. Neither consistent nor impactful.

EternalWinterL-Pruned-Branches

(L) InfiniteSnowfalImpasto + <WhiteMelancholy:0.2> + <octans:0.4> = EternalWinterL_Melancholic2Octans4

Stability reasons.

(L) InfiniteSnowfalImpasto + <WhiteMelancholy:0.2> + <niji-style-enhancer:0.4> = EternalWinterL_Melancholic2Niji4

Niji-style enhancer lora is not actually good for anything but Niji style.

Copying Yohan in Summer

Summer Free Style

(U) Yohan-Diffusion + AmalgamMix_ =  yohanAmalgumH
Weights: Lost. They used to be on the AmalgamMix page but that's been deleted from Civitai and the Model basically only exists in the Checkpoint Archive and the Arcane Archives repositories.
(C) MixSummerSolstice_v1 + yohanAmalgumH * 0.5 = summerFreeStyle
(C) summerFreeStyle + <add_detail:0.4> + <yohan1754-(free-style)-1341it-novae:0.35>= summerFreeStyleH

(U) summerFreeStyleH + schExcelsior_v30 =  MBW-F_ANATOMY summerFreeStyleH schExcelsior3
Weights: 0,0,0,0,0,1,1,1,1,0,1,0,0,0,0,0,0,1,1,0,0,1,0,0,0
(L) MBW-F_ANATOMY summerFreeStyleH schExcelsior3 + <krekkovV2_e8:0.5:LIMB> + <hizake_e8:0.5:FN> + <(MockAi) CuteSexyRobutts [147302]:0.7:NF> = MBW-F_ANATOMYplusLoRA summerFreeStyleH schExcelsior3

It's okay. I like it, still needs more experimentation. I think the lora combination is worth investigating but I'd like some Suzuki Ken'ya in there.

Mung Bean Soup

I don't even know why you need the recipe for this, I listed it on the civitai page. You can have it I guess but if you can't access civitai I have to wonder where you're even getting the checkpoint from.

Soup Recipe

(C) SanXianWonton2 + exquisiteDetails_art * 0.5 = SanXianWonton_equisiteArt
(U) SanXianWonton_equisiteArt + Lua's Night = MBW-YUKI e0479afa37 LuaN
Weights: 0,0.3,0.3,0.3,0.4,0.1,0.3,0.0,0.0,0.0,0.0,1.0,0.9,0.8,0.7,0.0,0.0,0.0,1.0,0.3,0.3,0.3,0.4,0.5,1.0,0.0
(U) SoushikiV1 + holyforever-768 = MBW-RCOS Soushiki holyforever
Weights: REVERSE_COSINE
(U) MBW-YUKI e0479afa37 LuaN + MBW-RCOS Soushiki holyforever = MBW-COS 2733bd1a18 84036bfb4a
Weights: COSINE
(C) MBW-COS 2733bd1a18 84036bfb4a + <hll6.3-st2-320-f:1.0> = LuDouTang_v11

For v1, just skip steps 3 and 4. It's just a shitty thing I whipped up, as far as SD1.5 goes, with pony diffusion v6 out and animagine 3 out there's basically no point in continuing further, especially with non vpred and zsnr models.

Tips and Tricks

Notes on LoRA tuning

Easy go-to combination is with a stylizer (dominant weight) + smoother (auxillary weight). For example, MockAi CSRB is the stylizer in this combination and the smoother is mstmix_v70. This is also the case for CSRB + Incase as in T1, where Incase is the smoother, and CSRB + Nat-the-lich/Suzuki Ken'ya + Rizdraws. In the case of the stylizer, they should point towards what features you want the model to inherit, in the case of Suzuki Ken'ya and CSRB it would be something like body type and the smoother should point towards desired texture. You can add on additional stylizers as long as the mix continues to seem stable but the more delta weights you throw on the closer the end result comes close to catastrophic forgetting.

Working observed combinations:

  • <LoRA:NAI-cutesexyrobutts:0.7> + <LoRA:incase2:0.4>
  • <LoRA:suzuki kenya_768_e8:0.4> + <LoRA:rizdraw-2295it-novae:0.3>
  • <LoRA:NAI-cutesexyrobutts:0.8> + <LoRA:nat-the-lich-1210it-novae:0.4>
  • <LoRA:style_CuteSexyRobutts [MockAi - v1.0]:0.7> + <LoRA:mstmix_v70:0.4>
  • <LoRA:style_kleggs:0.8> + <LoRA:mxlora_v185:0.55>

WhiteMelancholy can be injected into anything for oil-like texture. It is better than most comparative oil loras and more or less universally compatible.

Cosine-Reverse_Cosine Noise-Offset injection

Discovered this trick from SomethingV2_2. Good trick to expand the color-range and dynamic brightness. You run reverse cosine merge between a a high detail base mode usually based off some variant of detailedproject (in this case Gishiki, which is a tuned up version of hell-dpep5) and high brightness contrast model with some degree of offset noise baked in (in this case Lua's Night). Then you cosine merge it in. The result is a model with more details and brightness range.

If you don't want to bother to adjust the mergefodder yourself, you can go ahead and just use MBW-RCOS Gishiki LunarNight NRadianced. It should work out fairly okay.

MIX-Pro Recipe formula

(Roughness/detail merge + Body type merge) + Texture type merge

Original example:

  1. andite/mikapikazo (detail) + andite/CSRB (body type) = mikacute
  2. mikacute + nuigurumi/basil_mix (abyss-orange-mix trick) = mikacutebasil_mix!U-net
  3. CTD/mouseymix (primary texture) + andite/pastel-mix (compatible texture model stabilizer) = mouseypastel (texture merge)
  4. mikacutebasil_mix!U-net + mouseypastel = MIX-Pro-V3

Example with LoRA tuning:

  1. andite/mikapikazo (color detail) + andite/piromizu (feature detail) = mikapiro (detail merge)
  2. andite/CSRB (proportion detail) + andite/yohan (musculature detail) = (body type merge) = csrbyohan
  3. mikapiro + csrbyohan = mikapirocsrbyohan
  4. mikapirocsrbyohan + nuigurumi/basil_mix (abyss-orange-mix trick) = mikapirocsrbyohan_Basil
  5. mikapirocsrbyohan + thick-coat-cg-style (texture LoRA) + loconlora-airconditioner-style (brightness/scenery LoRA) = mikapirocsrbyohan_Basil+LoRA
  6. mikapirocsrbyohan_Basil+LoRA + mouseypastel = MIX-Pro-V4

Merging inherently averages out distinct qualia so if your original characteristics are being lost in the merge, you can either adjust the ratio to try to preserve them or alternatively, exaggerate the details in the mergefodder to disproportionate lengths via LoRA injection. Tuning after final-model merge is the exception. At this point your model should be stable. If it's not you may have to revisit some steps. If you are getting too much from various tuning-use loras, you should at this point consider isolating LoRA blocks.

Overall it's important to have a vision of what model you want to end up with/what characteristics you can bring across from the models you're merging in. You have to have a goal to move towards.

LoRA Block Weight

It's possible to selectively isolate LoRA block weights in order to only hit certain blocks. This is a list of all the block weights I've collected so far. None of the custom weights are totally reliable, they're all based on observing black box interactions, but they sometimes work.

BG:    0,0,0,0,1,0,0,1,0,0,0,0,1,1,1,1,1
NBG:   1,1,1,1,0,1,1,0,1,1,1,1,0,0,1,0,0
BG2:   1,0,0,0,1,1,0,0,0,0,0,1,1,1,1,0,0
2BG:   0,1,1,1,0,0,1,1,1,1,1,0,0,0,0,1,1
NF:    1,1,1,1,1,1,1,1,0,0,0,1,1,1,1,1,1 
NFD:   1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0
NFDB2: 1,1,1,1,1,1,1,1,0,0,0,1,0,0,0,1,1
NFB:   1,1,1,1,0,1,1,0,0,0,0,1,0,0,0,0,0 
FN:    1,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0
NONE:  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
ALL:   1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
INS:   1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0
IND:   1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0
LGHT:  1,0,0,0,0,1,0,0,1,0,1,0,0,1,0,0,0
INALL: 1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0
MIDD:  1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0
OUTD:  1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0
OUTS:  1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1
OUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1
NOKE:  1,0,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0
LIMB:  1,0,0,1,0,1,0.5,0,0.5,0.5,1,0,0,0.5,0,0,1,1,1,1,0,0,0,0,0,0
HAIR:  1,0,0,1,0,1,0,0,0,0,0,0,0,1,0,0,0,1,1,0,1
HLFALL:0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5

Custom Weights

  • LIMB: Should affect positioning primarily. Might have an effect on anatomy
  • NF/FN: Stands for No Face and Face, respectively. NF might be a little bit more reliable than FN, which is mostly just guesswork.
  • NOKE: Possibly better face weights? Seems to affect the neck and up area mostly.
  • LGHT: Affects lighting mostly. This one is pretty reliable in my experience, though I've only really used it with VisNoctis mostly.
  • BG/BG2: Background? This is mostly guesswork from other people's research. I've used them a little, they seem to have some effect, but it seems like the main power of background loras is mostly to pull the 'camera' out and show more of the scene so it's dubious how much an effect you can have on the background once you've isolated the blocks.
  • NBG/NBG2: All blocks other than the background ones. Should do the opposite.
  • HAIR: Should be pretty reliable, considering it's based on Mooey's research.
  • NFD/NFDB2/NFB: Some weird block weight combinations combining the NF weights with various stuff. NFD is combining NF with OUTD, NFDB is combining NFD with background, NFDB2 is combining NFC with BG2.

My personal opinion is that I consider the second of these slightly more reliable than the first. The FN weights are probably the most used, but I consider them possibly the least verifiable. They were sourced from this fanbox article which is now inaccessible. They're built around finger-fixing primarily, using the NF to avoid changing the core body shape (and FN is developed as a corollary that it should change the body shape.) LGHT is slightly more reliable, considering that I have used it and it has had good observable effects. BG is a big mystery. I think it works, but that might just be one-off ancedotal results speaking.

If all else fails, you can used the default block weights like INS, IND, MIDD, OUT, OUTS. These are guaranteed to have some sort of consistent effect, though I don't really know what they do. OUTD seems to be pretty good with isolating color changes to my limited experience.