Not intended for use, this is more to publicize my meager findings as the community slowly explores hyperparameter space in the hopes that it helps someone out. It seems training specific new concepts, especially characters, is a lot harder than doing alteration of existing concepts/bias towards style for SDXL.

There's a good chance I am mistaken about how certain parts of training work and have no idea what I'm talking about, so if I've made an error please let me know (preferably with some citations).

Dataset

Sourced mainly from Danbooru, some have had light cleaning or cropping.

Tagging done with wd14-swinv2-v2 threshold 0.15 with manual pruning after. Custom token hiryuuchan put in front w/ keep token 1 enabled.

124 images at 12x repeat/epoch
213 images at 4x repeat/epoch
41 images at 1x repeat/epoch

(2381 images/epoch w/ repeats counted)

v3

LoHa/Conv test mainly based on parameters lifted from https://civitai.com/models/120177/akiyama-mizuki-or-project-sekai

In hindsight, it seems it's better to train conv layers as well since XL and finetunes are still not super good at anime style, so you basically have to teach it the style at the same time to get your character attributes to be picked up properly.

Also I tried out Prodigy optimizer for the first time. It seems impressive.

Base Checkpoint

CounterfeitXL-α

Still haven't had time to try anything else.

Key Training Parameters [w/ commentary]

Trained using kohya-SS Lycoris training implementation via Easy_Training_Scripts XL branch by Derrian_Distro

prior_loss_weight = 1
mixed_precision = bf16 [required]
xformers = true [v0.0.20]
gradient_checkpointing = true [save some vram so batch 2 works]
cache_latents = true
no_half_vae = true [paranoia]
vae = sdxl_vae_v1.0-fixed.safetensors [using fixed vae to avoid any possible issues]
max_train_epochs = 10 [10 epoch * 1283 steps/epoch = 12830 steps total, probably overkill since I ended up using 5th epoch]

resolution = 1024 
batch_size = 2 [training conv is slower, this would've been 18 hours at batch 1]
network_dim = 64
network_alpha = 32.0
conv_dim = 8
conv_alpha = 4.0
algo = "loha"
min_timestep = 0
max_timestep = 1000 [No idea what these are intended for but it affects LR]

optimizer_type = "Prodigy" [Basically all the opt parameters lifted from za4beqsbv36z2s889's LoHa training]
lr_scheduler = "cosine" [Having LR get forced to pull back is probably beneficial imo]
learning_rate = 1.0
unet_lr = 1.0
text_encoder_lr = 1.0 [seems to have worked okay to train Tenc]
decouple = "True"
weight_decay = "0.1"
betas = "0.9,0.99"
use_bias_correction = "False"

min_bucket_reso = 640 [min used in SDXL training]
max_bucket_reso = 1536 [max used in SDXL training]
bucket_reso_steps = 64 [same step size used in SDXL training.  Yes technically you'll train on buckets that didn't exist in SDXL training but it's probably better than something even more arbitrary.]
[upscaling allowed/no no upscale flag, I'm more concerned with learning broad features than learning upscale artefacts atm.]
[no noise params specified, defaults to SDXL training noise params]

LR & Loss Plots

LR peaks around 8e-4 and then basically follows the cosine curve down

v1

Base Checkpoint

CounterfeitXL-α

Counterfeit is the best one for anime style I've tried generating with at time of starting this training, but new finetunes keep coming out faster than I can keep up.

Key Training Parameters [w/ commentary]

Trained using kohya-SS LoRA training implementation via Easy_Training_Scripts XL branch by Derrian_Distro

prior_loss_weight = 0.9 [increase fitting to dataset compared to 1.0 value]
mixed_precision = bf16 [required]
xformers = true [v0.20]
gradient_checkpointing = true [to reduce vram usage, technically not needed for 24 GB VRAM]
no_half_vae = true [paranoia]
vae = sdxl_vae_v1.0-fixed.safetensors [using fixed vae to avoid any possible issues]
max_train_epochs = 10 [10 epoch * 2381 steps/epoch = 23810 steps]
resolution = 1024 
batch_size = 1
network_dim = 128 [starting high because it's not clear how much is too little/too much yet]
network_alpha = 64.0 [seems overfitting/"frying" occurs easily, lowering alpha may help. I may try lower alpha in future]
max_timestep = 1000 [No idea what this is intended for but it affects LR]
network_train_unet_only = true [I'm not convinced anyone has figured out good ways to train the dual CLIP models of XL, so for now focusing on UNet.  However it's quite limiting to not train text encoder.] 
optimizer_type = "AdaFactor" [Currently usable, so I use it]
lr_scheduler = "adafactor" [see plot below of LR]
scale_parameter = "True" [no clue what this actually does]
relative_step = "True" [no clue what this actually does]
warmup_init = "True" [no clue what this actually does, but warmup is probably good for avoiding weird things from happening early]
learning_rate = 2e-4 [fairly low LR because it's so easy for it to go to shit with higher]
min_bucket_reso = 640 [min used in SDXL training]
max_bucket_reso = 1536 [max used in SDXL training]
bucket_reso_steps = 64 [same step size used in SDXL training.  Yes technically you'll train on buckets that didn't exist in SDXL training but it's probably better than something even more arbitrary.]
[upscaling allowed/no no upscale flag, I'm more concerned with learning broad features than learning upscale artefacts atm.]
[no noise params specified, defaults to SDXL training noise params]

LR & Loss Plots

LR peaks at 10k steps/about epoch 4. It might not be a coincidence this is when generations start resembling Hiryuu more strongly (see post w/ xy grids).