NerdyRodent

YuE - FOSS Music Generation!

Added 2025-02-08 18:56:23 +0000 UTC

Video - https://youtu.be/6FBnKIjqT04

Open Source music generation at home takes another step forward thanks to the change of license for YuE, which is now Apache 2.0!

Like Suno or Udio, you can provide a set of lyrics along with some tags in order to create some AI music.

Text prompts aren't your only control though, as you can also upload vocal or instrumental audio clips (30s) to use instead!

All this for free, at home, on your own computer. It's like getting infinite credits :)

You'll need Linux or MS Windows and at least a 6GB VRAM GPU.

Various options exist for running YuE, such as YuEGP which features a gradio interface:

Example screenshot from YuEGP

Note that I've not tested the Microsoft Windows versions, but I imagine the easiest way to use YuE on Windows is probably Visions of Chaos, followed by YuE for Windows.

The others are pretty easy to get going too, with everything being the typical "conda create & pip install", but that may not be everyone's cup of tea. I am British though, so it is ;)

YuE-exllamav2 features a special requirements file with pre-compiled binaries for AMD / ROCM card owners. Not 100% if the others will work without an NVidia card.

Installation Notes

1 - The Original Repo

The first repo I installed was the original one. I used pip to install torch rather than conda, but their way should work just fine as well!

conda create -n yue python=3.12
conda activate yue
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
cd YuE/inference/
git clone https://huggingface.co/m-a-p/xcodec_mini_infer

For MS Windows, there are the flash-attention pre-compiled wheels, or you can build it yourself in around 2 hours - https://huggingface.co/lldacing/flash-attention-windows-wheel

To avoid this, use either Visions of Chaos, YuEGP or YuE-exllama

The original repo has only a command interface, e.g.

cd YuE/inference/
python infer.py \
    --cuda_idx 0 \
    --stage1_model m-a-p/YuE-s1-7B-anneal-en-cot \
    --stage2_model m-a-p/YuE-s2-1B-general \
    --genre_txt ../prompt_egs/genre.txt \
    --lyrics_txt ../prompt_egs/nr_lyrics.txt \
    --run_n_segments 2 \
    --stage2_batch_size 4 \
    --output_dir ../output \
    --max_new_tokens 3000 \
    --repetition_penalty 1.1

Running the command automatically downloads the models.

MS Windows users should also note that "\" symbolises a new line escape in Linux. The equivalent character in MS Windows is the carat "^".

Test 1 took 15 mins on a 3090 and used 16GB VRAM.

2. YuEGP

Command interfaces are great for automation, but if you want a web / gradio interface, then YuEGP is a great option.

This version is also faster and has lower VRAM requirements.

Installation was smooth up until running the patch transformers.sh/.bat assumes venv. It's a simple copy command, but assumes venv over conda.

To find your conda transformers location, run:

pip show transformers

Location: /home/nerdy/anaconda3/envs/yue/lib/python3.12/site-packages

I then copied the files into the required location for conda:

cp transformers/generation/* /home/nerdy/anaconda3/envs/yue-v2/lib/python3.12/site-packages/transformers/generation/
cp transformers/models/llama/* /home/nerdy/anaconda3/envs/yue-v2/lib/python3.12/site-packages/transformers/models/llama/

For the test, I used profile 1 and for uploading my voice I used ICL mode:

python gradio_server.py --profile 1 --compile --icl

Test 1 took 4mins 30 seconds, and used around 14GB VRAM.

3. YuE-exllamav2

This one doesn't have a gradio interface, but YuE-exllamav2-UI does. The UI version is a fork of the non-UI version, but is focused on docker.

If you do like docker, then YuE-exllamav2-UI is for you!

I'm not keen on docker myself, but found just copying the interface file from the UI version to the non-UI version worked a charm. No docker required :)

This repo was updated after my original tests to include a much easier "wheels-based" requirements file. Isn't that just the way?

Recent updates included a new file for ROCM, so AMD users should give this a go!

The latest requirements.txt file also features pre-compiled wheels for MS Windows, so you should get triton and flash attention 2 installed easily - even on MS Windows!

Some people may not like pre-compiled binaries from unknown sources, so it's just an option and the original requirements file was renamed.

For inference, I simply copied the src/yue/interface.py file from YuE-exllamav2-UI into src/yue (for YuE-exllamav2) and then edited it to point locally:

BASE_REPO_DIR = "/home/nerdy/github/YuE-exllamav2"
BASE_YUE_DIR = "/home/nerdy/github/YuE-exllamav2/src/yue"
BASE_MODELS_DIR = "/home/nerdy/github/YuE-exllamav2/m-a-p"
BASE_OUTPUTS_DIR = "/home/nerdy/github/YuE-exllamav2/outputs"
BASE_INPUTS_DIR = "/home/nerdy/github/YuE-exllamav2/inputs"

The interface also needs link xcodec_mini_infer in src/yue/xcodec_mini_infer instead, so I linked it in.

The final thing was getting it to see the models which had already been downloaded automatically by the other versions.

Interestingly enough, the huggingface-cli command simply copied the files from the cache, so that saved a bit of time downloading:

huggingface-cli download m-a-p/YuE-s1-7B-anneal-en-icl --local-dir ./m-a-p/YuE-s1-7B-anneal-en-icl
huggingface-cli download m-a-p/YuE-s1-7B-anneal-en-cot --local-dir ./m-a-p/YuE-s1-7B-anneal-en-cot
huggingface-cli download m-a-p/YuE-s2-1B-general --local-dir ./m-a-p/YuE-s2-1B-general

Test 1 took 4mins 40 seconds, and used around 16GB VRAM.

5090 Owners

I don't have one so I can't test, but I'm fairly sure CUDA 12.4 is too old for the 5090 ;) CUDA 12.8 should work, but if using Python > 2.5.1, do note that in 2.6, model loading defaults change from false to true (more secure). This means you may need to edit some code to get things to load.