Music Audio and Voice

Loop Copilot: AI-driven music generator

editing and modifying the generated music, adding or deleting instruments, adding sound effects, etc.


Loop Copilot: AI-driven, from music generation to editing, synthesis, and ensemble in one go

Loop Copilot is a system for generating music using natural language. It not only allows you to use natural language to generate the music style, rhythm or melody you want, but also supports further editing and modification of the generated music through multiple rounds of dialogue.

This includes editing and modifying the generated music, adding or deleting instruments, adding sound effects, etc.

How to generate music:

Describe requirements: You can describe the music you want through natural language, such as “I want a light jazz song.”

Generate music: The system will generate corresponding music based on your description.

How to edit and modify:

Suggest modifications: If you feel that the generated music is not perfect enough, you can use natural language to propose modifications, such as “I think the drum beats can be more powerful.”

System response: Loop Copilot will make corresponding adjustments based on your feedback.

Other advanced features:

In addition to basic generation and editing functions, Loop Copilot also supports more advanced music production tasks, such as reverb, adding instruments, etc. You only need to describe your needs in natural language, such as “add some echo effects” or “I want the piano part to be more prominent”, and the system will understand and execute it.

  • 🔄 Regenerate specific areas: If the user is not satisfied with a specific piece of music, they can ask the system to regenerate it.
  • 🎺 Add or remove specific instruments: Users can add or remove specific instrument sounds through natural language commands.
  • 🎵 Adding sound effects: Although the document does not explicitly mention which specific sound effects are supported, it does mention supporting fine-grained music editing, which may include various sound effects.
  • 🌐 Global attribute modification: Through the global attribute table (GAT), users can change some global attributes of the entire song, such as rhythm, key, etc.

Project Paper

Project address:

Working principle:

Loop Copilot provides a highly interactive and user-friendly environment that enables users to easily generate and modify musical compositions. The system does not rely solely on one AI model, but integrates multiple back-end models designed to perform specific tasks.

In order to ensure the continuity and consistency of the music creation process, Loop Copilot maintains a global attribute table for recording various attributes related to music.

  • 🔠 Input preprocessing: First, the system receives the user’s natural language input and performs preprocessing to unify the input form.
  • 🔍Task analysis: Next, these inputs are analyzed using large language models (LLM) to determine the user’s intent and the specific tasks that need to be performed.
  • ⚙️ Task execution: Once a task is determined, the system will call one or more back-end models in a predetermined order to execute these tasks. Each of these models is specialized to perform a specific type of task, such as music synthesis, rhythm generation, etc.
  • 🎶 Response generation: After all tasks are executed, LLM will be responsible for generating a comprehensive output, which is usually a music clip or a natural language response for user interaction.
  • 📊 Global attribute table update: During the entire process, the global attribute table will be continuously updated to record all necessary information related to the current musical work, thereby ensuring the continuity and consistency of the entire creative process.

How does the system ensure the coherence of musical attributes during iterative editing?

Loop Copilot uses a special “global attribute sheet” to ensure the continuity of the music during iterative editing.

What is a global attribute table?

The global property table is a record table that saves all important information and settings related to the current music piece. For example, it will record the rhythm, melody style, instrument type, etc. you choose.

How does it work?

1. Record initial settings: When you first start creating music, all basic settings and selections will be recorded in this global property table.

2. Multi-round dialogue update: This table will be updated every time you have a multi-round dialogue with the system and make new choices or modifications. For example, if you select “Uptempo” on the first pass, and then say “I want to add some guitar sound” on the second pass, both information will be saved.

3. Maintain coherence: Because all information is properly recorded and updated, when the system generates or edits music, it will refer to this table to ensure that all elements are coordinated.

4. Iterative optimization: You can interact with the system multiple times, and you may have new ideas or modifications each time. The global property sheet will ensure that these changes are consistent with the previous settings, so that the entire musical composition sounds more coherent.

In this way, global attribute sheets help Loop Copilot ensure that various parts of the music work well together during multiple edits and revisions.

Loop Copilot: AI-driven music generator

Loop Copilot