[ad_1]
Final week, MBW revealed a extensively shared report about TikTok and its mum or dad ByteDance‘s ambitions within the area of AI-generated music.
Inside that article, we instructed you concerning the intensive analysis carried out by ByteDance’s Speech, Audio & Music Intelligence (SAMI) workforce within the area of AI music.
A type of analysis initiatives describes a generative mannequin referred to as MeLoDy that was educated, in accordance with ByteDance’s SAMI researchers, on 257,000 hours of music (learn the paper right here) that got here from 6.4 million audio information.
We additionally instructed you about two US patents, secured by ByteDance in latest months, that present IP safety within the US for the Chinese language firm’s AI music-related applied sciences.
Now, we’ve uncovered an further batch of AI music-related patent and trademark filings from ByteDance that can increase various eyebrows within the music business – overlaying subjects together with “information mining” and “music era”.
Apparently, these US patents and trademark functions have been filed by a mysterious ByteDance affiliate referred to as Lemon Inc.
There’s not a whole lot of details about Lemon Inc. on-line. We do know, nevertheless, that (a) it’s integrated within the Cayman Islands, and (b) It’s been very busy over the previous couple of years submitting functions to the US Patent and Trademark Workplace. (It even utilized for a guide publishing-related trademark at the beginning of 2023).
Lemon Inc. will not be talked about particularly on ByteDance’s company construction diagram (see under, and right here).
Nevertheless, the World Mental Property Group, by way of its Arbitration reviews, just lately described Lemon Inc. as being “affiliated with Bytedance Ltd (“Bytedance”), an Web know-how firm that gives cellular functions (“apps”) corresponding to TikTok”.
Lemon Inc. is to not be confused with one other ByteDance product referred to as Lemon8, the Instagram/Pinterest rival launched within the US final yr, which some speculate is being pushed out there as a “back-up” within the occasion of a possible TikTok ban.
As reported by Axios final yr, Lemon8 was developed by Singapore-based Heliophilia Pte., Ltd., which is registered on the identical deal with as TikTok in Singapore.
Why Lemon Inc. is getting used as a car to use for logos and patents in the US on behalf of ByteDance is a query for an additional day.
For now, feast your eyes on what else ByteDance has been as much as…
Lemon Inc. PAtent 1: ‘Automated and quick era of music audio content material for movies’
In what gives the clearest glimpse at what ByteDance could be planning to make use of its AI music-related tech for, its affiliate Lemon Inc owns a patent in the US for a “technique of robotically and effectively producing music audio content material for movies”.
The patent utility was filed in July 2022, granted in September 2023, and is assigned to each Lemon Inc. and ByteDance (see under).
In accordance with the submitting, which you’ll learn for your self right here, this patent covers “methods that allow environment friendly good music era {and professional} music modifying”.
“After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video will probably be robotically generated. Such music or audio could also be royalty-free and could be of any size.”
Excerpt from patent
The submitting provides: “After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video will probably be robotically generated. Such music or audio could also be royalty-free and could be of any size.”
The submitting explains additional that the tech can extract “data from the video, whereby the extracted data contains data indicating movement velocity within the video, data indicating movement saliency within the video, data indicating scene transition within the video, and timing data related to the video”.
It will probably then generate a “plurality of units of music notes matching the plurality of segments of frames primarily based not less than partly on the extracted data utilizing a mannequin, whereby the mannequin is pre-trained and learns to correlate video movement velocity with music observe density, correlate video movement saliency with music observe energy, correlate video scene transition with music construction, and correlate video timing with music beat”.
Simply to additional spotlight the shut hyperlinks between Lemon Inc, TikTok and ByteDance, right here:
Amongst the named inventors of this tech within the patent submitting are Los Angeles-based researchers named Zhihao Ouyang and Daiyu Zhang, who in accordance with LinkedIn are a Senior Analysis Engineer at TikTok, and ML Analysis Engineer at ByteDance, respectively.
Lemon Inc Patent 2: Computing system and technique for music era
Lemon Inc additionally has one other patent pending for a “Computing system and technique for music era”.
This patent was filed by Lemon Inc USA in June 2022 and assigned to each Lemon Inc and ByteDance in September 2023 (see under). It was revealed in December.
The submitting explains that “Packages have been developed that may generate music primarily based on a lyric inputted by a consumer” however that the music generated by such packages “typically lacks musical qualities that many individuals respect, and thus isn’t very song-like”.
It continues: “For instance, auto-generated music from such packages can undergo from misalignments in lyrics and melody notes, scattered or disjointed group and track construction, mismatched rhythm tracks, and lack of a catchy repeating melody.
“Because of this, such packages haven’t achieved widespread use. Because of this, a barrier presently exists to speedy track improvement utilizing such packages.”
The patent utility, which you’ll see right here, describes “a music era system” that options “a processor and a reminiscence operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates”.
The invention additionally contains “a music era program saved within the reminiscence and executed by the processor to be configured to obtain a consumer enter of lyrics, determine a plurality of syllables within the lyrics, decide a syllable sample within the recognized plurality of syllables, match the syllable sample to a specific rhythm template of the plurality of rhythm templates”.
The invention is then proposed to “generate a melody primarily based on the chosen rhythm template, generate a music file encoding the melody and the lyrics, and output the music file encoding the melody and the lyrics”.
The diagram under units out the way it works:
Lemon Inc.’s Emblems
Along with its lively and pending patents within the area of AI Music, Lemon Inc. has additionally filed for a number of logos in the US that consult with AI, music era and music tech basically.
A type of logos listed on the US Patent and Trademark Workplace (USPTO) is for a service referred to as ‘BytePlus‘ which seems to have been formally registered in Might 2023.
TikTok and Lemon Inc mum or dad firm ByteDance launched BytePlus in 2021 reportedly as a brand new division to promote its AI know-how. ‘BytePlus’ is listed in ByteDance’s company construction diagram.
The trademark listed on the USPTO describes a “Platform as a Service (PaaS)” and covers what the submitting states are “pc software program platforms for producing and delivering personalised suggestions to prospects within the fields of synthetic intelligence, information mining, information analytics, and machine studying covers a variety of AI-related actions.
The trademark additionally covers “machine-learning primarily based music era, and machine language translation; software program consultancy providers regarding synthetic intelligence, information mining, information analytics, and machine studying”.
Elsewhere on the USPTO database, a trademark is registered to Lemon Inc. for Mawf, ByteDance’s machine-learning-driven music-making app that analyzes incoming audio alerts after which “re-renders” these alerts utilizing what it says is machine-learned fashions of musical devices.
The Mawf trademark was utilized for in July 2022 and formally registered by the USPTO in September.
The submitting covers “Laptop software program, specifically, downloadable software program for creating music movies, recorded music composition software program”.
Lemon Inc. has additionally filed (in Might 2023) two functions to trademark logos for what the filling seems to explain as a music-making app (see under). These logos resemble the logos of ByteDance’s Ripple app.
The appliance described within the pair of filings, which look like beneath overview at present, cowl “Downloadable software program for processing music and audio information; Downloadable music-composition software program; Downloadable software program for creating, recording, and modifying audio to create music and audio; downloadable software program for creating music and audio; Downloadable cellular utility software program enabling customers to share music and audio they created for different customers to view, entry, and remix inside the utility; Downloadable software program to be used in music and audio manufacturing“.
Each logos for various colour variations of the brand seem to have been filed in April and Might 2023 and have been lively as of January this yr.
Lemon Inc. additionally filed a trademark utility for ‘Ripple‘ in February 2023 and the mark was solely simply revealed on the USPO’s Trademark Official Gazette final month.
Ripple is the AI-powered music-making app launched by ByteDance in June final yr that may flip a hummed melody right into a track.
Apparently, Lemon Inc additionally filed a trademark utility again in 2021 for SoundOn, which is the DIY music distribution service launched by TikTok in March 2022.
SoundOn lets artists add their music on to TikTok and Resso, and it might probably additionally distribute artists’ music to platforms like Spotify and Apple Music.
And simply in case you’re nonetheless not satisfied about Lemon Inc’s affiliation with TikTok mum or dad ByteDance, Lemon Inc. utilized for a trademark for “ByteDance” in January.
MBW’s reporting on ByteDance’s and TikTok’s AI-related music actions in the US arrives in opposition to a backdrop of two large ongoing tales involving the corporate and its subsidiary TikTok.
The primary story is a looming TikTok ban in the US, as a result of potential pressured sale of its US operations by way of a invoice handed by the Home of Representatives final week, which is now one massive step nearer to turning into legislation.
The opposite story that we’ve been following intently is TikTok’s falling out with Common Music Group, which refuses to resume its licensing cope with TikTok for a lot of causes, together with the app’s compensation for artists and songwriters, and UMG’s considerations about TikTok’s use of AI.
On March 1, Common Music Publishing’s catalog of ~4 million songs turned unlicensed to be used on TikTok, becoming a member of UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (thus far with out renewal) on February 1.
In a assertion issued to UMPG’s songwriters on February 29, the corporate turned a lot of its consideration to the position AI-generated audio is enjoying on TikTok.
UMPG claimed that, thus far, TikTok has not offered Common with any assurances that the platform received’t prepare its AI fashions on the music firm’s songs.
MBW has been writing for a while now concerning the hypothetical potential for TikTok and different providers to fill their catalogs with music generated by AI, which might, within the course of, dilute the market share of conventional rightsholders.
In February final yr, we revealed an ‘MBW Reacts’ article asking if TikTok might pull off a “heist” on the music business on this regard, following its aggressive funding in generative AI know-how.
The “heist” we have been referring to: Utilizing licensed music as a cornerstone within the rise of TikTok to effectively over a billion customers globally, earlier than utilizing first-party, AI-created songs to crowd out music owned by conventional music rightsholders on the platform.
We wrote: “With music enjoying such a key position in TikTok’s rise, if main label content material does disappear from the platform – and the hole is someway efficiently crammed by indie and AI-driven creations – TikTok may very well be mentioned to have pulled off one of many greatest heists in music enterprise historical past. A bait and change for a billion customers.”Music Enterprise Worldwide
[ad_2]
