本文是对于如何应用cuda和Stable-Diffusion生成视频的残缺指南，将应用cuda来减速视频生成，并且能够应用Kaggle的TESLA GPU来收费执行咱们的模型。
#install the diffuser package #pip install --upgrade pip !pipinstall--upgradediffuserstransformersscipy #load the model from stable-diffusion model card importtorch fromdiffusersimportStableDiffusionPipeline fromhuggingface_hubimportnotebook_login模型加载
模型的权重是是在CreateML OpenRail-M许可下公布的。这是一个凋谢的许可证，不要求对生成的输入有任何权力，并禁止咱们成心生产非法或无害的内容。如果你对这个许可有疑难，能够看这里
https://huggingface.co/CompVi...
咱们首先要成为huggingface Hub的注册用户，并应用拜访令牌能力使代码工作。咱们应用是notebook，所以须要应用notebook_login()来进行登录的工作
执行完代码上面的单元格将显示一个登录界面，须要粘贴拜访令牌。
ifnot (Path.home()/'.huggingface'/'token').exists(): notebook_login()而后就是加载模型
model_id="CompVis/stable-diffusion-v1-4" device="cuda" pipe=StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16) pipe=pipe.to(device)显示依据文本生成图像
%%time #Provide the Keywords prompts= [ "a couple holding hands with plants growing out of their heads, growth of a couple, rainy day, atmospheric, bokeh matte masterpiece by artgerm by wlop by alphonse muhca ", "detailed portrait beautiful Neon Operator Girl, cyberpunk futuristic neon, reflective puffy coat, decorated with traditional Japanese ornaments by Ismail inceoglu dragan bibin hans thoma greg rutkowski Alexandros Pyromallis Nekro Rene Maritte Illustrated, Perfect face, fine details, realistic shaded, fine-face, pretty face", "symmetry!! portrait of minotaur, sci - fi, glowing lights!! intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha, 8 k ", "Human, Simon Stalenhag in forest clearing style, trends on artstation, artstation HD, artstation, unreal engine, 4k, 8k", "portrait of a young ruggedly handsome but joyful pirate, male, masculine, upper body, red hair, long hair, d & d, fantasy, roguish smirk, intricate, elegant, highly detailed, digital painting, artstation, concept art, matte, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha ", "Symmetry!! portrait of a sith lord, warrior in sci-fi armour, tech wear, muscular!! sci-fi, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha", "highly detailed portrait of a cat knight wearing heavy armor, stephen bliss, unreal engine, greg rutkowski, loish, rhads, beeple, makoto shinkai and lois van baarle, ilya kuvshinov, rossdraws, tom bagshaw, tom whalen, alphonse mucha, global illumination, god rays, detailed and intricate environment ", "black and white portrait photo, the most beautiful girl in the world, earth, year 2447, cdx" ]显示
%%time #show the results images=pipe(prompts).images images #show a single result images[0]第一个文本：a couple holding hands with plants growing out of their heads, growth of a couple, rainy day, atmospheric, bokeh matte masterpiece 的图像如下
将生成的图像显示在一起
#show the results in grid fromPILimportImage defimage_grid(imgs, rows, cols): w,h=imgs[0].size grid=Image.new('RGB', size=(cols*w, rows*h)) fori, imginenumerate(imgs): grid.paste(img, box=(i%cols*w, i//cols*h)) returngrid grid=image_grid(images, rows=2, cols=4) grid #Save the results grid.save("result_images.png")如果你的GPU内存无限（可用的GPU RAM小于4GB），请确保以float16精度加载StableDiffusionPipeline，而不是如上所述的默认float32精度。这能够通过通知扩散器冀望权重为float16精度来实现:
%%time importtorch pipe=StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16) pipe=pipe.to(device) pipe.enable_attention_slicing() images2=pipe(prompts) images2[0] grid2=image_grid(images, rows=2, cols=4) grid2如果要更换噪声调度器，也须要将它传递给from_pretrained:
%%time fromdiffusersimportStableDiffusionPipeline, EulerDiscreteScheduler model_id="CompVis/stable-diffusion-v1-4" # Use the Euler scheduler here instead scheduler=EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler") pipe=StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, torch_dtype=torch.float16) pipe=pipe.to("cuda") images3=pipe(prompts) images3[0][0] #save the final output grid3.save("results_stable_diffusionv1.4.png")看看这图就是更换不同调度器的后果
#results are saved in tuple images3[0][0] grid3=image_grid(images3[0], rows=2, cols=4) grid3 #save the final output grid3.save("results_stable_diffusionv1.4.png")查看全副图片
创立视频。
根本的操作曾经实现了，当初咱们来应用Kaggle生成视频
首先进入notebook设置:在加速器抉择GPU，
而后装置所需的软件包
pipinstall-Ustable_diffusion_videos fromhuggingface_hubimportnotebook_login notebook_login() #Making Videos fromstable_diffusion_videosimportStableDiffusionWalkPipeline importtorch #"CompVis/stable-diffusion-v1-4" for 1.4 pipeline=StableDiffusionWalkPipeline.from_pretrained( "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, revision="fp16", ).to("cuda") #Generate the video Prompts 1 video_path=pipeline.walk( prompts=['environment living room interior, mid century modern, indoor garden with fountain, retro,m vintage, designer furniture made of wood and plastic, concrete table, wood walls, indoor potted tree, large window, outdoor forest landscape, beautiful sunset, cinematic, concept art, sunstainable architecture, octane render, utopia, ethereal, cinematic light, –ar 16:9 –stylize 45000', 'environment living room interior, mid century modern, indoor garden with fountain, retro,m vintage, designer furniture made of wood and plastic, concrete table, wood walls, indoor potted tree, large window, outdoor forest landscape, beautiful sunset, cinematic, concept art, sunstainable architecture, octane render, utopia, ethereal, cinematic light, –ar 16:9 –stylize 45000', 'environment living room interior, mid century modern, indoor garden with fountain, retro,m vintage, designer furniture made of wood and plastic, concrete table, wood walls, indoor potted tree, large window, outdoor forest landscape, beautiful sunset, cinematic, concept art, sunstainable architecture, octane render, utopia, ethereal, cinematic light, –ar 16:9 –stylize 45000', 'environment living room interior, mid century modern, indoor garden with fountain, retro,m vintage, designer furniture made of wood and plastic, concrete table, wood walls, indoor potted tree, large window, outdoor forest landscape, beautiful sunset, cinematic, concept art, sunstainable architecture, octane render, utopia, ethereal, cinematic light, –ar 16:9 –stylize 45000', 'environment living room interior, mid century modern, indoor garden with fountain, retro,m vintage, designer furniture made of wood and plastic, concrete table, wood walls, indoor potted tree, large window, outdoor forest landscape, beautiful sunset, cinematic, concept art, sunstainable architecture, octane render, utopia, ethereal, cinematic light, –ar 16:9 –stylize 45000'], seeds=[42,333,444,555], num_interpolation_steps=50, #height=1280, # use multiples of 64 if > 512. Multiples of 8 if < 512. #width=720, # use multiples of 64 if > 512. Multiples of 8 if < 512. output_dir='dreams', # Where images/videos will be saved name='imagine', # Subdirectory of output_dir where images/videos will be saved guidance_scale=8.5, # Higher adheres to prompt more, lower lets model take the wheel num_inference_steps=50, # Number of diffusion steps per image generated. 50 is good default ) 将图像扩充到4k，这样能够生成视频
fromstable_diffusion_videosimportRealESRGANModel model=RealESRGANModel.from_pretrained('nateraw/real-esrgan') model.upsample_imagefolder('/kaggle/working/dreams/imagine/imagine_000000/', '/kaggle/working/dreams/imagine4K_00')为视频增加音乐
为视频减少音乐能够通过提供音频文件的将音频增加到视频中。
%%capture !pipinstallyoutube-dl !youtube-dl-fbestaudio--extract-audio--audio-formatmp3--audio-quality0-o"music/thoughts.%(ext)s"https://soundcloud.com/nateraw/thoughts fromIPython.displayimportAudio Audio(filename='music/thoughts.mp3')这里咱们应用youtube-dl下载音频（须要留神该音频的版权），而后将音频退出到视频中
# Seconds in the song. audio_offsets= [7, 9] fps=8 # Convert seconds to frames num_interpolation_steps= [(b-a) *fpsfora, binzip(audio_offsets, audio_offsets[1:])] video_path=pipeline.walk( prompts=['blueberry spaghetti', 'strawberry spaghetti'], seeds=[42, 1337], num_interpolation_steps=num_interpolation_steps, height=512, # use multiples of 64 width=512, # use multiples of 64 audio_filepath='music/thoughts.mp3', # Use your own file audio_start_sec=audio_offsets[0], # Start second of the provided audio fps=fps, # important to set yourself based on the num_interpolation_steps you defined batch_size=4, # increase until you go out of memory. output_dir='dreams', # Where images will be saved name=None, # Subdir of output dir. will be timestamp by default )本文代码你能够在这里找到：
https://avoid.overfit.cn/post/781a2bd8a4534f7cb2d223c141d37df8
作者：Bob Rupak Roy