External Downloader

[arg0nx]

Hello,

Any possibility to add option for external downloader or add some arguments?
eg. "--external-downloader aria2c" and "--http-chunk-size 10M"

On some files I get very slow downloads (same on ydls and youtube-dl directly) but when using external downloader or adding chunk size it goes well with youtube-dl

TY

[wader]

Hi, interesting, do you have an exempel URL i can test with? Not sure how youtube-dl work with streaming to stdout like ydls do and external downloader

[arg0nx]

Direct download (YDLS downloads same speed as this)

youtube-dl` https://www.dailymotion.com/video/x8207y9
[dailymotion] Downloading Access Token
[dailymotion] x8207y9: Downloading media JSON metadata
[dailymotion] x8207y9: Downloading metadata JSON
[dailymotion] x8207y9: Downloading m3u8 information
[download] Destination: Virtual and In-person Events to Celebrate Juneteenth Around the U.S.-x8207y9.mp4
[download] 100% of 11.15MiB in 00:06

Adding chunk:

youtube-dl --http-chunk-size 10M https://www.dailymotion.com/video/x8207y9
[dailymotion] Downloading Access Token
[dailymotion] x8207y9: Downloading media JSON metadata
[dailymotion] x8207y9: Downloading metadata JSON
[dailymotion] x8207y9: Downloading m3u8 information
[download] Resuming download at byte 8387584
[download] Destination: Virtual and In-person Events to Celebrate Juneteenth Around the U.S.-x8207y9.mp4
[download] 100% of 11.15MiB in 00:00

[wader]

Thanks, will have a look in a few days, on vacation. If you want to experiment, what would be needed for ydls is for youtube-dl to somehow use external downloader while writing to stdout, which kind i guess might make it a bit harder to do parellell downloads

[arg0nx]

External downloader is unneded if You can manage to make it possible to use option CHUNK SIZE

That is internal YoutubeDL command

Ty

[wader]

Hi again, i tried to reproduce without external downloader and only use chunk size but i get about the same speed for dailymotion. But with external downloader aria2c i get much faster speed. Unfortunately it looks like aria2c does not support streaming to stdout which would be required they way ydls works currently, to respond fast with http body and store as little as possible on disk.

There is an open issue for aria2c to support stdout aria2/aria2#190 also found this won't fix issue for youtube-dl about streaming and aria2c ytdl-org/youtube-dl#25345

Could the reason you saw faster download with --http-chunk-size 10M be the log line [download] Resuming download at byte 8387584? you can retry and make sure to delete the .part file to not resume.

There shouldn't in theory be any problems supporting streaming and parallell downloads just have to use a bit more disk or memory to buffer to get ordering right. I had a quick look at the other supported external downloaders but couldn't find any that seem to support it.

[wader]

Hi, want to investigate or work on this more or should i close?

[arg0nx]

Hi,
Maybe if possible to add chunk size?

This is test from 2 separate videos, so it is not resuming download

TY

youtube-dl https://www.dailymotion.com/video/x7y1b78
[dailymotion] Downloading Access Token
[dailymotion] x7y1b78: Downloading media JSON metadata
[dailymotion] x7y1b78: Downloading metadata JSON
[dailymotion] x7y1b78: Downloading m3u8 information
[download] Destination: Basic Instinct - Scene from the film-x7y1b78.mp4
[download]  47.3% of 26.37MiB at 169.92KiB/s ETA 01:23

youtube-dl --http-chunk-size 10M https://www.dailymotion.com/video/x4gmtws
[dailymotion] Downloading Access Token
[dailymotion] x4gmtws: Downloading media JSON metadata
[dailymotion] x4gmtws: Downloading metadata JSON
[dailymotion] x4gmtws: Downloading m3u8 information
[download] Destination: Bollywood's The Most Shocking Wardrobe Malfunction _ View Pic's-x4gmtws.mp4
[download] 100% of 20.50MiB in 00:04

[wader]

For me i see this:

https://www.dailymotion.com/video/x7y1b78 (downloads a 27651229 byte file, media bitrate 463 kb/s)
no chunk size: 2MB/s at first then slows down
10M chunk size: 2MB/s at first then slows down

https://www.dailymotion.com/video/x4gmtws (downloads a 21496392 byte file, media bitrate 2138 kb/s)
no chunk size: 2MB/s at first then slows down
10M chunk size: 2MB/s until done

I wonder if they do some throttling based on media bitrate etc?

[arg0nx]

Probably some throttling with token because download from YT is getting full speed.
Also I get few other sites with same issue where chunk helps a bit

[wader]

Do you think this would be a global option or a per request option? Also thinking this should probably be off by default? Have some fear sending range requests might affect some sites. And ideas?

[arg0nx]

Better peer request, there is many sites this is unneeded I believe with multiple request it might get to much memory consuption or something else can go bad...

http://ydls/?format=mp4&chunk=10&url=http.....

Default disabled

[wader]

Initial version #208 requires wader/goutubedl#89 you can use go module replace directive if you want to try it out.

To test:

curl -OLJ 'http://0:8080?url=https://www.dailymotion.com/video/x4gmtws&format=mp3&httpchunksize=10M'

I noticed that --http-chunk-size options is a bit misleading what it does https://github.com/ytdl-org/youtube-dl/blob/5208ae92fc3e2916cdccae45c6b9a516be3d5796/youtube_dl/downloader/http.py#L87 it also adds some randomness to the chunk size, which make a lot of sense for what the purpose is.

Also been thinking if ydls should have some kind of known sites config? maybe hard to maintain?

[arg0nx]

Great work mate,

I don't know how much other people would use site config, I personaly don't need it.
Maybe just some other usefull Youtube DL command to implement in the magic way YDLS works (post)

OR
Making few own Youtube DL config with all commands I need from Youtube DL and pass file to YDLS
For example:

Config1=
Filetype=mp4
Chunk size 10
--user-agent UA
--referer URL

Config2=
Filetype=mp3

Then with YDLS
Http://ydls:8080/config1/http://youtube.com
Http://ydls:8080/config2/http://youtube.com

[wader]

Was thinking that ydls would ship (in ydls.json etc) with known-good site configurations, ex that dailymotion works and is probably faster with chunk size.

Could be something like this in ydls.json:

...
"URLConfig": {
  "https?://*.\.dailymotion\.com/.*" {
     "httpchunksize": 10M"
  }
}
...

(URL regex might be messy, could possibly look at what extractor youtube-dl choose also)

How would you like to use user-agent and referer? use own hardcoded values for some reason?

[arg0nx]

Actually, that would be good with json, great idea...

User agend and referrer are just example commands, but for instance some videos need login with commands "-u username -p password"

So to add to json

...
"URLConfig": {
  "https?://*.\.dailymotion\.com/.*" {
     "httpchunksize": 10M"
     "Login": username"
     "Password": password"
     "UserAgent": Mozilla 5.0...."
  }
}
...