Yes, I use it all the time. I watch entire channels through it, with mpv:
mpv <channel name> --ytdl-format='bestvideo[height<=144]+bestaudio' --ytdl-raw-options='continue=,ignore-errors='
If you just want to download the audio from a youtube video, the process is as follows:
First, check the video for available formats:
youtube-dl -F <youtube-video-url>
This will result in a list of available formats:
[...]
format code extension resolution note
171 webm audio only DASH webm audio , audio@ 48k (worst)
140 m4a audio only DASH audio , audio@128k
160 mp4 192p DASH video
133 mp4 240p DASH video
134 mp4 360p DASH video
135 mp4 480p DASH video
17 3gp 176x144
36 3gp 320x240
5 flv 400x240
43 webm 640x360
18 mp4 640x360 (best)
As you can see in that list, there's a decent audio track available (format 140).
Next you just download that one:
youtube-dl -f 140 <youtube-video-url>
and you're done!
Alternatively, you can let youtube-dl figure out the best quality:
youtube-dl -f 'bestaudio' <youtube-video-url>
This might not be an MP4-file, it might be WebM, if the best available audio is Opus.
And in case you need a specific format, you can use things like bestaudio[ext=m4a], etc.
This is no conversion, btw. It's just downloading the audio as it comes from youtube, in its available format. The only thing it does, it just strips away the video track.