Putting MP4s in Scratch and how it behaves

Disclaimer: While it is possible to upload MP4 files to Scratch, I would not advise you to do so yourself. I believe that what I have done here is legally okay, because there is no part of any legal instruments I found at Scratch that limits the user to upload files of a certain type. It's just I don't want people to abuse this unintended feature for things the ST may not like. I do this purely because no one, not even the servers stopped me from doing this, and I think it is quite interesting. But please, don't abuse them. Call me a hypocrite or something.

One thing I noticed when I use those YouTube downloader sites (not that you should use them) are the files they give. [small]Yes, I know the thing that powers essentially all of them.[/small] Even though the files are in the .mp3 extension, the contents are always a MP4 container with only a single audio stream. And yet they seem to play back fine in Scratch. What's up with that?

Initially, I thought the files are re-encoded into an MP3 file in the editor. To see it in action, I uploaded such file to Scratch from the editor. It imported fine, and I can save the project and upload the file to the Assets since the file is just around 8 MBs.

However, when I went to check the assets link with FFprobe, it turns out that the file is still encoded in MP4, and still on AAC (a successor of MP3)

clicky@asusArch /m/other > ffprobe https://assets.scratch.mit.edu/f568e57649758bf9504ca58f29...
ffprobe version n6.0 Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 13.2.1 (GCC) 20230801
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://assets.scratch.mit.edu/f568e57649758bf9504ca58f29...':
  Metadata:
    major_brand    : dash
    minor_version  : 0
    compatible_brands: iso6mp41
    creation_time  : 2019-08-17T22:30:30.000000Z
  Duration: 00:07:59.96, start: 0.000000, bitrate: 129 kb/s
  Stream #0:0[0x1](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 2 kb/s (default)
    Metadata:
      creation_time  : 2019-08-17T22:30:30.000000Z
      handler_name    : ISO Media file produced by Google Inc.
      vendor_id      : [0][0][0][0]

Now, the idea of putting something that should be incompatible into Assets is not something new. Everyone who has tinkered around with "that extension" or some bookmarklet people have coded before might have known (and likely abused) this. But the fact that Scratch itself just allows these files without converting or even checking them is just astonishing. Even the built-in audio editor can edit these files without any sort of hiccups.

Armed with this knowledge, I began to wonder...

Does Scratch support Opus?

Yes.

With my FFmpegの術, I made an audio file, in a MP4 container, encoded in Opus, and uploaded it to Scratch. It works, and we can even edit it if we want to.

This is a huge thing. Opus is so much better at compressing audio than MP3, full stop. Imagine a single, good-sounding 30 minute audio file in a Scratch project. Piece of cake for Opus, but virtually impossible using MP3. You don't have to take my word for it, just listen and compare it yourself here.

Not that you would want to upload a half-hour audio file to Scratch.

How does Scratch handle files with multiple audio streams?

Scratch only picks the first audio stream. AFAIK, there's no way to switch to a different audio stream, simply because Scratch isn't expecting an audio file to have multiple audio streams.

However, the streams are still accessible, if you download it from the Assets link, and you have a player that supports them.

Does Scratch support Matroska (and its relative WebM)?

Unfortunately, no. The audio editor just blanks out and you can't play them.However, the file itself is still intact, if you download it from the Assets link. It's just Scratch that can't read the file.

Weirdly, WebM files seems to work perfectly. Perhaps that's where the Web name come from. Or probably the supported containers are ones supported on your browser.

What happens if you export the files back?

This is what I found to be the most interesting. At least in my copy of Firefox, the file will be saved as an .m4a file, as if Firefox knew this is a MPEG-4 file (which it does). Not sure how other browsers reacts with them, but I'm pretty sure it will do more or less the same thing. This also happens with WebM files.

How does Scratch handle video streams?

The video stream gets ignored. It seems like the only thing Scratch cares about is that first audio stream.

The same thing also apply to subtitles.

How does Scratch handle files with no audio streams?

Unsurprisingly, it doesn't work. The same symptoms from trying to upload Matroska file applies here.

* * *

Seeing the current trend of Scratch fixing bugs people perceive as a feature, I can only beg the question: would the ST fix this unintended behavior, and if so, how?

Full Assets cleanup is out of the question. They would have to inspect millions of assets uploaded to Scratch, convert them to MP3s without losing any quality if they fancy it, and somehow keeping them at least 10 MB (or the earlier 50 MB). It's not impossible, but it's impracticable.

Inspecting the uploaded files in the backend is much more sensible. So sensible in fact I'm surprised they haven't implemented it yet. Maybe it's just laziness or they haven't thought of it, maybe the struggle for the server to scan every uploaded assets or the team implementing it is not worth it, or maybe they ignore it since it's merely an edge case that only a minority would use. What they didn't realize is that they let a door open for geeks like me and others, to put any sort of media, in whatever codecs we might use, to our heart's content.

Maybe someday the ST will close this door for good (or evil). Probably after I posted this topic they will say "oh yeah, we haven't thought of that" and patch the servers to thwart everyone trying to upload a video on the Assets. Maybe this topic will end in the dustbin for being potentially dangerous or something. I don't know. But I hope you know that it's possible.

(Now, should I post this on Scratch…?)