RE: MIDI (a.k.a. Daggerfall Remastered Music Project)

Discuss modding questions and implementation details.
User avatar
ByteMixer
Posts: 27
Joined: Sun Mar 25, 2018 4:43 am
Contact:

RE: MIDI (a.k.a. Daggerfall Remastered Music Project)

Post by ByteMixer » Sun Mar 25, 2018 6:49 am

Update:

A little information about this project:

The aim of this remaster is to use the samples and tools I have at my disposal to produce a cohesive and comprehensive remastering of all the midi files available from the Daggerfall Unity GitHub. Some of the midi files are clones/duplicates that are only slightly different due to the different soundcard profiles of the time (MPU-401, SoundCanvas, SoundBlaster, AdLib, etc.) So I'll only be remastering one "branch" of the files; which ever seem to have more data, and/or behave better with my sample libraries. So that the files load into Daggerfall Unity without much issue, they will be rendered to OGG Vorbis format so that they can be moved to the appropriate folder and automatically loaded in by the game.

For the orchestral instrument samples, I'm using a combination of VSCO2-Community-Edition sample libraries, and Virtual-Playing-Orchestra-3. These are loaded using Plogue's Sforzando VSTi software. I also have a smattering of orchestral samples from EastWest (from a Christmas release freebie they did back in 2012)mixed in or layered for certain tracks. Synthetic tones are done using a combination of a VA-synth called Tyrell Nexus 6 by UHE, and samples from the Alchemy Player pack. Reaper 5 is my DAW of choice. For the mixing/balance/"mastering" part, I'm using a combination of Terry West's MHorse P3 processor, Analog87 Equalizer, and LookAhead Limiter. On some of the percussion and timpani tracks, I also use a little bit of compression, either NI Supercharger, or Red3 Compressor. The rest is just adjusting levels, and making sure things sound good between different songs (same pan positions, reverb levels, overall character, gain, etc.)

I do not have a deadline for getting this done, but I anticipate being finished before the Fall. (hopefully!), barring interruptions from rehearsals, practices, and gigging with my band. :)

Phase 1 of this project is complete, and includes all the available tracks from 02 -30, including a couple extras, 5strong and D1 whether they're actually used in the game itself or not.

https://drive.google.com/file/d/11HYJ3p ... f_XYm/view

Download the rar file, and extract with an appropriate utility (winrar, 7zip, etc.).

1: Create a folder called Sound inside the DaggerfallUnity_Data\StreamingAssets folder.

2: copy the .OGG files directly into the Sound folder as loose files.

Also, for sake of immediacy, I uploaded a hefty selection of the music tracks onto my soundcloud for previewing.

https://soundcloud.com/j-bot-1/sets/dag ... ed-music-1

Enjoy!

------------------------------------------------------------------------------------------------------
I didn't want to necro the previous thread about midi, so I thought I'd start a new one, also my topic is a little different.

Also, Hi, I'm new here, freshly arrived after finding out about the Unity port a few days ago. :D

Apologies if this post gets a bit long, but I wanted to cover some basic information about Midi and Midi data, and then go into something I'd like to contribute if it is worthwhile or feasible (though that bit might be better in the Modding forum, and I can copy/paste the idea over there). The development here still appears to be going strong, and I have high hopes, much like I did Dagger XL before it fizzled.

Anyway, on the subject of midi: as some may know, MIDI is an acronym for musical instrument digital interface. Many consider Dave Smith as the father of MIDI (he's also the owner of Dave Smith Instruments, formerly Sequential Circuits, and designer of many notable hardware synthesizers over the years)

Standard midi is literally just data set up as a set of instructions or events to tell an instrument/player/soundcard what to do when reading the file. It's set up as a timeline of events. Note-On, Note-Off, Midi Key number (like A3, C# 3, Eb4, etc.) Aftertouch, Pitch, Modulation, Velocity, Volume, and other "CC" messages that can be uses to control various aspects of a software or hardware instrument. The old standard values range from 0-127 (or 1-128) There are other data values such as Sysex, RPN, and NRPN that also handle different things, but I won't go into detail on those, as I don't really mess with them myself.

Neither here nor there, but a new MIDI standard is currently in the works ("HD Midi") which implements a wider range of values. This would suggest smoother change of values over time, and less "aliasing" when a parameter moves through a range of values.

Moving on, a soundfont is just a catalogue of instrument samples arranged by patch numbers. When that catalogue is available to the player, the instruments are chosen based on which patch is assigned to a track in the midi file being played. Usually a track within a midifile is set to play a certain patch. There can also be a patch change event within that track to swap to a different instrument. This was often done to accommodate instruments where polyphony or memory was limited. The patches are usually determined by GM Standard, a.k.a. General Midi, where patch 1 is a grand piano, 41 is a violin, 57 a trumpet, and so on and so forth. There are other standardized instrument/patch banks as well, like Yamaha's XG banks, or Roland's GS banks.

I'm somewhat glossing over the more granular details, but that's kind of how midi works in a nutshell; a file is loaded, and data is streamed to a music device during playback. Notes and Events are triggered by the data during playback, and the instrument reacts accordingly.

----------------------------------------------------------------------

Now, that said, most digital audio workstation (DAWs) are capable of importing midi files and expanding them so it's 1 instrument per track. Just the data, not the audio that the data "performs." MIDI has no audio attached to it, which is why a player, instrument or soundcard needs an instrument bank, soundfont or sample set to be able to "perform" the midi file. Many home keyboards can play back midi files using a General Midi Bank. Some pro keyboards can do it as well, but they don't always have a general midi bank, and the patch change in the midi file will often just trigger a particular synth or workstation patch on a professional device. It really depends on the device, though, and how the architecture of the instrument is handled.

Now, my memory here is sketchy, but back in the DOS days, when you had your SB16s, AWE32's, Turtle Beach Montegos, and SB Live Platnimum 5.1's, a game would basically have a dll built in that sends a stream out to the soundcard, and the soundcard could be configured to send midi data out to an external instrument via the serial Joystick port (if the soundcard didn't have built-in MIDI 5-pin DINs). This was usually set either in a config file, or within the game itself, depending on the game, or utility or whatever. Or during setup while installing a game with various choices like "Roland GS, SC88, GM, Sound Canvas, etc.)


At least that's what I recall, I'm pretty sure there are some gaps and maybe not everything I went on about is as accurate as I would like. But I hope that helps with some understanding.

------------------------------------------------------------

Now, as for where I'd like to come in, and maybe this section belongs in the Mods forum, I'm not entirely sure, since I'm not a coder. But I guess it's similar to replacing a texture, and easily just as involved. It's something I'd be doing in my free time, (which I have a bit of right now) so I don't have a time table for it, much less an ETA.

What I'd like to do, and I don't know if this is worthwhile or if there is any point to it, but I'd like to take a stab at basically remastering the old Daggerfall midi files, rendering them into high-fidelity mp3/wav audio files (mostly likely mp3 as Wav files are huge). If someone can extract the midi (.mid) files from Daggerfall, (or point me where to go get them if already available somewhere) I can import them into Reaper (my DAW of choice), and basically load high quality VST instruments and samples into the tracks, rebalance the mix, improve the atmosphere and sound stage, and render to 320kbps mp3 or standard 16-bit 44.1KHz WAV files. I also have a few, more professional keyboards that I can route out to, and record audio back into the DAW. e.g. for some of the acoustic samples, I can route midi out to my Roland VR-09, and record its outputs onto an audio track. For synthetic instruments, I can route out to my Nord Lead 2X. etc. etc.

I have a bit of time these days, and I'd be willing to take a crack at it, but I can't really give a time frame to which I'd be done (plus job/moving status in September is totally up in the air right now) I also have commitments to a couple bands I play keys with at the moment, and the odd live recording work that comes my way if and when it happens. But I can work on it when I have time. I can do a few files, as scratch tracks, and put them up on google drive (my dropbox is usually a bit full as I swap files around with band members.....yes I'm a keyboardist in a band).

Granted, depending on the soundfont/patch/style of the Daggerfall midi files, there would be room for artistic interpretation of the track as far as synthetic instruments go, but I do have available decent acoustic samples that I can load into a track, and then adjust the entire mix as far as stereo/acoustic space (reverb, etc.) and get a pretty balanced mix. As well as being able to route to and record from a couple hardware keyboards/synthesizers (Nord Lead 2X, Roland JD-Xi, and a Roland VR-09)

--------------------------------------------------------------

I'm thinking about approaching this in phases:

1st phase, quick and dirty "scratch" tracks; getting VSTis and samples into the tracks, doing some very basic rough mixes, and popping out the mp3/wav files as a first step, to get an idea of how things might sound. Basically, similar to what happens with a soundfont, but I'm just popping the appropriate instruments into the tracks, and quickly getting them output/rendered to see how they sound. Might be a bit "dry" sounding, but probably not any worse than a soundfont. I could make them available pretty quickly for use. And, again, can still route to my keyboards for a more personal touch, adding in and/or layering sounds you won't find in any soundfont.

2nd phase, polishing up the mixes per music track by music track (adding FX and EQ/Compression where needed, adjusting levels, space, depth and width, etc.) basically rebalancing the mixes on a track by track basis. Also, possibly beef up some of the more ambient (dungeon) music by layering a couple different synth instruments together to create more movement and providing a more evocative experience.

3rd/final phase: (kind of the hard part, as I might start second guessing myself here) adjusting the tracks as a whole, so they "fit" together as a complete project, and all have about the same volume levels and the same "character", so that nothing is too loud or too quiet when changing between music tracks, and they all sound like they're part of the same piece of work as a whole. I may not go too overboard with this, and mostly focus on making sure dynamics between music pieces are fairly balanced and stable.

Granted they'll still be midi files, and probably still feel a bit "robotic" but at least the fidelity will be much higher. I can't really say that it'd be on the same level as say the remastered music for Descent XL (I think they basically did complete remixes from scratch of all the old music, and it's bloody amazing) but it will be better than what exists currently, well, barring ones artistic tastes I suppose, haha! But I would do my absolute best to preserve and enhance the moods and feelings of the music.

I'm making an educated guess that the midi files in-game are simply looped before changing areas or regions. On a region/location change, the first track is faded out and the new track starts. So I guess the music isn't really more dynamic than fade-out fade-in from one track to another when changing regions, so I won't really be too concerned with setting loop points. Not sure if the rendered music file would keep the mid-song loop point anyway. I'm guessing the midi files are also all sitting in the MIDI.BSA file so the game can access them on an as needed basis.

I don't know if Unity works with wav/mp3 files as part of the base engine, or if the game could hook into such a player to load/unload the files as needed. But regardless, some code would probably have to be ported and hook into the music player, and modified somewhat even if the new music files have the same filename as the old midi files.

The other issue is that high quality mp3 files can get to be pretty big. e.g. one of my songs is about 6:00 min long with 10 instrument tracks (some midi with samples, some recorded audio and vocals), and the 320kbps mp3 file is roughly 13MB in size. I could export as 256kbps, and that might drop it to around 8MB, but still a mammoth increase over, say a 12KB midi file! A stereo 16-bit 44.1KHz WAV file of that same song would be almost 60MB.

Anyway, hope the information helps, and let me know if basically doing a music remastering would be something worthwhile or not. Maybe Unity will play more nicely with more standard audio formats. (just please, please don't make me learn FMOD or WWISE!!!!!!!!)
Last edited by ByteMixer on Sun Jul 08, 2018 6:33 am, edited 5 times in total.
"Whatever you do, make good art." - Neil Gaiman

User avatar
Interkarma
Posts: 2959
Joined: Sun Mar 22, 2015 1:51 am

Re: RE: MIDI

Post by Interkarma » Sun Mar 25, 2018 7:51 am

Welcome to the forums ByteMixer! :)

You can at this time override the default MIDI songs with OGG format tracks or alternative MIDI versions. See the "Sound & Movies" section of modding guide. At this time it isn't possible to mod the SoundFont used, but this can be possible in future.

http://www.dfworkshop.net/projects/dagg ... y/modding/

Daggerfall Unity uses its own MIDI synth internally (based on C# Synth Project) so any MIDI files compatible with that will be fine, but you'll be stuck with the default SoundFont for now. You'll have the most freedom overriding the MIDI tracks with whatever you like in OGG format.

There are no plans to have remastered tracks in default Daggerfall Unity, as the base install is intended to be very classic-like. But the mod system exists for this purpose. I'll move this discussion to the mod discussion forums so you can discuss with other modders.

User avatar
heilkitty
Posts: 15
Joined: Sun Dec 03, 2017 4:00 pm

Re: RE: MIDI

Post by heilkitty » Sun Mar 25, 2018 3:43 pm

ByteMixer wrote:
Sun Mar 25, 2018 6:49 am
If someone can extract the midi (.mid) files from Daggerfall, (or point me where to go get them if already available somewhere)
https://github.com/Interkarma/daggerfal ... rces/Songs (those files with .mid.bytes extension are, what you're looking for).

User avatar
ByteMixer
Posts: 27
Joined: Sun Mar 25, 2018 4:43 am
Contact:

Re: RE: MIDI

Post by ByteMixer » Sun Mar 25, 2018 4:58 pm

Great! I'll look into it when I have a chance. (have a cover jam session this afternoon, and rehearsal with the band I'm in on Tuesday nights).

And since Interkarma mentioned OGG format already being available, once I'm done with the tracks, I can render to .ogg in reaper.

I see that since I initially posted, someone has already taken initiative on this in converting the music to OGG format using his/her own soundfonts/samples, and that's fine. My goal will be a bit more in depth, though. :)
"Whatever you do, make good art." - Neil Gaiman

Ravenborn
Posts: 30
Joined: Fri Jul 21, 2017 2:25 pm

Re: RE: MIDI

Post by Ravenborn » Sun Mar 25, 2018 9:43 pm

Wow! I didn't even see this until I had already rendered quite a few midis. LOL I should browse other sections more often.
If you feel you can do better (which I don't doubt, I have zero musical experience other than playing around with soundfonts and you seem competent and experienced) definitely grab the bull by the horns and run with it! :D It'll help me narrow my focus so I'm not all over the place, which I have a habit of doing with every project I get involved in.

User avatar
ByteMixer
Posts: 27
Joined: Sun Mar 25, 2018 4:43 am
Contact:

Re: RE: MIDI

Post by ByteMixer » Mon Mar 26, 2018 12:07 am

No worries man, always good to have choices. And honestly what I'm doing will only be a little more in depth. It's still the same overall concept: swapping out tracks with higher quality samples. I'm just taking things a step further with respect to the mix balance and overall cohesion. I might embellish a little bit more, and add a little more polish than doing a straight up conversion, but some of it will be up to artistic interpretation, and some people might prefer things sounding a little closer to the classic midi files. Also, this might take me awhile, but I'll upload examples as I make progress.

But yeah, we'll see how it all turns out!
"Whatever you do, make good art." - Neil Gaiman

User avatar
ByteMixer
Posts: 27
Joined: Sun Mar 25, 2018 4:43 am
Contact:

Re: RE: MIDI

Post by ByteMixer » Mon Mar 26, 2018 4:47 am

Okay, spent a few hours on a first track tonight to see how it would go. I took some minor liberties, but otherwise, should be pretty close sounding to the classic sounds.

This is a test track up on my soundcloud, so you all can kind of see where I plan on going with this project.

I'll probably go back and polish this track up at some point, and possibly route the midi to my external keyboards and pull in some audio recordings of the midi "performance"

This is rendered as OGG Vorbis at 320kbps.

I'll allow downloads of the track once I complete a section. I'm thinking I'll tackle this by category; dungeon, outdoors, shops, towns, etc. But I'll keep the file names the same for easy swapping in.

Here's the soundcloud link:

https://soundcloud.com/j-bot-1/daggerfa ... track-02fm
"Whatever you do, make good art." - Neil Gaiman

User avatar
King of Worms
Posts: 550
Joined: Mon Oct 17, 2016 11:18 pm
Location: Scourg Barrow
Contact:

Re: RE: MIDI

Post by King of Worms » Mon Mar 26, 2018 2:24 pm

Beautiful, I was hoping some1 will do thing like this in a serious manner. There were attempts on utube, some very good, there were even some projects last year here on forum... and this year as well.. but noone really approached it in a way it deserves. I have my hopes in this project, seems like its in a good hands now!

Im far from home and only with my phone device... so im limited here... but i know there are also unused traxks i dagger, and they are really freaking good!

You can find em in my thread... called something like a texture catalogue.. 2nd page i guess... would be bombif u can remade em as well

Good luck, have fun and thank you!!!
Pardon the phone grammar

User avatar
ByteMixer
Posts: 27
Joined: Sun Mar 25, 2018 4:43 am
Contact:

Re: RE: MIDI

Post by ByteMixer » Mon Mar 26, 2018 6:25 pm

I do plan to go through the whole list that's available on github. Excluding duplicates, of course. I think some of the naming conventions were to differentiate in the game settings which files were to be used for FM, and which for Roland GS or Sound Canvas protocols. It seems some of the files had more detailed music depending on which the game was told to use. I'll have to double check that.

I'll work on whichever are the more detailed versions.

FWIW, someone has already done a HQ remake of the music from the old Dagger XL forums. Pieces of 8-bit from the Dagger XL forum. You might have seen it on youtube. https://www.youtube.com/watch?v=RANo9sT ... 0&index=31

Unfortunately, I don't know if he ever got around to making them available for download.

His work is really quite good, and what I'm doing is probably going to be along the same lines, but with some minor artistic liberties to embellish. Hopefully I'll achieve a similar quality between all the tracks. (There are something like 75 different midi tracks) It's going to take me time to cover everything, but I plan to be in it for the long haul.

A neat thing I read: apparently there were two different composers from the game. The known composer, Eric Heberling, handled the midi tracks that actually have names (Dungeon, Sunny Day, etc.) While the other guy (name unknown) worked for a company Bethesda owned, and he handled the numbered, nameless tracks 02, 11, 25, etc.
"Whatever you do, make good art." - Neil Gaiman

User avatar
ByteMixer
Posts: 27
Joined: Sun Mar 25, 2018 4:43 am
Contact:

Re: RE: MIDI

Post by ByteMixer » Tue Mar 27, 2018 4:28 am

Okay, unfortunately, it seems one of my plug-ins, VSCO2 crashes reaper on loading a project. Well, phooey!

No worries though, it's just a wrapper for easy loading of SFZ samples, though those versions themselves were tweaked by BigCat. Anyway, I'm using Sforzando plug-in now, and progress is back on.

I updated 02fm, and re-uploaded it onto soundcloud for critiques as well as 5Strong. I'm pretty happy with how they're turning out overall. At least these two tracks seem to both be comparable in level/volume, so hopefully I can keep that up so that none of the tracks have a sudden increase or decrease in levels.

For the record, depending on which versions of instruments I feel fit better, I'm pretty much using a subset of EWQLSO (East West samples) that was released for free one Christmas back in 2008. And the SFZ samples are by Versilian Studios. Depending on the feeling of the track, I might use one or the other on its own, or layered. I tend to layer string sections because I like how the sound fills out, and it sometimes helps to emulate actual orchestras where you've got Vins 1, Vins 2, Violi, Celli, and Contrabass (not that I overlap all at once, I try to break things into sections based on the note range). In other cases, I (e.g. a choir instrument) I'll pull from Sonatina Orchestra samples. I also have a couple UIowa samples that are decent if I want to fill out a section a little better, or try to split a track into divisi.

Thankfully, in the midi files often some the sections are already broken up, and I can fill in the blanks pretty easily. :)

https://soundcloud.com/j-bot-1/daggerfa ... q-remaster
https://soundcloud.com/j-bot-1/daggerfa ... q-remaster

I'll set up a Google Drive or something for allowing downloads once I make some tangible progress and complete one of the category phases. If I can do a track a day, I'm feeling pretty good, but there will be slowdowns on days I have rehearsals or gigs (gigs will probably start up in April). So I probably won't get much done tomorrow.

But for now, enjoy! I'm leaving the other version of 02fm up for the moment for comparisons sake. I can always go back and make small adjustments. I may pull the french horn down a bit in 5Strong, since it's a little forward, for example.

Enjoy for now!
"Whatever you do, make good art." - Neil Gaiman

Post Reply