In the meantime I played around with waifu2x scaling and figured out that you can get decent results for many VGA mode game screenshots if you blur the image a bit before processing. Here's the results I got for
Arena using official screenshots from the preview:
To get these results I followed these steps:
- open each image in GIMP, convert to RGB, apply Gaussian blur at 0.7x0.7 pixel radius, export the result
- scale up 4x the blurred image with waifu2x
- open the scaled image in mtPaint, apply Kuwahara-Nagao blur at 2 pixel radius with Protect details enabled, Gamma correction disabled
- resize the scaled image in mtPaint to 640x480 using sharpened bicubic interpolation without gamma correction
- load the palette of the original indexed image in mtPaint, convert the 640x480 image to indexed mode with this palette and no dither
NOTE: The goblin sprite in the first image has come out so well because it apparently uses the original sprite dimensions and is not scaled up or down depending on the distance. However this means that theoretically waifu2x can be used to scale up monster sprites as well as other art assets for the game.