Saturday 24 February 2024

My Perfect AWS Console

Yeah that's literally it. I love AWS and use a decent portion of their offerings, but could really honestly get by with 2 of the OG AWS features, and one relative newcomer. ## AWS S3 The performance these days is absolutely top-notch (without even going down the [Directory Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html) route). It's cheap enough that with a well-designed path structure, you can put just-about any workflow that can be represented with JSON into it. As in, you probably don't need [Step Functions](https://aws.amazon.com/step-functions/). ## AWS Lambda I can't remember the last time I've needed a server that hangs around all the time, whether for work or side-gigs. Lambdas just fit _so well_ with modern request-response patterns that it's difficult to justify anything else. Add some [Provisioned Concurrency](https://docs.aws.amazon.com/lambda/latest/operatorguide/provisioned-scaling.html) if you really need nice warm caches and connections, but you still get the super-fast deployment and observability of functions-in-the-cloud. And you're not limited to 30-second execution time any more either (it's currently [up to 15 minutes](https://blog.awsfundamentals.com/lambda-limitations)), so you can wait for those slow 3rd-party APIs. *Protip:* The Lambda Test Console allows you to store (and share!) test JSON payloads for each lambda. This can be a superb way to perform ad-hoc jobs, or re-process things that didn't quite work right the first time. Add a `dryRun?: boolean` option to the input shape and pass it though your lambda code to check things before opening the taps. ## AWS AppSync Sure the web console is a little clunky and bug-ridden (it won't reauthenticate its own IAM session so your queries will eventually just ... die) but if you've got a GraphQL interface deep inside some WAF-protected VPN, this is a great way to give it a poke.

Sunday 28 January 2024

My Apps, 2024

Following a bit of a blogger-trend, here's the stuff I use on the daily. I've omitted things I simply don't use, like custom launchers, podcast listeners, RSS, tracking and/or Mastodon clients; - Mail service: __GMail__ - Tasks: __Drafts in GMail__ - Cloud storage: __Dropbox__ - Web browser: __Chrome__ - Calendar: __Google Calendar__ - Weather: __BOM__ (Melbourne, Australia) app - Video: __Netflix, Disney Plus, Amazon Prime Video__ - Music (Listening): __Spotify__, Spotify via Google Home and/or Chromecast - Music (Creation): __GarageBand__ - Passwords: __1Password__ - Notes: __Drafts in GMail__ - Code: __Visual Studio Code__ - Terminal: __Terminal.app__ - Search: __Google__ This list has shown me how much I depend on Google a) not being evil; and b) not just giving up on a product because they're bored of it ... which concerns me a little. While it still exists though 😉, I *do* highly recommend the use of __Drafts in GMail__ as your general-purpose, cross-platform notes/todo app. You can attach files of arbitrary size, they sync to everything/everywhere *fast* (faster than Dropbox) and it's free (free-r than Dropbox... hmmm 🤔)

Saturday 30 December 2023

2023 End-of-year wrapup

Another year over, with some satisfying accomplishments: - [Frenzy.js](https://frenzyjs.themillhousegroup.com/) is now fully-playable, albeit lacking a few things from the original - I've been [checking out Svelte](https://blog.themillhousegroup.com/2023/08/searching-for-next-spa.html) and enjoying learning that refreshing approach to front-end dev - and in work-related stuff, I've learnt heaps about migrations, ETL, AWS tech like Step Functions, Terraform and TypeScript In 2024, my goals will be: - Finish Frenzy.js completely and [document it](https://blog.themillhousegroup.com/search/label/frenzy.js) - Build [something cool](https://blog.themillhousegroup.com/2023/08/searching-for-next-spa.html) in Svelte and show the world - Harness _The Biggest Buzzword since The Blockchain_ (AI/ML) to do something useful That last one is a biggie but it does seem like an idea whose time has finally come (unlike a certain "let's make money off suckers using a slow database" technology one could mention!)

Sunday 26 November 2023

Mermaid.js is incredibly cool

As mentioned [earlier](https://blog.themillhousegroup.com/2023/10/markdown-and-mermaid-on-blogger-in-2023.html) I'm now able to embed [Mermaid.js](https://mermaid.js.org/) diagrams directly into this blog and I'm going to just post *exclusively* about just how great Mermaid is. If you've __ever__ drawn a technical diagram in __anything__ from MS Paint through [Draw.io](https://www.drawio.com/) and even [Excalidraw](https://excalidraw.com/) and then had to convert the diagram into a PNG or JPEG in order to embed it into a document, knowing full-well that the slightest change is going to involve re-doing a significant chunk of that work, **get Mermaid into your life right now**. ### Mmmm Pie I mean, look at this: ```mermaid pie title Pets adopted by volunteers "Dogs" : 386 "Cats" : 85 "Rats" : 15 ``` That's *four* lines of simple text: ``` pie title Pets adopted by volunteers "Dogs" : 386 "Cats" : 85 "Rats" : 15 ``` ### Flowcharts with flair ```mermaid flowchart TD subgraph First Approach A[Start] --> B{Does it work?} B -->|Yes| C[OK] B --->|No| D[Rethink] end subgraph Second Approach F[Start] --> G{Does it work?} G -->|Yes| H[OK] G -->|No| J[Oh dear] end D --> F ``` Is just: ``` flowchart TD subgraph First Approach A[Start] -- > B{Does it work?} B -->|Yes| C[OK] B --->|No| D[Rethink] end subgraph Second Approach F[Start] --> G{Does it work?} G -->|Yes| H[OK] G -->|No| J[Oh dear] end D --> F ``` I really like how easy it is to declare those `subgraph` elements for visual grouping. And did you notice how a different length "arrow" hints to Mermaid to render a longer arc? Delicious. Expect to see a lot more diagrams like these around here in the future!

Monday 23 October 2023

Markdown (and Mermaid) on Blogger in 2023

Building on the fine work of [cs905s](href="https://github.com/cs905s/md-in-blogger") on GitHub, I wanted to Markdown-enable my own blog and thought the process could be brought up-to-date - seven years have passed after all, and _that_ work was in turn based on something from [2011](http://blog.chukhang.com/2011/09/markdown-in-blogger.html)! Blogger has, slowly but surely, changed since those instructions were written, so at the very least, here's an updated guide on how to do it. ### Extra Goals I had a couple of extra requirements over the previous solution, however. - I want to write **my whole post** in [GitHub-flavoured Markdown](https://github.github.com/gfm/); I use it all day (for PR descriptions, in Slack, writing documentation) and I'm pretty sick of the verbosity of `BOO` compared to `**BOO**`! - I also _never want to have to type `<` ever again_ so I want the script to perform that replacement for me. - I have fallen in love with [Mermaid](https://mermaid.js.org/) for Markdown-inspired/embedded diagrams and want them to _Just Work_ in my blog in the same way GitHub does it, with a ` ```mermaid ... ``` ` code fence For a fairly-techy blog, just those changes, plus being able to use backticks and other speedups, _just like it's GitHub/Slack_ is really valuable to me. Here's an example embedded Mermaid flowchart, just because I can now: ```mermaid flowchart TD A[Have Blogger-hosted Blog] -->|Configure Markdown| B(Write blog post) B --> C{Has markdown-enabled label ?} C -->|Yes| D[Render Markdown to HTML post body] D --> E[Hide original Markdown area] C -->|No| F[Leave post body untouched] F --> G{Has pre with class markdown ?} G -->|Yes| H[Render pre to HTML] E --> I{Has pre with class mermaid ?} H --> I I --> |Yes| J[Render with Mermaid.js] I --> |No| K[Done] ``` So once you've added a `markdown-enabled` **Label** on your post, the entire **blog post body** will be considered the Markdown source. I decided to "opt-in" like this as I've got a couple-of-hundred non-Markdown-annotated blog posts that I didn't really fancy going back and opting-out of. Well, actually I did try to automate this but lost data in the process, so aborted that little yak-shaving side-mission. The script will also remove that particular label from the DOM so nobody will see that "load-bearing label" 😉. The source is [here on GitHub](https://github.com/themillhousegroup/md-in-blogger), and I'll endeavour to keep it working well on the Blogger platform over time. Check the README for the step-by-step instructions if you want to Markdown-enable your own blog!

Saturday 30 September 2023

Frenzy.js - the saga of flood-fill

It may surprise you to know that yes, I am back working on Frenzy.js after a multi-year hiatus. It's a bit surreal working on an "old-skool" (pre-hooks) React app but the end is actually in sight. As of September 2023 I have (by my reckoning) about 80% of the game done;

  • Basic geometry (it's all scaled 2x)
  • Levels with increasing numbers of Leptons with increasing speed
  • Reliable collision detection
  • High-score table that persists to a cookie
  • Mostly-reliable calculation of the area to be filled
  • Accurate emulation of the game state, particularly when the game "pauses"

The big-ticket items I still need to complete are:

  • Implement "chasers" on higher levels
  • Fine-tune the filled-area calculation (it still gets it wrong sometimes)
  • Animated flood-fill
  • Player-start, player-death and Lepton-death animations
  • (unsure) Sound.

It all comes flooding back

To remind you (if you were an 80s Acorn kid) or (far more likely) educate you on what I mean by "flood-fill", here's Frenzy doing its thing in an emulator; I've just completed drawing the long vertical line, and now the game is flood-filling the smaller area:

I wanted to replicate this distinctive style of flood-fill exactly in my browser-based version, and it's been quite the labour of love. My first attempt (that actually worked there were many iterations that did not) was so comically slow that I almost gave up on the whole idea. Then I took a concrete pill and decided that if I couldn't get a multiple-GHz multi-cored MONSTER of a machine to replicate a single-cored 2MHz (optimistically) 8-bit grot-box from the early 1980s, I may as well just give up...

The basic concept for this is:

Given a polygonal area A that needs to be flood-filled;

Determine bottom-rightmost inner point P within A.
The "frontier pixels" is now the array [P]
On each game update "tick":
  Expand each of the "frontier pixels" to the N,S,E and W; but
    Discard an expansion if it hits a boundary of A
    Also discard it if the pixel has already been filled
  The new "frontier pixels" is all the undiscarded pixels 
Stop when the "frontier pixels" array is empty
I got this to be pretty efficient using a bit-field "sparse array" to quickly check for already-filled pixels. In the browser, I could perform the per-tick operations in less than 0.1 milliseconds for any size of A. Not too surprising given the entire game area is only 240x180 pixels, and the maximum possible polygonal area could only ever be half that big: 21,600 pixels.

The problem now became efficiently shifting the big pile'o'filled-pixels from the algorithm onto the HTML5 canvas that is the main gameplay area. I'm using the excellent React Konva library as a nice abstraction over the canvas, but the principal problem is that a canvas doesn't expose per-pixel operations in its API, and nor does Konva. The Konva team has done an admirable job making their code as performant as possible, but my first cut (instantiating a pile of tiny 1x1 Rects on each tick) simply couldn't cope once the number of pixels got significant:

This has led me down a quite-interesting rabbit-hole at the intersection of HTML5 Canvas, React, React-Konva, and general "performance" stuff which is familiar-yet-different. There's an interesting benchmark set up for this, and the results are all over the shop depending on browser and platform. Mobile results are predictably terrible but I'm deliberately not targeting them. This was a game for a "desktop" (before we called them that) and it needs keyboard input. I contemplated some kind of gestural control but it's just not good enough I think, so I'd rather omit it.

What I need to do, is find a way to automagically go from a big dumb pile of individual filled pixels into a suitable collection of optimally-shaped polygons, implemented as Konva Lines.

The baseline

In code, what I first naïvely had was:

type Point = [number, number]

// get the latest flood fill result as array of points
const filledPixels:Array<Point> = toPointArray(sparseMap);

// Simplified a little - we use some extra options
// on Rect for performance...
return filledPixels.map((fp) => 
  <Rect x={fp[0]} 
        y={fp[1]}
        width={1}
        height={1}
        lineCap="square"
        fillColor="red"
  />
);

With the above code, the worst-case render time while filling the worst-case shape (a box 120x180px) was 123ms. Unacceptable. What I want is:

// Konva just wants a flat list of x1,y1,x2,y2,x3,y3
type Poly = Array<number>;

// get the latest flood fill result as array of polys
const polys:Array<number> = toOptimalPolyArray(sparseMap);

// far-fewer, much-larger polygons
return polys.map((poly) => 
  <Line points={poly} 
        lineCap="square"
        fillColor="red"
        closed
  />
);

So how the hell do I write toOptimalPolyArray()?

Optimisation step 1: RLE FTW

My Googling for "pixel-to-polygon" and "pixel vectorisation" failed me, so I just went from first principles and tried a Run-Length-Encoding on each line of the area to be filled. As a first cut, this should dramatically reduce the number of Konva objects required. Here's the worst-case render time while filling the worst-case shape (a box 120x180px): 4.4ms

Optimisation step 2: Boxy, but good

I'd consider this to be a kind of half-vectorisation. Each row of the area is optimally vectorised into a line with a start and end point. The next step would be to iterate over the lines, and simply merge lines that are "stacked" directly on top of each other. Given the nature of the shapes being filled is typically highly rectilinear, this felt like it would "win" quite often. Worst-case render time now became: 1.9ms

Optimisation step 3: Know your enemy

I felt there was still one more optimisation possible, and that is to exploit the fact that the game always picks the bottom-right-hand corner in which to start filling. Thus there is a very heavy bias towards the fill at any instant looking something like this:

----------------
|              |
|              |
|             P|
|            LL|
|           LLL|
|          LLLL|
|         LLLLL|
|        LLLLLL|
|       LLLLLLL|
|      LLLLLLLL|
|     LLLLLLLLL|
|    LLLLLLLLLL|
|   LLLLLLLLLLL|
|  LLLLLLLLLLLL|
| LLLLLLLLLLLLL|
|SSSSSSSSSSSSSS|
|SSSSSSSSSSSSSS|
|SSSSSSSSSSSSSS|
----------------
where
  • P is an unoptimised pixel
  • L is a part line, that can be fairly efficiently represented by my "half-vectorisation", and
  • S is an optimal block from the "stacked vectorisation" approach
You can see there are still a large number of lines (the Ls and the P) bogging down the canvas. They all share a common right-hand edge, and then form a perfect right-triangle. I started implementing this change but ended up aborting that code. Worst-case render time is already significantly below the "tick" rate, and the code was getting pretty complex. Okay, it's not optimal optimal, but it's Good Enough. Whew.

Sunday 27 August 2023

Searching for the next SPA

I've been quite taken with a particular style of casual, "clever" game that rose to prominence during The COVID Years but still has a charm that keeps me visiting almost daily:

  • Wordle (the original, and the "rags to riches" ideal)
  • Quordle (a beautiful initial implementation, albeit reduced now)
  • Heardle (recently escaped from the clutches of Spotify)
and most-recently:

There are a heap of common factors amongst these games (and I'll optimistically include my own Cardle here too) that I think make feel so "nice":

  • Rejection of obvious monetization strategies
  • Feel resolutely mobile-first in UI/UX (large elements, zero scrolling!)
  • Delightful levels of polish (micro-interactions, animation etc)
  • Focused; not just a Single *Page* App, but almost a Single *Pane* App

Of course there's also the little matter of having a great idea with suitably nice mechanics and frequently creativity (Connections I think excels in having creative, challenging content by Wyna Liu that is pitched just *chef's kiss*) but I think there are still areas of the word-association, letter-oriented game landscape to be explored.

For this next one I also will be trying out Svelte after reading the superb "Things you forgot (or never knew) because of React" by Josh Collinsworth which very nicely articulated what feels a little "ick" about React development these days, and paints a very nice picture on what's on the other side of the fence. The scoped-styling and in-built animation abilities in particular seem like a perfect fit for this kind of app.

Now I just need an idea...