Next.js Server Actions - how do I send complex data

[Next.js](https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions-and-mutations#forms) has some extraordinary integration between front- and back-ends. It takes some thinking about, but the thing that stumped me the longest was getting relatively complex objects (i.e. not just simple strings) across the "Server Actions" boundary. ## The Rules There are lots of [rules](https://react.dev/reference/rsc/use-server#serializable-parameters-and-return-values) about __what__ you can send. Basically, it has to be serializable, which makes sense. But *how* do you send these things? Unfortunately, most of the documentation limits itself to [trivially-simple](https://react.dev/reference/rsc/use-server#server-actions-in-forms) examples; like [a single simple string](https://react.dev/reference/react-dom/components/form#handle-form-submission-with-a-server-action) My specific use case was trying to get an object that contained an array of very simple objects across the boundary: ``` type DTO = { id: number; foo: string; people: Person[]; } type Person = { name: string; age: number; } const dataToSend: DTO = { id: 123, foo: "bar", people: [ { name: "Tom", age: 12, }, { name: "Dick", age: 45, }, { name: "Harry", age: 78, }, ] ]; ``` While the order of the objects within the array was unimportant, it was crucial that the elements of the object could be rehydrated (for want of a better expression) correctly on the server side - i.e. with the `Harry`-is-`78` connection intact, guaranteed. ## On the client (version 1) This should not be too hard, I thought to myself. I mean, sending form contents was pretty-much the original "interactive internet" mechanism; `/cgi-bin` and all that old stuff. I found a useful page that explained a [decent mechanism](https://mattstauffer.com/blog/a-little-trick-for-grouping-fields-in-an-html-form/) to serialize arrays, which I implemented like this in my React form: ``` {dto.people.map((p, i) => ( <> ))} ``` I went down that rabbit hole for far too long, before realising ... I *can* just send a string - a **JSON** string! ## Client, version 2 This new version feels much more React-idiomatic, and does a lot less grubbing around in `form` minutae. It uses the `useFormState` hook as well as a `useState` to allow the client to snappily update while the user is doing data-entry. Then when it's submission-time, everything gets shipped off in the everything-old-is-new-again Server Actions kinda way (various validation and button-enablement niceties removed for simplicity): ``` "use client" const initialState = { message: '' } export const SelectionForm = ({ dto: DTO}) => { const [state, formAction] = useFormState(createOrUpdateServerAction, initialState); const [inFlightPeople, setInFlightPeople] = useState(dto.people); const onPeopleChanged = (newPeople: Person[]) => { setInFlightPeople(newPeople); } return (

// render the people, wire in the onPeopleChanged handler

Stepping up, and back, with the new Next.js "app" directory

I'm toying around with a new web-based side project and I thought it was time to give the latest Next.js version a spin. Although I've used Create-React-App (generally hosted on Netlify) more recently, I've dabbled with Next.js in one capacity or another since 2018, and this time some server-side requirements made it a better choice.

The killer feature of the 2023 "beta version" of Next.js (which I assume will eventually be named Next.js 14) is the app directory, which takes Next's already-excellent filesystem-based routing (i.e. if you create a file called bar.tsx in a directory called foo, you'll find it served up at /foo/bar without writing a line of code) and amps it up. A lot.

I won't try and reiterate their excellent documentation, but their nested layouts feature is looking like an absolute winner from where I'm sitting, and I'd like to explain why by taking you back in time. I've done this before when talking about React-related stuff when I joked that the HTML <img> tag was like a proto-React component. And I still stand by that comparison; I think this instant familiarity is one of the fundamental reasons why React has "won" the webapp developer mindshare battle.

Let me take you back to 1998. The Web is pretty raw, pretty wild, and mostly static pages. My Dad's website is absolutely no exception. I've meticulously hand-coded it in vi as a series of stand-alone HTML pages which get FTP'ed into position on his ISP's web server. Although I'm dimly aware of CSS, it's mainly still used for small hacks like removing the underlines from links (I still remember being shown the way to do this with an inline style tag and thinking it would never take off) - and I'm certainly not writing a separate .css file to be included by every HTML file. As a result, everything is styled "inline" so-to-speak, but not even in the CSS way; just mountains of widths and heights and font faces all over the place. It sucked, but HTML was getting better all the time so we just put up with it, used all that it offered, and automated what we could. Which was exactly what I did. If you dare to inspect the source of the above Wayback Machine page, you'll see that it uses HTML frames (ugh), which was a primitive way of maintaining a certain amount of UI consistency while navigating around the site.

The other thing I did to improve UI consistency, was a primitive form of templating. Probably more akin to concatenation, but I definitely had a header.htm which was crammed together with (for-example) order-body.htm to end up with order.htm using a DOS batch file that I ran to "pre-process" everything prior to doing an FTP upload - a monthly occurrence as my Dad liked to keep his "new arrivals" page genuinely fresh. Now header.htm definitely wasn't valid HTML as it would have had unclosed tags galore, but it was re-used for several pages that needed to look the same, and that made me feel efficient.

And this brings me to Next.js and the nesting layouts functionality I mentioned before. To achieve what took me a pile of HTML frames, some malformed HTML documents and a hacky batch file, all I have to do is add a layout.tsx and put all the pages that should use that UI alongside it. I can add a layout.tsx in any subdirectory and it will apply from there "down". Consistency via convention over configuration, while still nodding to the hierarchical filesystem structures we've been using since Before The Web. It's just really well thought-out, and a telling example of how much thought is going into Next.js right now. I am on board, and will be digging deeper this year for sure.

Introducing ... Cardle!

UPDATE [July 2023] - I've let the cardle.xyz domain expire after a year, but you can still play the game over at cardle.themillhousegroup.com

Yes, it's yet-another Wordle clone, this time about cars:

https://www.cardle.xyz

Like so many other fans of Wordle, I'd been wanting to try doing a nice self-contained client-side game like this, and after trying the Australian Rules player version of Wordle, Worpel (named after a player), I saw a pattern that I could use. Worpel uses "attributes" of an AFL player like their height, playing position, and team, and uses the Wordle "yellow tile" convention to show if you're close in a certain attribute. For example, if the team is not correct, but it is from the correct Australian state. Or if the player's height is within 3 centimetres of the target player's.

After a bit of head-scratching I came up with the 5 categories that I figured would be challenging but with enough possibilities for the "yellow tile" to be helpful. There's no point having a category that can only be right (green tile) or wrong (black tile). The least-useful is probably the "model name" category but of course that is vital to the game, and having played the game literally hundreds of times now, it has on occasion proved useful to know that a certain character appears in the target car's name (obviously cars like the Mazda 6 are hugely helpful here!)

It has been a while since I last did a publicly-visible web side-project, and I wanted to see what the landscape was like in 2022. The last time I published a dynamic website it was on the Heroku platform, which is still pretty good, but I think there are better options these days. After a bit of a look around I settled on Netlify, and so far they've delivered admirably - fast, easy-to-configure and free!

There has been some criticism bandied about for create-react-app recently, saying it's a bad starting point, but for me it was a no-brainer. I figure not having to know how to optimally configure webpack just leaves me more brain-space to devote to making the game a bit better. So without any further ado, I'd like to showcase some of my favourite bits of the app.

Tile reveal animation

Wordle is outstanding in its subtle but highly-effective animations that give it a really polished feel, but I really didn't want to have to use a Javascript animation library to get a few slick-looking animations. The few libraries I've tried in the past have been quite heavy in both bundle-size and intrusiveness into the code. I had a feeling I could get what I wanted with a suitable CSS keyframes animation of a couple of attributes, and after some experimenting, I was happy with this:

      @keyframes fade-in {
        from {
          opacity: 0;
          transform:scale(0.5)
        }
        50%  { 
          transform:scale(1.2); 
          opacity: 0.5; 
        }
        to {
          opacity: 1;
          transform:scale(1.0)
        }
      }

I really like the "over-bulge" effect before it settles back down to the correct size. The pure-CSS solution for a gradual left-to-right "reveal" once a guess has been entered worked out even better I think. Certainly a lot less fiddly than doing it in Javascript:

.BoxRow :nth-child(1) {
  animation: fade-in 200ms;
}
.BoxRow :nth-child(2) {
  animation: fade-in 400ms;
}
.BoxRow :nth-child(3) {
  animation: fade-in 600ms;
}
.BoxRow :nth-child(4) {
  animation: fade-in 800ms;
}
.BoxRow :nth-child(5) {
  animation: fade-in 1000ms;
}

Those different times are the amount of time the animation should take to run - giving it the "sweeping" effect I was after:

Mobile-first

As developers we get far too used to working on our own, fully up-to-date, desktop, browser of choice. But a game like this is far more likely to be played on a mobile device. So I made a concerted effort to test as I went both with my desktop Chrome browser simulating various mobile screens and on my actual iPhone 8. Using an actual device threw up a number of subtle issues that the desktop simulation couldn't possibly hope to replicate (and nor should it try) like the extraordinarily quirky stuff you have to do to share to the clipboard on iOS and subtleties of font sizing. It was worth it when my beta-testing crew complimented me on how well it looked and played on their phones.

Performance

The site gets 98 for mobile performance (on slow 4G) and 100 for desktop from PageSpeed, which I'm pretty chuffed about. I spent a lot of time messing around with Google Fonts and then FontSource trying to get a custom sans-serif font to be performant, before just giving up and going with "whatever you've got", i.e.:

font-family: 'Segoe UI', 'Roboto', 'Oxygen',
          'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
          sans-serif;

... sometimes it's just not worth the fight.

The other "trick" I did was relocating a ton of CSS from the various ComponentName.css files in src right into a <style> block right in the head of index.html. This allows the browser to get busy rendering the header (which I also pulled out of React-land), bringing the "first contentful paint" time down appreciably. Obviously this is not something you'd want to be doing while you're in "active development" mode, but it doesn't have to be a nightmare - for example, in this project I made good use of CSS Variables for the first time, and I define a whole bunch of them in that style block and then reference them in ComponentName.css to ensure consistency.

How do I find all the HTML elements that aren't getting my preferred font?

A quickie script that saved a lot of manually combing through the DOM

Someone noticed that certain elements in our React app were not getting the desired font-face, instead getting plain-old Arial. I wanted to be able to programmatically sniff them out in the browser, so here's what I came up with, mainly thanks to an answer on Stack Overflow for finding all elements on a page, and the MDN documentation for the getComputedStyle function which browsers helpfully expose.

Whack this in your browser's Javascript console and you should be able to hover on any element that is listed to see it on the page:

  // Start where React attaches to the DOM
  const reactRoot = document.getElementById("root"); 
  
  // Get all elements below that
  const kids = reactRoot.getElementsByTagName("*"); 
  
  for (var e of kids) { 
    if (window.getComputedStyle(e)["font-family"] === "Arial") { 
      console.log(e); // Allows it to be hovered in console
    }
  }
  

In case you were wondering, the culprit here was a button that didn't have its font-family set - and Chrome (perhaps others) will use its default (user-agent stylesheet) font style for that in preference to what you have set on the body, which you might be forgiven for assuming gets cascaded down.

Building a retro game with React.js. Part 5 - Fill 'er up

By now I had a good portion of the "player's part" of the game complete. Movement around the game area, drawing lines, detecting intersection with existing lines and preventing illegal movement were all done, using the div-based "drawing" that I was familiar with. But now it was time to draw the filled colour block that results when a closed area has been completed. And I was looking at a wall of complexity that I didn't know how to scale.

The answer was to challenge my own comfort zone. Allow myself to quote, myself:
I could do that with an HTML canvas, but I'm (possibly/probably wrongly) not going to use a canvas. I'm drawing divs dammit!
After all, if ever there was a time and place to learn how to do something completely new, isn't it a personal "fun" project? After a quick scan around the canvas-in-React landscape I settled upon React-konva, which has turned out to be completely awesome for my needs, as evidenced by the amount of code changes actually needed to go from my div-based way of drawing a line to the Konva way:
The old way:

const Line = styled('div')`
  position: absolute;
  border: 1px solid ${FrenzyColors.CYAN};
  box-sizing: border-box;
  padding: 0;
`;

renderLine = (line, lineIndex) => {
  const [top, bottom] = minMax(line[2], line[4]);
  const [left, right] = minMax(line[1], line[3]);
  const height = (bottom - top) + 1;
  const width = (right - left) + 1;
  return <Line key={`line-${lineIndex}`}
                  style={{ top: `${top}px`, 
                           left: `${left}px`,
                           width: `${width}px`,
                           height: `${height}px` }} />;
}

The new way:

import { Line } from 'react-konva';

renderLine = (line, lineIndex) => {
  return (
    <Line
      key={`line-${lineIndex}`}
      points={line.slice(1)}
      stroke={FrenzyColors.CYAN}
      strokeWidth={2}
    />
  );
}

... and the jewel in the crown? Here's how simple it is to draw a closed polygon with the correct fill colour; by total fluke, my "model" for lines and polygons is virtually a one-to-one match with the Konva Line object, making this amazingly straightforward:

renderFilledArea = (filledPolygon) => {
  const { polygonLines, color } = filledPolygon;
  const flatPointsList = polygonLines.reduce((acc, l) => {
    const firstPair = l.slice(1, 3);
    return acc.concat(firstPair);
  }, []);

  return (
    <Line
      points={flatPointsList}
      stroke={FrenzyColors.CYAN}
      strokeWidth={2}
      closed
      fill={color}
    />
  );
}

Time Tracking

Probably at around 20 hours of work now.

Building a retro game with React.js. Part 4 - Drawing the line somewhere

In the previous instalment of this series, I was able to get the player's "sprite" (actually just a div with a border!) to move around the existing lines on the edge of the screen. The next logical step is to allow the player to draw their own lines, which, upon joining at both ends to existing lines, will become part of the "navigable" world the player can manoeuvre through.

Bugs Galore

It was at this point where I started being plagued by off-by-one errors; it seemed everywhere I turned I was encountering little one-pixel gaps when drawing lines, because:

• My on-screen lines are actually 2px wide
• My line-drawing function was doing an incorrect length calculation (had to do (right - left) + 1)
• I was not updating my position at the right time, so was storing my "old" position as the current line's end point; and;
• I was naively using setState and expecting the new this.state to be visible immediately

My solution to almost all of these problems (with the exception of the UI line-drawing function) was to write a heap of unit tests; these generally flushed things out pretty quickly.

Writing the line-drawing function was a weird experience. Virtually every software development "environment" I've ever used before, from BBC Basic on my Acorn Electron on, has had a function like drawLine(startX, startY, endX, endY);. And I could do that with an HTML canvas, but I'm (possibly/probably wrongly) not going to use a canvas. I'm drawing divs dammit! Here's what my function looks like:

renderLine = (line, lineIndex) => {
  const [top, bottom] = minMax(line[2], line[4]);
  const [left, right] = minMax(line[1], line[3]);
  const height = (bottom - top) + 1;
  const width = (right - left) + 1;
  return <Line key={`line-${lineIndex}`}
                  style={{ top: `${top}px`, 
                           left: `${left}px`,
                           width: `${width}px`,
                           height: `${height}px` }} />;
}

Where minMax is a simple function that returns [min, max] of its inputs, and Line is a React-Emotion styled div:

const Line = styled('div')`
  position: absolute;
  border: 1px solid ${FrenzyColors.CYAN};
  box-sizing: border-box;
  padding: 0;
`;

Notice that I resisted the temptation to pass the top, left etc into the Line wrapper. The reason for this is that doing so results in a whole new CSS class being created, and getting applied to the line, every time one of these computed values changes. This seems wasteful when most of the line's attributes remain the same! So I use an inline style to position the very-thin divs where I need it.

Time Tracking

Up to about 12 hours by my rough estimate.

Building a retro game with React.js. Part 3 - I Like To Move It

So with most of the graphical pieces in position, it's time to make things move around.

Again, starting with the easy stuff, I wanted the four directional keys to move the Player around. But in Frenzy, you can only move (as opposed to draw) along the boundaries of the game area and on lines you have already drawn. So if we look at my first iteration of the code in GameArea to handle a request to move the Player left, it's something like this:

 
update = () => {
  if (this.keyListener.isDown(this.keyListener.LEFT)) {
    this.moveLeft();
  }
};

moveLeft = () => {
  if (this.canMove(Direction.LEFT)) {
    this.setState({
       playerX : this.state.playerX -1
    });
  }
}

I ended up bundling quite a lot of smarts into the Direction enumeration in order to make the logic less "iffy" and more declarative. That one Direction.LEFT key encapsulates everything that is needed to check whether a) the player is on a line that has the correct orientation (horizontal) and b) there is room on that line to go further to the left.
A line looks like this:

[Orientation.HORIZONTAL, 0, 0, 478, 0], // startX, startY, endX, endY

and Direction looks like this:

export const Direction = {
  LEFT: {
    orientation: Orientation.HORIZONTAL,
    primaryCoord: (x, y) => y,
    lineToPrimaryCoord: (line) => line[2],
    secondaryCoord: (x, y) => x,
    testSecondary: (c, line) => c > Math.min(line[1], line[3])
  },
  ...
}

My test for whether I can move in a certain direction is:

static canPlayerMoveOnExistingLine = (playerX, playerY, direction, lines) => {
  const candidates = lines.filter(line => {
    return (line[0] === direction.orientation)
  });
    
  const pri = direction.primaryCoord(playerX, playerY);
  const primaryLines = candidates.filter(candidateLine => {
    return direction.lineToPrimaryCoord(candidateLine) === pri;
  });

  if (primaryLines.length > 0) {
    const sec = direction.secondaryCoord(playerX, playerY);
    const found = primaryLines.find(line => {
      return direction.testSecondary(sec, line);
    });

    return typeof found !== 'undefined';
  }
  return false;
} 

Declared static for ease of testing - easy and well worth doing for something like this where actually moving the player around is time-consuming and tedious. It's working well as it stands, although as we all know, naming things is hard. It's pretty easy to follow the process though. At this point I'm holding a lines array in this.state and doing filter and find operations on it as you can see above. We'll have to wait and see whether that will be fast enough. It may well be a good idea to keep a currentLine in state, as most of the time it will be unchanged from the last player movement. Next up, it's time to start drawing some new lines on the screen!

Kudos

I am starting to build up some tremendous respect for the original author of this game; although often dismissed as "very simple" there are some tricky little elements to coding this game and I'm only just scratching the surface. To achieve the necessary performance on an 8-bit, 1MHz processor with RAM measured in the handfuls of kilobytes is super impressive. Assembly language would have been necessary for speed, making the development and debugging a real pain. I haven't even started thinking about how to do the "fill" operation once a line has been drawn and it encloses some arbitrary space, but I suspect the original developer "sniffed" the graphics buffer to see what was at each pixel location - a "luxury" I don't think I'll have!

Time Tracking

Up to about 6 hours now.

Building a retro game with React.js. Part 2 - Screens

Continuing to nibble my way towards the hard parts, I've finished the basic designs of the welcome page, main game page and the paused-game page, including wiring up the key listeners so it all works as expected.

One thing I hadn't thought about was the behaviour of listening to the P key to pause and also resume. The first iteration of the code would flicker madly between the game page and the paused page whenever the P key was pressed - the game loop runs so fast that the app would transition multiple times between the two pages, because the isDown test would simply toggle the state of this.state.paused. I messed around for a while with storing the UNIX time when the key was first pressed, and comparing it to the current time, before realising I really just needed to know what "direction" was being requested, and then waiting for this "request" to finish before completing the transition.

...

debounce = (testCondition, key, newState) => {
  if (testCondition) {
    if (this.keyListener.isDown(key)) {
      // No. They are still holding it
      return true;
    }
    this.setState(newState);
   }
   return false;
};
 
handleKeysWhilePaused = () => {
  if (this.debounce(this.state.pauseRequested, P, {
    pauseRequested: false,
    paused: true
  })) {
    return;
  }

  if (this.debounce(this.state.unpauseRequested,P, {
    unpauseRequested: false,
    paused: false
  })) {
    return;
  }

  if (this.keyListener.isDown(Q)) {
    this.props.onGameExit();
  }

  if (this.keyListener.isDown(P)) {
    this.setState({unpauseRequested: true});
  }
}

handleGameplayKeys = () => {
  if (this.keyListener.isDown(P)) {
    this.setState({pauseRequested: true});
  }
}

update = () => {
  if (this.state.pauseRequested || this.state.paused) {
    this.handleKeysWhilePaused();
  } else {
    this.handleGameplayKeys();
  }
};
...

Effectively I've implemented an isUp(), which is surprisingly not in the react-game-kit library. Oh well, it's all good learning.

Time tracking

I'd estimate the total time spent on the game so far would be 3-4 hours.

A New Old Thing; Building a retro game with React.js. Part 1 - Background and Setup

I've blogged before about entering the fast-paced world of React.js, after a couple of years I'm still (on the whole) enjoying my day job working with it. Over this period React has done a pretty good job of delivering the "maintainable large JavaScript application" promise, but in the apps I built we've seen a few problems that stemmed from our developers' differing levels of experience with design patterns, immutability concepts, higher-order functions and higher-order components.

At the risk of being immodest, I'm comfortable with those concepts - Design Patterns from waaaay back and the functional paradigms from my five-year (and counting) love affair with Scala. What I wanted to explore was - what would happen if I built a React app by myself, endeavouring to write the cleanest, purest software based upon the best starting-point we currently have? How productive could I be? How long would it take to build a working full app? How would maintenance go? How quickly could I add a new feature?

As my day job is basically building CRUD apps, I wanted to do something a lot more fun for this side-project. And what could be more fun than a game? (Mental note: ask people working at Electronic Arts...) There's also a pleasing circularity in building a game and documenting how I did it - back in my earliest days of having a computer, aged about 7, I would buy magazines with program listings and laboriously enter them, line-by-line, while marvelling at how anyone could really have been good enough to know how to do this.

The Game

I'll admit, I've never built a non-trivial game before, but I think attempting an 8-bit home computer game I remember fondly from my childhood, on top of 2018's best front-end technologies, should be about the right level of difficulty.

The game I'll be replicating is called Frenzy; a Micro Power publication for the BBC B, Acorn Electron and Commodore 64. My machine was the Electron - basically a low-cost little brother to the mighty Beeb; highly limited in RAM and CPU, titles for this platform usually needed substantial trimming from their BBC B donor games, despite using the same BBC BASIC language.

Check out the links above for more details and screenshots, but the game is basically a simplified version of "Qix" or "Kix" where the object is to fill in areas of the screen without being hit by one or more moving enemies.

Just for the hell of it, I'm going to place this game front-and-centre on my homepage at http://www.themillhousegroup.com, which I just nuked for this purpose. The page is now a React app being served off a Play Scala backend as per my new-era architecture, and the key technologies I'm using so far are:

• React via Create-React-App
• react-game-kit for game secret-sauce
• react-emotion for component styling

I'm sure more will follow.

Initial Development

To develop the game, I decided to start from the start. The welcome page would need to be suitably old-skool but would force me to consider a few things:

• What screen size should I be working to?
• Can I get a suitably chunky, monospaced font?
• Press Space to start sounds easy, but how do I make that actually work?

Decisions

The original Frenzy must have operated in the BBC's graphical MODE 1 because it used a whopping 4 colours and the pixels were square. So that means the native resolution was 320x256. While it would be tempting to stick to that screen size and thus have it fit on every smartphone screen, I've decided to double things and target a 640x512 effective canvas.
Some searching for 8-bit fonts led me to "Press Start 2P" which, while intended to honour Namco arcade machines, is near enough to the chunky fonts I remember fondly from my childhood home computer that I can't go past it:

As a tiny nod to the present, the "screen" is actually slightly transparent and has a drop shadow - showing how far we've come in 34 years!
The final piece of the welcome screen was achieved by mounting the FrenzyGame component in a React-Game-Kit Loop and using the KeyListener to subscribe to the keys I care about - a quick perusal of the demo game showed me how to use it:

class FrenzyGame extends Component {

  constructor(props) {
    super(props);
    this.keyListener = new KeyListener();

    this.state = {
      gameOver: true  
    };
  }

  componentDidMount() {
    this.loopSubscriptionId = this.context.loop.subscribe(this.update);
    this.keyListener.subscribe([
      this.keyListener.SPACE
    ]);
  }

  componentWillUnmount() {
    this.context.loop.unsubscribe(this.loopSubscriptionId);
    this.keyListener.unsubscribe();
  }

  update() {
    if (this.state.gameOver) {
      if (this.keyListener.isDown(this.keyListener.SPACE)) {
        this.setState({ gameOver: false });
      }
    }
  };

  ...

  render() {
    return this.state.gameOver 
      ? this.renderWelcomeScreen() 
      : this.renderGame();
  }
}

Stack Evolution part 2

Referring back to my go-to stack from part 1 of this series:

Javascript	JQuery, Moment.js, etc
"Presentation"	JSON/AJAX	HTML/LESS
Controllers	Play (Reactive, Async)
Services	Play - RESTful API calls	Mondrian
Persistence	MongoDB (via ReactiveMongo)

I am simply delighted with the performance, scalability, maintainability and reliability of the entire stack from the Controllers layer down - i.e. Scala, Play and Mongo. (Incidentally, I've been running these apps on Heroku with MongoDB provided by MLab, and they have been similarly excellent). So that will not be changing any time soon.

What is no longer tenable is the mixture of HTML (including form submissions), LESS and per-page Javascript. At the top of the table, (i.e. front-end technologies), there is just too much awesomeness happening in this space to ignore. To me, React.js is the current culmination of the best thinking in the front-end world. The way every concept is the most reduced-down thing that could work (as opposed to the competition's kitchen-sink approach) really makes it a pleasure to learn and use.

Currently I'm absolutely loving Create-React-App as a brilliant bootstrapper that continues to add value even once you're up and running. It's got finely-honed and sensible defaults for things like Webpack, is upgradeable in-place, is beautifully documented and is almost psychic in always offering good suggestions or output as to what it's just done, or what can be done next. I currently have no plans to "eject" Create-React-App from any of the front-end projects I'm working on - it's just too useful to keep around.

Into this mix I've also added React Cosmos - this is a "component showcase" system that allows a super-rapid way to see all of the possible "states" of a given React component. The React props that a component needs are specified in fixture files, and Cosmos supplies a nice web UI to browse around and check that changes made to a component are working well and looking good in all of its potential states. It works excellently with the hot-reloading facilities of Create-React-App and really helps nail down component interfaces.

Another element I'm using to try and keep front-end complexity in check is Styled Components. Go have a read of their Github page for the full run-down but basically, I can get the best of both worlds with global CSS used where appropriate, keeping it DRY, together with individual components that won't mess with each other. It also massively helps in stopping the "mental CSS selector" problems during refactoring as observed by Ryan Florence. Extremely cool.

So to summarise, here's my 2017-and-beyond software stack:

Javascript	React.js (with Cosmos)
"Presentation"	JSON/AJAX	JSX/CSS/Styled Components
Controllers	Play (Reactive, Async)
Services	Play - RESTful API calls	Mondrian
Persistence	MongoDB (via ReactiveMongo)

Stack Evolution part 1

It wouldn't take too close a reading of this blog to determine that I'm a huge fan of the Play Framework. For the past 4-or-so years it has played a pivotal role in all of my Millhouse Group projects, and frequently during my "day job" as well.

The "revenue-earning" configuration I've rolled out time and time again looks almost-unfailingly like this:

Javascript	JQuery, Moment.js, etc
"Presentation"	JSON/AJAX	HTML/LESS
Controllers	Play (Reactive, Async)
Services	Play - RESTful API calls	Mondrian
Persistence		MongoDB (via ReactiveMongo)

Yep. Decidedly unsexy, and yes, occasionally the javascript can get a bit funky and coupled to the HTML, but it works and with careful attention to the principles of Single Responsibility and Least Surprise, a "vertical slice" through the functionality would look like:

Javascript	/assets/js/author.js
"Presentation"	/views/html/author/author.scala.html /assets/css/author.less
Controllers	/controllers/AuthorController.scala
Services	/models/Author.scala /services/AuthorService.scala
Persistence	[Authors collection in MongoDB]

... where no source file is more than 200 lines of code. Nothing too controversial there, I think you'd agree.

However...

My exposure over the last 18 months to React.js has truly opened my eyes to the potential of a true front-end application (as opposed to the very 2010-era progressively-enhanced-markup approach I've described above). In the next post I'll show the architecture I've been calling the CRAP-stack which has been making working on the Javascript front-end as pleasant as doing the heavy-lifting with Scala in the back-end (i.e. very!)

The CRAP Stack, Part 3 - Front-End Routes with a Play server

As I continue to develop my React app that is hosted on a Play backend, I've come across the need to support "front-end routes"; that is, URLs that look like this:

  http://myapp.com/foo/bar

where there is no explicit entry for GET /foo/bar in Play's routes and nor is there a physical asset located in /public/foo/bar for the Assets controller to return to the client, as we set up in the last instalment:

  # Last of all, fall through to the React app
  GET /       controllers.Assets.at(path="/public",file="index.html")
  GET /*file  controllers.Assets.at(path="/public",file)

What we'd like is for the React application at index.html to be served up, so that it can then consume/inspect/route from the original URL via the Window.location API.

As it stands, the last line of routes will match, the Assets controller will fail to find the resource, and your configured "client error handler" will be called to deal with the 404. This is not what we want for a "front-end route"!

We want requests that don't correspond to a physical asset to be considered a request for a virtual asset - and hence given to the React app. And after a bit of fiddling around, I've come up with a FrontEndServingController that gives me the most efficient possible way of dealing with this. The Gist is available for your copy-paste-and-improve pleasure, but the key points are:

The fall-through cases at the bottom of routes become:

  GET /       controllers.FrontEndServingController.index
  GET /*file  controllers.FrontEndServingController.frontEndPath(file)

Those methods in FrontEndServingController just being:

  val index = serve(indexFile)

  def frontEndPath(path: String) = serve(path)

  private def serve(path: String) = {
    if (physicalAssets.contains(path)) {
      logger.debug(s"Serving physical resource: '$path'")
      assets.at(publicDirectory, path, true)
    } else {
      logger.debug(s"Serving virtual resource: '$path'")
      // It's some kind of "virtual resource" -
      // a front-end "route" most likely
      assets.at(publicDirectory, indexFile, true)
    }
  }

We're still using Play's excellent built-in AssetsController to do the hard work of caching, ETags, GZipping (all the classic webserver jobs) - we have injected it as assets using Dependency Injection - composition FTW. That true argument tells it to use "aggressive caching" which is ideal for this scenario where the bundle files we're serving up already have a cache-busting filename.
And now the "clever" bit being a recursive scan of the /public directory when we start up, assembling a definitive (and immutable!) Set[String] of what's actually a physical asset path:

  lazy val physicalAssets:Set[String] = {
    val startingDirectory = new File(physicalPublicDirectory)
    deepList(startingDirectory)
  }

  private def deepList(f: File): Set[String] = {
    val these = f.listFiles.toSet
    val inHere = these.filter(_.isFile).map { f =>
      f.getPath.replace(physicalPublicDirectory, "")
    }
    val belowHere = these.filter(_.isDirectory).flatMap(deepList)
    inHere ++ belowHere
  }

A Top Shelf Web Stack - Scala version - Play 2.5 + create-react-app

There are tutorials on the web about using ReactJS with Play on the back end, such as this one by Fabio Tiriticco but they almost always achieve the integration via the WebJars mechanism, which, while kinda neat and clever, can never be "first-class citizens" in the incredibly fast-moving JavaScript world. My day job uses a completely separate front- and back-end architecture which has shown me that letting NPM/Webpack et al manage the front-end is the only practical choice if you want to use the latest-and-greatest libraries and tooling.

A fantastic example is create-react-app, by Dan (Redux) Abramov et al, which makes it ludicrously-simple to get going with a React.JS front-end application that incorporates all of the current best-practices and configuration. I came across a very fine article that discussed hosting a create-react-app front-end on a Ruby-on-Rails server (in turn on Heroku) and figured it would be a good exercise to do a version with the Play Framework 2.5 (Scala) on the back end. This version will have a lot fewer animated GIFs and general hilarity, but hopefully it is still a worthwhile exercise!

I won't go into setting up a simple Play app as complete instructions for both beginners and experts are provided at the Play website, and it can be as simple as typing:

  % sbt new
  % sbt run

Once you've got your Play app all happy, getting the front-end going is as simple as running two commands in your project's root directory:

  % npm install -g create-react-app
  % create-react-app client

to create a React application in the client directory. You can check it works with:

  % cd client
  % npm start

Great. Now's a good time to commit your new client directory to Git, although you'll definitely want to add client/node_modules to your .gitignore file first. Let's modify the backend to have a tiny little JSON endpoint, call it from React when the app mounts, and display the content. First, we just add one line to our package.json so that backend data requests get proxied through the front-end server, making everything just work with no CORS concerns:

  "private": true,
  "proxy": "http://localhost:9000/",
  "dependencies": {
    "react": "^15.5.4",
    "react-dom": "^15.5.4"
  },

Make sure you kill-and-restart your React app after adding that line. Next, let's whip up a Play endpoint that returns some JSON: In conf/routes:

GET          /dummy-json       controllers.DummyController.dummyJson 

In app/controllers/DummyController.scala:

class DummyController extends Controller {

  val logger = Logger("DummyController")

  def dummyJson = Action {
    logger.info("Handling request for dummy JSON")
    Ok(Json.obj(
      "foo" -> "foo",
      "bar" -> "bar",
      "bazzes" -> Seq("baz1", "baz2", "baz3")
      )
    )
  }

Check that's all good by hitting http://localhost:9000/dummy-json directly with your browser. Now we put our front-end hat on and get the React app to fetch the JSON when it mounts:

class App extends Component {

  componentDidMount() {
    console.log('Mounted');
    fetch('/dummy-json',{ accept: 'application/json'})
      .then(response => response.json() )
      .then(json => console.log(json) )
      .catch(error => console.error(error));
  }
 ...
}

Setting the accept header is not strictly necessary but it helps create-react-app to know that this request should be proxied. Plus it's generally good form too. Now when your app hot-reloads, watch your browser's Network tab. You'll see the request go out on port 3000, the server log the request and respond on 9000, and the response arrive back on port 3000. Let's finish off the local-development part of this little demo by wiring that response into our app's state so that we can render appropriately:

class App extends Component {

  constructor() {
    super();
    this.state = {};
  }

  componentDidMount() {
    console.log('Mounted');
    this.fetch('/dummy-json').then( result => {
      this.setState({
        result
      });
    });
  }

  fetch (endpoint) {
    return new Promise((resolve, reject) => {
      window.fetch(endpoint, { accept: 'application/json'})
      .then(response => response.json())
      .then(json => resolve(json))
      .catch(error => reject(error))
    })
  }

  render() {
    let { result } = this.state;
    return (
      <div className="App">
        <div className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          {result ? (
            <h3>{result.foo}</h3>
          ) : (
            <h3>Loading...</h3>
          )
        }
        </div>
        <p className="App-intro">
          To get started, edit src/App.js and save to reload.
        </p>
      </div>
    );
  }
}

So easy! In the next installment, we'll consider deployment to Heroku.

IE9/10: Silent failure to POST a form

Spent far too long debugging an issue reported against one of my sites where a simple (but important) form submit would almost-silently fail in "IE". After downloading the massive-but-excellent IEVMs I was able to narrow it to Internet Explorer 9 and 10 (I don't care about older versions, but it would certainly affect them too).

After checking and fixing the minor HTML validity issues, replacing the submit button with an input type="submit" and various other insignificant tweaks, I actually read the error message IE was showing me:

I'd been obsessing over the way the request was being aborted so fast and was thinking there was some kind of weird cross-origin thing blocking my POST (my page is first step in a payment process - summarising the costs, and clicking the big Pay Now button caused a POST to a third-party payment processor). I'd completely failed to notice the final bullet point in the "More Information" section which mentions SSL and TLS versions. After heading into Settings -> Internet Options -> Advanced -> Security and checking "Use TLS 1.2", everything started working perfectly.

I have no idea why Microsoft would implement new versions of an important protocol but then default the support to OFF. Must be one of the backwards-compatibility hacks that (apparently) they could then finally be rid of in IE 11, where it is turned ON by default.

Hope it helps someone.

ReactJS - Early Thoughts

It is very-much a recruiter's term but I, like many who started out with "back-end" skills, am endeavouring to be a proper Full Stack Developer.

I have always been comfortable with HTML, and first applied a CSS rule (inline) waaay back in the Netscape 3.0 days:

  <a href="..." style="text-decoration: none;">Link</a>

I'm pretty sure there were HTML frames involved there too. Good times. Anyway, the Javascript world had always been a little too wild-and-crazy for me to get deeply into; browser support sucked, the "standard library" kinda sucked, and people had holy wars about how to structure even the simplest code.

Up to about 2015, I'd been happy enough with the basic tools - JQuery for DOM-smashing and AJAX, Underscore/Lodash for collection-manipulation, and bringing in Bootstrap's JS library for a little extra polish. The whole thing based on full HTML being rendered from a traditional multi-tiered server, ideally written in Scala. I got a lot done this way.

I had a couple of brushes with Angular (1.x) along the way and didn't really see the point; the Angular code was always still layered on top of "perfectly-good" HTML from the server. It was certainly better-structured than the usual JQuery mess but the hundreds of extra kilobytes to be downloaded just didn't seem to justify themselves.

Now this year, I've been working with a Real™ Front End project - that is, one that stands alone and consumes JSON from a back-end. This is using Webpack, Babel, ES6, ReactJS and Redux as its principal technologies. After 6 weeks of working on this, here are some of my first thoughts:

• Good  ES6 goes a long way to making Javascript feel grown-up
• Bad The whole Webpack-Babel-bundling thing feels really rough - so much configuration, so hard to follow
• Good React-Hot-Reloading is super, super-nice. Automatic browser reloads that keep state are truly magic
• Bad You can still completely, silently toast a React app with a misplaced comma ...
• Good  ... but ESLint will probably tell you where you messed up
• Bad It's still Javascript, I miss strong typing ...
• Good  ... but React PropTypes can partly help to ensure arguments are there and (roughly) the right type
• Good  Redux is a really neat way of compartmentalising state and state transitions - super-important in front-ends
• Good  The React Way of flowing props down through components really helps with code structure

So yep, there were more Goods than Bads - I'm liking React, and I'm finally feeling like large apps can be built with JavaScript, get that complete separation between back- and front-ends, and be maintainable and practical for multiple people to work on concurrently. Stay tuned for more!

Building reusable, type-safe Twirl components

I've been doing quite a lot of work on a Play Framework 2.4.x app recently, a hit upon a little problem that others have noted as well. I'm trying to make the view layer as nice a place to be as the "main" codebase - after all, it's all Scala - and so I'm extracting out anything re-usable into a components package.

Here's a simple example. I'm using Bootstrap (of course), and I'm using the table-striped class to add a little bit of interest to tabular data. The setup of an HTML table is quite verbose and definitely doesn't need to be repeated, so I started with the following basic structure:

@(items:Seq[_], headings:Seq[String] = Nil)
  <table class="table table-striped">
      @if(headings.nonEmpty) {
      <thead>
          <tr>
            @for(heading <- headings) {
                <th>@heading</th>
            }
          </tr>
      </thead>
      }
      <tbody>
        @for(item <- items) {
            <tr>
                ???
            </tr>
          }
        }
      </tbody>
  </table>

Which neatens up the call-site from 20-odd lines to one:

  @stripedtable(userList, Seq("Name", "Age")

Except. How do I render each row in the table body? That differs for every use case!
What I really wanted was to be able to map over each of the items, applying some client-provided function to render a load of <td>...</td> cells for each one. Basically, I wanted stripedtable to have this signature:

@(items:Seq[T], headings:Seq[String] = Nil)(fn: T => Html)

With the body simply being:

   @for(item <- items) {
      <tr>
        @fn(item)
      </tr>
   }

and client code looking like this:

  @stripedtable(userList, Seq("Name", "Age") { user:User =>
    <td>@user.name</td><td>@user.age</td>
  }

...aaaaand we have a big problem. At least at time of writing, Twirl templates cannot be given type arguments. So those [T]'s just won't work. Loosening off the types like this:

@(items:Seq[_], headings:Seq[String] = Nil)(fn: Any => Html)

will compile, but the call-site won't work because the compiler has no idea that the _ and the Any are referring to the same type. Workaround solutions? There are two, depending on how explosively you want type mismatches to fail:

Option 1: Supply a case as the row renderer

  @stripedtable(userList, Seq("Name", "Age") { case user:User =>
    <td>@user.name</td><td>@user.age</td>
  }

This works fine, as long as every item in userList is in fact a User - if not, you get a big fat MatchError.

Option 2: Supply a case as the row renderer, and accept a PartialFunction

The template signature becomes:

@(items:Seq[_],hdgs:Seq[String] = Nil)(f: PartialFunction[Any, Html])

and we tweak the body slightly:

   @for(item <- items) {
      @if(fn.isDefinedAt(item)) {
        <tr>
          @fn(item)
        </tr>
      }
   }

In this scenario, we've protected ourselves against type mismatches, and simply skip anything that's not what we expect. Either way, I can't currently conceive of a more succinct, reusable, and obvious way to drop a consistently-built, styled table into a page than this:

  @stripedtable(userList, Seq("Name", "Age") { case user:User =>
    <td>@user.name</td><td>@user.age</td>
  }

nginx as a CORS-enabled HTTPS proxy

So you need a CORS frontend to your HTTPS target server that is completely unaware of CORS.

I tried doing this with Apache but it couldn't support the creation of a response to the "preflight" HTTP OPTIONS request that is made by CORS-compliant frameworks like jQuery.

Nginx turned out to be just what I needed, and furthermore it felt better too - none of that module-enabling stuff required, plus the configuration file feels more programmer-friendly with indenting, curly-braces for scope and if statements allowing a certain feeling of control flow.

So without any further ado (on your Debian/Ubuntu box, natch) :

Get Nginx:

sudo apt-get install nginx

Get the Nginx HttpHeadersMore module which allows the CORS headers to be applied whether the request was successful or not (important!)

sudo apt-get install nginx-extras

Now for the all-important config (in /etc/nginx/sites-available/default ) - We'll go through the details after this:

#
# Act as a CORS proxy for the given HTTPS server(s)
#
server {
  listen 443 default_server ssl;
  server_name localhost;

  # Fake certs - fine for development purposes :-)
  ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
  ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

  ssl_session_timeout 5m;

  # Make sure you specify all the methods and Headers 
  # you send with any request!
  more_set_headers 'Access-Control-Allow-Origin: *';
  more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE';
  more_set_headers 'Access-Control-Allow-Credentials: true';
  more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept';

  location /server1/  {
    include sites-available/cors-options.conf;
    proxy_pass https://<actual server1 url>/;
  }

  location /server2/  {
    include sites-available/cors-options.conf;
    proxy_pass https://<actual server2 url>/;
  }
}

And alongside it, in /etc/nginx/sites-available/cors-options.conf:

    # Handle a CORS preflight OPTIONS request 
    # without passing it through to the proxied server 
    if ($request_method = OPTIONS ) {
      add_header Content-Length 0;
      add_header Content-Type text/plain;
      return 204;
    }

What I like about the Nginx config file format is how it almost feels like a (primitive, low-level, but powerful) controller definition in a typical web MVC framework. We start with some "globals" to indicate we are using SSL. Note we are only listening on port 443 so you can have some other server running on port 80. Then we specify the standard CORS headers, which will be applied to EVERY request, whether handled locally or proxied through to the target server, and even if the proxied request results in a 404:

  more_set_headers 'Access-Control-Allow-Origin: *';
  more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE';
  more_set_headers 'Access-Control-Allow-Credentials: true';
  more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept';

This last point can be important - your JavaScript client might need to inspect the body of an error response to work out what to do next - but if it doesn't have the CORS headers applied, the client is not actually permitted to see it!
The little if statement that is included for each location provides functionality that I simply couldn't find in Apache. This is the explicit response to a preflighted OPTIONS:

  
    if ($request_method = OPTIONS ) {
      add_header Content-Length 0;
      add_header Content-Type text/plain;
      return 204;
    }

The target server remains blissfully unaware of the OPTIONS request, so you can safely firewall them out from this point onwards if you want. A 204 No Content is the technically-correct response code for a 200 OK with no body ("content") if you were wondering.
The last part(s) can be repeated for as many target servers as you want, and you can use whatever naming scheme you like:

 location /server1/  {
    include sites-available/cors-options.conf;
    proxy_pass https://server1.example.com;
  }

This translates requests for:

    https://my.devbox.example.com/server1/some/api/url

to:

    https://server1.example.com/some/api/url

This config has proven useful for running some Jasmine-based Javascript integration tests - hopefully it'll be useful for someone else out there too.

SOTSOG 2013H1

Some cool stuff that makes life way easier to do webapps in 2013:

• Play! Framework 2.1 - Scala, Hit-refresh recompilation, built-in LESS, proper request routing (no annotations!) and no XML. Nuff said.
• LESS CSS - Does a great job of de-repetitioning CSS.
• Angular.JS - the most unobtrusive client-side framework I've ever used - just feels like "dynamic HTML"

Also (not yet used, but fully intend to):

• Grunt
• GruntStart
• play-grunt-angular-prototype

I note that in 2013, Standing on the Shoulders Of Giants is more like Standing on the Shoulders of Giants Standing on the Shoulders of Giants - all of the above technologies build on something a bit older, a bit cruftier, a bit trickier.
Have we finally hit the Industrial Revolution in software development?

I <3 WWW

As I'm not an iOS or Android developer I attended the August "Mobile Focused" Melbourne YOW! Night with a certain degree of reluctance.

I had visions of skinny-jeaned, black-plastic-spectacled iOS-developing hipsters telling me how, like, awesome their, like, app was because like, it totally like mashed up the tweeting paradigm by like ironically cross-pollenating the artisan tumblr mixtape vibe (thanks Hipster Ipsum!).

And while there was a certain amount of that, I was somewhat refreshed to hear from a couple of speakers (native mobile app developers, mind you) that a great deal of the time, when you strip away all of the hype, what the customer needs can be addressed with a mobile-focused website.

I've seen first-hand the hype-storm that (in particular) the iPad has created. One of the speakers at YOW! painted a pretty accurate picture:

• CEO's wife buys CEO a new iPad because they're shiny
• CEO can't really use iPad but notes these apps on the home screen have pretty icons
• CEO sees other companies have apps
• CEO decides his company needs an app
• CEO doesn't exactly know what app is for, but knows that it's needed NOW

What follows (MyCompanyApp v1.0) is usually an app-ification of what the company's current website offers. It is a pointless, money-wasting exercise that gives the CEO his app icon that he can show off in the golf club, but actually has zero advantage over his website (actually negative advantage because it doesn't self-update like a web page).

If the iPad had a way to add a shortcut to a webpage as a home-screen icon, the whole thing could have been done in 30 seconds.

As mobile devices get increasingly capable, well-written, lightweight web pages with advanced features like HTML5 Geolocation can get pretty damn close to the "native app experience" - with none of the approval-process or old-installed-version hassles.

So that's the basket where I'm putting my eggs for the next little while; I just hope the egg doesn't end up on my face :-)

The CSS F(l)ail

I've written before about CSS and why it seems to be so hard for a Typical Java Developer (TJD) to craft "nice" HTML+CSS presentation layers.

Here's the behavioural pattern I've observed when a TJD has to do some front-end web-work

:

• Panic - rather than embrace the opportunity to learn a new paradigm/technology, as they would probably enjoy doing if it was a Java API


• Copy-and-paste from the existing codebase - despite knowing that this leads to unmaintainable code no matter what the language


• Copy-and-paste from the web - without understanding what is going on


• Randomly tweak values in the browser using Firebug or Chrome Developer Tools until it "looks right" (I've started to call this CSS flailing)


• Give Up And Use Tables (for non-tabular data). The ultimate fail

In posts-to-come I'm going to try and identify some CSS "patterns" that a software developer can actually understand (as opposed to copy-and-pasting) which hopefully will put an end to the above behaviours, any of which would be regarded as unforgivable if it was perpetrated in the squeaky-clean corridors of Java-land ;-) but are somehow considered par-for-the-course in the unfamiliar yet forgiving world of UI...