I've had this post brewing for a long time and tried several versions of it. If your reading this then hurray this one must have been "good enough". I hope the following ramblings make some sense. Be warned, this is a long one but it needs to be.
Umbraco CMS currently has never been stronger. Version 7 is absolutely amazing and rock solid. These are golden days that we live in as an Umbraco developer, agency and for our editors. Bravo. Its a testament of just how far it has come on that I raise the thorny issue of rebuilding a large chunk of it, are we getting to the point where we need to think about rebuilding the back office? I know right, what am I thinking? That makes no sense, I just said it was rock solid so why rebuild it? Hear me out though.
This is a bit of a controversial topic as it means change and as humans we don't cope with change all that well. So let me quickly cross off some of the obvious queries and knee jerk reactions you might have before we get into the meat of this thought experiment to try to calm some nerves and sent the scene:
Its a task for HQ but as a community we can start and shape the discussion, after all its us that will be using it.
We don't have to rebuild the back office at all if we don't want to!
Its a really big job and I don't expect anything to start on it for 18 months at least
Due to above lead time I'm not suggesting any tech choice at all at this time, just ideas. This post is framework agnostic on purpose.
I know you have probably invested heavily in learning Angular and all that jazz and don't want to lose that knowledge or have the heart to retool, I get that.
The current back office has been amazing and will continue to be for some years yet
Right, that is the obvious stuff covered. You all good? Great, lets do some more digging and see why I and others are starting to murmur about a rebuild, whats wrong with the current one and what benefits we might get from it.
Very short history lesson
Honestly this is the short version but a little history is needed so we know how we got here and where we might go. The current back office is in its 5th year now, I remember watching Per Plough talking in a key note about Belle (the code name for the new look back office) at Las Vegas at the first uWestFest. In that talk he explained the choice for using Angular, the departure from .net for making custom backend widgets and property editors (woohoo! good bye user controls).
Angular was in version 1 at the time and is really one of the first frameworks for making web applications, websites that do complicated stuff, what we would call now a Single Page Application (SPA). It was a bold but right choice. It gave the back office the javascript goodness it needed to make it as smart and smooth as it turned out. One of the additional drivers was the explosion in JavaScript in general, thanks to Node.js and GitHub the amount of JavaScript goodies we might be able to leverage in a JavaScript backend was huge, reason enough to get in the game. Trouble is not much of that awesome additional JavaScript did make it into the back office.
Mostly folks rolled their own JavaScript so we didn't get much of the promised return of being able to dip into the pool of other tools already out there. The only exception that springs to mind is Marc Stöcker's Sir Trevor package that leveraged a gird like editor from ITV News. This though died a death once the Grid was released in Core. Our recent package Ernest was our attempt at seeing what you could do if we could play with some of this other JavaScript tech, you can read more about the hows and why if you like. We also had a point where a handful of developers in the community create a bunch of property editors etc. that the rest of us could use so we didn't all need to become Angular experts. Those that were gave us 90% of what we could ever want.
Umbraco as it is is stuck on Angular version 1.1.5 (or there about) with rumours of it even having some customisation in there too...but of course it has. Angular moved on of course but for some reason the Back office didn't (I suspect they changed the API at some point which broke some stuff before a big release). Then something odd happened within the Angular development team.
Other frameworks started coming out, some really big apps were made with Angular, new ideas got added to the mixing pot, lessons were learnt. To fix these issues and make changes they decided to completely rewrite it...over several years...with no backwards compatibility. It didn't go down well from those that had invested in the framework, everyone was now in effect in a technology dead end, including of course Umbraco. Angular v2 (actually v5 I believe...no not THAT v5) is out and the version Umbraco is on is now referred to as AngularJS. Confused? You should be, its a bit of a mess. There are ways to allow for upgrading to the latest Angular framework but its nothing really simple so although this might sound like an option it really just muddies the waters even more in my opinion.
What is wrong with mature tech?
So we have a framework that has done us proud for 5 years, what is wrong with that? Why would we want to change it when it sounds like its working?
There is nothing wrong with this, in fact its great that we've had so long a run out of it and its still going strong. Its allowed HQ to mostly switch the back office over from user controls and hard coded web forms of V6 fame over to API endpoints which now power the JavaScript front-end via Ajax (we still call it that right?). There are only a handful of legacy bits and bobs still floating around and the odd iframe escape hatch here and there...I'm looking at you Rollback and Audit Trail!
This means that we "mostly" have a separated API back office from the front-end back office. As a result this means that the investment in development effort to switch from Umbraco V6 to V7 would not need to be repeated should we chose to switch out the front-end of the back office in the future. Rather than a complete rewrite we only need change the front-end layer and continue to use the existing API's if that makes sense (I am of course aware its a huge task and that sentence makes it sound like its easy).
The trouble is while it might be mature its not really moved on. Its difficult to teach it new tricks. In the meantime the JavaScript world has not stood still, nor has the learning or the best practises or the options and opinions of how to do stuff. AngularJS in its railway siding has stood still. Again this might be just fine. I suspect we will hear people say "That's me out then, I'll go else where if you change" but this is the internet, things are changing all the time, you have to be constantly learning to keep up. That is not to say we should throw stuff away or to chase the shiny new things just for the sake of it but we've had 5 years of a stable framework (with another 18+ months still to come) that is like a century in internet time. We can't stand still and I feel the time is approaching were we need to think about the future.
We could just keep using it as it is. It works right? Its certainly an option and to be clear we will have to until a future replacement is ready anyway assuming we rebuild at all. But there is a danger that its not just the tech that becomes legacy but the knowledge of how to do it too. Currently getting docs on AngularJS v1.1 is not easy or obvious, there are not many Stackover flow questions flying around for the version we are on, the knowledge is hard to find/attain and is not all that transferable. Umbraco training is the only place you can get a few hours of hands on time in AngularJS. New front-end developers coming to Umbraco are going to have a hard time of it and also be amazed it doesn't do some of the cool stuff they just take for granted these day. Dare I say that it could be that AngularJS is becoming the the XSLT of previous versions? It did the job but folks found it old, quirky and odd (unlike Razor which is just quirky and odd).
Looking under the hood
I've recently been doing a lot of digging into the back office as part of a optimisation project for a client and if honest I had an itch to scratch. Its the sort of digging most won't do as really we only normally have to build property editors and dashboard but when you start digging it sort of becomes clear that a lot of the core of the back office, the really deep stuff has the feel that it is still pretty much like it was 5 years ago when the first prototype was built. Its got that "all best intentions are to come back to sort this out once we've got this prototype up and running" feel about it, you know, the ones you never actually get the chance to come back to.
This is not a direct criticism (we all have code like that), what is there works really well and enabled all this in the first place, but imagine if we could re-visit it and get it bang on rather than "it works". Then rather than the black magic feel that it has at the minute others would be able to dig into it more, pull requests would be easier, code layout would be better, services would be obvious and standardised. There is a benefit right there but you'd need a reason to go in there and do that, a rebuild (or retool) would be the perfect time. As is I suspect few dare go too deep (even in HQ) and poke it from a distance with sticks. This is of course just my opinion/judgement from looking around it, I could be wrong.
As an example we've got a site that download nearly 3.5Mb of JavaScript every single page load. This is all needed to run the back office. But its EVERYTHING that is needed to run the back office, not technically true as it downloads even more on the fly later. Angular needs all the Controllers you might need up front before it bootstraps the app so Umbraco sends everything down the wire upfront. So if your an editor you still load all the code to edit users, DocTypes, data type settings, etc. stuff they don't have the access to, so why are they having to download it? Lots of bloat. Not a problem on our developer machines but editors are rarely on cutting edge kit are they? Plus having that much JS running all day can make a browser sick. We've got clients that reboot the browser twice a day to make the back office "snappy" again.
Editors spend every day in the back office, why are we slowing them down with code they will never run? JavaScript eats CPU, parsing it and running it takes resources. This is not a minification issue its a bloat issue and its a limitation of the framework. This could be mildly/nearly fixed (and its something we are working on, don't worry there will be a pull request) but in modern frameworks you can load stuff on the fly at the point of need via something called Code Splitting and its just a given. To find out you can't in Angular is an issue, not a complete deal breaker just not ideal, others can do it why can't we? I've got about 20 other issues like that that are similar, stuff that should be easy to do but isn't as the framework has stood still. Sure we could hit what is there with hammers to put some of this stuff right but is that time well spent? Back in the days when just trying to make this SPA stuff easier was the goal AngularJS was amazing but now we've "cracked SPA's" the focus switched to making them better, faster, more flexible and responsive but you can't do it with AngularJS as is.
Transferable skills
Some of you I know use Angular on the front end of you websites and the fact you can use it both in the back office of Umbraco and on your client sites is fantastic. I'd get why you wouldn't want to change the back office tech. But again time doesn't stand still and finding staff who can pick up a site or a widget or a package is starting to become a problem, because you won't relearn a tech they have to learn an old one that is in a dead end.
By updating the tech we get to tap into the existing knowledge of new hires rather than have to train them up. Again we've had 5 years out of AngularJS in Umbraco, that is not a bad run now is it?
What about all those packages we've written, if you rebuild won't we lose those? Well yes, yes you will. But you'll have plenty of heads up that its coming and in the meantime we will have had V8 and maybe even V9 by then which will have had their own share of breaking changes. If the packages are good enough then there is time to convert them over. Luckily some of the biggest packages are already in Core (hello Nested Content) so would get converted anyway, "most" others don't go into crazy depths with Angular or if they do they tend to use stuff in the Core that can change between versions and break anyway (Angular "tweaks" are never considered a breaking change by HQ it seems).
Wait, I mentioned an iframe escape hatch for old legacy code from v6 a while back, could we not do something like that for the older AngularJS stuff? Hell no! How long do you think your code should stick around for? V8 is the "clean up" version were we get to dump a lot of really old legacy code, it would be a damn shame to then straight away start a fork of the back office and try to run two versions of it at once. This is meant to be a clean up and a rebuild not trying to run multiple out dated frameworks in one. The biggest code to be changed will be the Core code that runs the back office, that to me is where the biggest gains can be had for performance, maintainability, consistency and extensibility. Older packages will simply have to be updated or retired. If the are that good then others will upgrade them and bring them into the new style if you don't have the heart or time so fear not.
I'd draw a line under old AngularJS code if it was me but then its not me in charge is it? So yes you "could" run your property editor code in whatever new framework we use, I'm sure we could add a bunch of shims and proxies to make it work one way or another its just not pretty and I don't think its needed.
What do we actually "develop" in the back office?
When there is talk of a rebuild or "we should use framework X in the back office" discussions there are some that think that this would cause all their skills to be scrapped and a massive effort to relearn new technology (or yet another framework). Thing is, what do we actually "do" in the back office as front-end developers? With a few notable exceptions most of us build a fancy property type, which might need to make an API call to either Umbraco or a 3rd party end point, maybe a dash board, create a new section, maybe add a menu option. That is it really, that is 95% of what we might want to do.
If in whatever new back office tech we use if we can make those tasks easy to do then its not really a complete "re-learn" of technology. It could be that we even make it easier to do these things. Not having to have to create 3-5 files for a property editor for instance would be a benefit for me right away! Imagine getting rid of the complication of Dependency Injection, in modern JavaScript you just include what you need rather similar to a using statement in C# rather than needing a framework to inject it in for you from magic strings, another hurdle to learning removed for new folks and one less lot of square brackets and dollar signs for us old timers to remember about.
Again, if you often dig deeper than that then I can see why you might not want to relearn yet more tech if you've become good enough to dive that deep regularly. However, if you are going that deep then you should be able to see the cracks and the bits that are not right and were we could improve and you also know JavaScript really well and in that case you should be able to turn your hand to any JavaScript powered tech, especially if we find one that makes it easier or at least gives us to the chance to put right some of those cracks and rough bits. I get wanting to stick with tech you know, I only shut down my last Classic ASP site 5 year ago so I get it but I killed it as I was the only one that knew it and I didn't want to train anyone else up on it as it was dead knowledge.
How do we develop for the back office?
When we talk about new JavaScript tech it doesn't take long for groans about "crazy complicated tool chains" start to be heard. Modern JavaScript development from the outside seems to have a million and one tools you HAVE to use and they all seem super complicated, impenetrable and stupidly named. The reality is though that all these tools are also maturing and developing (its one of the reason there seems to be so many, lots of brains on lots of problems) and part of that is making things easier to use and ideally a faster experience for the end user. Noble aims. We add caching to make things faster in our code despite the additional complications it brings because it makes for a better end user experience. Optimisations often come with a complication trade off but thats not to say we can't try to make it easier for everyone hence the amount of tools in use these day.
Take WebPack for instance, its become the current defacto standard for "compiling" your JavaScript and to be fair its held that crown for the last 3 years. By compiling your code and assets you get a whole world of goodies available to you like usage of new language features before they come out, convert your SASS to CSS, embed your CSS in your JS and reduce a network request, use short cut syntax or domain languages, get type checking, awesome minification way beyond just removing spaces and new lines, code splitting and so much more. Then it can bundle all your code up into compact bundles of code and then do something call "tree shaking" which removes un-used code to reduce your file size. All quite magic really. Its now a sponsored open source product expertly headed up by the amazingly friendly and helpful Sean Larkin. Lots of big companies are paying for its development as their products depend on it. Sean and the team are doing lots of great work to promote it, make it easier, faster and better documented and they are doing a fine job of it too!
However the thought of having to use that tool chain probably terrifies you if your not already using it. You probably would prefer to see your JavaScript in the browser raw, vanilla js edited in a text editor of your choice. I'd agree with you if you are developing a property editor, you shouldn't need to do all that. We don't need to compile your 20 line property editor. You should be able to hack at a JavaScript file save it and it just works in the back office. Lets make that a requirement to have as minimal tool chain as possible but ideally no tool chain when developing property editors. Hardcore devs can of course opt-in to running their stuff through a tool chain for bonus points.
The core of the back office on the other hand and if you need to "go deeper" should use tech like WebPack as the gains would benefit everyone. If you want to work on core we'd have a guide on how to get setup with WebPack et al and a config file already written for you so it all just "works" without you having to do much other than have Node installed and follow a few one time command line instructions (or like we do just run a .bat file). This is about getting gains for everyone not just JS whizz kids who like playing with fancy stuff, that won't help any of us now will it?
How might we do it?
HQ would have to do this one, this is not one the community could do. But as a community we can help shape what we would want it to go like and let HQ know that its something we want doing and focusing on (assuming it is).
I really like the current look and feel, I don't think there is any reason to depart from it currently and it would only add more work to what would already be a bit job. I'd suggest that on the whole we leave it as is and just rebuild the tech under it.
I'd suggest the ideas of what it should be like are fleshed out between now and then and that the intentions to rebuild are announced with a rough version number to target for. Right now we could do a bit of optimisation work in the existing back office code to tidy it up a bit and get it fit for the next 2 years. Then it can hold the fort while the new rebuild beings and is completed along side it. This is very similar to how we would rebuild a large website, shore up the existing site to buy us time while we work on the new one along side it, once ready swap it out. Again I think starting this is about 18 months off given HQ's apparent current work load and aims but seeing the new road map at Code Garden might give us a better indication of time. If we shout about it enough they might pull it forward but I think starting it in 18 months or so is about right for a target.
Then ideally the new back office gets released as V9 or possible V10 (which I suspect won't be called that and will be Version X or something, that seems to be the in thing to do these days) and there will be much rejoicing.
The first bit though is discussing what we want. Along with the ease of development, faster performance, more modern tools/best practises mentioned above I'd also like to see more formalisation of the data and services in the back office. For instance we have a ServerVariables global object in there at the minute that is a bit of a holding pent for all sorts of data no one can seem to find a real home for. I'd rather than was cleaned up some. For the optimisation work I've been trying to find a way to get the current users locale so I can remove 300Kb of JavaScript language files for moment.js and only download the one the user needs. Trouble is I can't find a sensible place to get this from and I'm looking to add it to the ServerVariables collection like everyone else is doing. Ideally we should have a CurrentUser or LoggedInUser object that I could use to get this data from. This would power the avatar, login info and all that as well as store the locale. I'd like to see more formalisation of data like this for ease of use in the front-end of the various bits of the back office data.
There is much more to discus on this so I've created an Umbraco issue (link is at the bottom so you don't get distracted) so our discussions don't get lost on twitter. I'm also keep to chat this through with anyone at Code Garden in an Open Space or at the bar :)
We should use [insert framework here]!
No we shouldn't. Right now is not the time to pick a framework. Right now we should be discussing what we want the back office to do for us in 2 years time and then see what fits. Anything could happen in that time frame. Even me mentioning WebPack above was a bit naughty, it makes it sound like its a given but it was only for example. Just because its hot now doesn't mean it will still be at the time when we break ground on a rebuild in 18-24 months time.
The danger of picking or discussing a framework now is it can cause Confirmation Bias in others or even the Halo Effect which can cause us to miss out on a true best fit for all. By naming tech now we are in danger of framing the discussion about the tech rather than what it is we want the back office to do for us regardless of tech. Let's instead talk about the changes we would like to see, the gains we would want to have, the friction we would like to ease and the legitimate worries that we would have to tackle to ensure current users can adopt any future change, new users can pick it up quickly and give it a strong foundation to take Umbraco forward for another 5 years.
Its joked that frame works pop up all too often but in reality they are quite rare beast. Yes you get some pet projects or side projects cropping up but there are a lot less than you might think that take hold and stick around. AngularJS has done the Umbraco project amazingly well and will continue to do so for another 18-24 months with ease while hopefully we work towards a more up to date solution however now is the time to be thinking about succession. Now is the time we can talk about as a community how we might want that new front-end setup to do, look and feel like, effectively so we have a job description before we start interviewing candidates for the roll.
I'm very excited that this is a discussion we are having now, it once again shows just how far Umbraco has matured and how rock solid it is that we can plan for features 2 years down the road. Bring it on.
To comment on this post and share your ideas please head over to the Umbraco Issue Tracker, see you there.