ALU

Featured Article

Rust 2018 is here… but what is it?

This post was written in collaboration with the Rust Team (the “we” in this article). You can also read their announcement on the Rust blog.

Starting today, the Rust 2018 edition is in its first release. With this edition, we’ve focused on productivity… on making Rust developers as productive as they can be.

A timeline showing the different channels: beta, Rust 2018, and Rust 2015, with features flowing from beta to the other two. The timeline is surrounded by icons for tooling and for 4 domains: WebAssembly, embedded, networking, and CLI. A red circle surrounds everything except for Rust 2015 and is labeled with Developer Productivity.

But beyond that, it can be hard to explain exactly what Rust 2018 is.

Some people think of it as a new version of the language, which it is… kind of, but not really. I say “not really” because if this is a new version, it doesn’t work like versioning does in other languages.

In most other languages, when a new version of the language comes out, any new features are added to that new version. The previous version doesn’t get new features.

Rust editions are different. This is because of the way the language is evolving. Almost all of the new features are 100% compatible with Rust as it is. They don’t require any breaking changes. That means there’s no reason to limit them to Rust 2018 code. New versions of the compiler will continue to support “Rust 2015 mode”, which is what you get by default.

But sometimes to advance the language, you need to add things like new syntax. And this new syntax can break things in existing code bases.

An example of this is the async/await feature. Rust initially didn’t have the concepts of async and await. But it turns out that these primitives are really helpful. They make it easier to write code that is asynchronous without the code getting unwieldy.

To make it possible to add this feature, we need to add both async and await as keywords. But we also have to be careful that we’re not making old code invalid… code that might’ve used the words async or await as variable names.

So we’re adding the keywords as part of Rust 2018. Even though the feature hasn’t landed yet, the keywords are now reserved. All of the breaking changes needed for the next three years of development (like adding new keywords) are being made in one go, in Rust 1.31.

Timeline with a line connecting Rust 2015 to the start of Rust 2018 at release 1.31.

Even though there are breaking changes in Rust 2018, that doesn’t mean your code will break. Your code will continue compiling even if it has async or await as a variable name. Unless you tell it otherwise, the compiler assumes you want it to compile your code the same way that it has been up to this point.

But as soon as you want to use one of these new, breaking features, you can opt in to Rust 2018 mode. You just run cargo fix, which will tell you if you need to update your code to use the new features. It will also mostly automate the process of making the changes. Then you can add edition=2018 to your Cargo.toml to opt in and use the new features.

This edition specifier in Cargo.toml doesn’t apply to your whole project… it doesn’t apply to your dependencies. It’s scoped to just the one crate. This means you’ll be able to have crate graphs that have Rust 2015 and Rust 2018 interspersed.

Because of this, even once Rust 2018 is out there, it’s mostly going to look the same as Rust 2015. Most changes will land in both Rust 2018 and Rust 2015. Only the handful of features that require breaking changes won’t pass through. 

Rust 2018 isn’t just about changes to the core language, though. In fact, far from it.

Rust 2018 is a push to make Rust developers more productive. Many productivity wins come from things outside of the core language… things like tooling. They also come from focusing on specific use cases and figuring out how Rust can be the most productive language for those use cases.

So you could think of Rust 2018 as the specifier in Cargo.toml that you use to enable the handful of features that require breaking changes…

Timeline with arrows pointing to the couple of Rust 2018 features that aren't passing through to Rust 2015.

Or you can think about it as a moment in time, where Rust becomes one of the most productive languages you can use in many cases — whenever you need performance, light footprint, or high reliability.

In our minds, it’s the second. So let’s look at all that happened outside of the core language. Then we can dive into the core language itself.

Rust for specific use cases

A programming language can’t be productive by itself, in the abstract. It’s productive when put to some use. Because of this, the team knew we didn’t just need to make Rust as a language or Rust tooling better. We also needed to make it easier to use Rust in particular domains.

In some cases, this meant creating a whole new set of tools for a whole new ecosystem.

In other cases, it meant polishing what was already in the ecosystem and documenting it well so that it’s easy to get up and running.

The Rust team formed working groups focused on four domains:

  • WebAssembly
  • Embedded applications
  • Networking
  • Command line tools

WebAssembly

For WebAssembly, the working group needed to create a whole new suite of tools.

Just last year, WebAssembly made it possible to compile languages like Rust to run on the web. Since then, Rust has quickly become the best language for integrating with existing web applications.

Rust logo and JS logo with a heart in between

Rust is a good fit for web development for two reasons:

  1.  Cargo’s crates ecosystem works in the same way that most web app developers are used to. You pull together a bunch of small modules to form a larger application. This means that it’s easy to use Rust just where you need it.
  2. Rust has a light footprint and doesn’t require a runtime. This means that you don’t need to ship down a bunch of code. If you have a tiny module doing lots of heavy computational work, you can introduce a few lines of Rust just to make that run faster.

With the web-sys and js-sys crates, it’s easy to call web APIs like fetch or appendChild from Rust code. And wasm-bindgen makes it easy support high-level data types that WebAssembly doesn’t natively support.

Once you’ve coded up your Rust WebAssembly module, there are tools to make it easy to plug it into the rest of your web application. You can use wasm-pack to run these tools automatically, and push your new module up to npm if you want.

Check out the Rust and WebAssembly book to try it yourself.

What’s next?

Now that Rust 2018 has shipped, the working group is figuring out where to take things next. They’ll be working with the community to determine the next areas of focus.

Embedded

For embedded development, the working group needed to make existing functionality stable.

In theory, Rust has always been a good language for embedded development. It gives embedded developers the modern day tooling that they are sorely lacking, and very convenient high-level language features. All this without sacrificing on resource usage. So Rust seemed like a great fit for embedded development.

However, in practice it was a bit of a wild ride. Necessary features weren’t in the stable channel. Plus, the standard library needed to be tweaked for use on embedded devices. That meant that people had to compile their own version of the Rust core crate (the crate which is used in every Rust app to provide Rust’s basic building blocks — intrinsics and primitives).

On the left: Someone riding a bucking microprocesser chip, saying "Whoa, Rusty!". On the right, someone riding a tame microprocessor chip saing "Good Rusty, nice and steady"

Together, these two things meant developers had to depend on the nightly version of Rust. And since there were no automated tests for micro-controller targets, nightly would often break for these targets.

To fix this, the working group needed to make sure that necessary features were in the stable channel. We also had to add tests to the CI system for micro-controller targets. This means a person adding something for a desktop component won’t break something for an embedded component.

With these changes, embedded development with Rust moves away from the bleeding edge and towards the plateau of productivity.

Check out the Embedded Rust book to try it yourself.

What’s next?

With this year’s push, Rust has really good support for ARM Cortex-M family of microprocessor cores, which are used in a lot of devices. However, there are lots of architectures used on embedded devices, and those aren’t as well supported. Rust needs to expand to have the same level of support for these other architectures.

Networking

For networking, the working group needed to build a core abstraction into the language—async/await. This way, developers can use idiomatic Rust even when the code is asynchronous.

For networking tasks, you often have to wait. For example, you may be waiting for a response to a request. If your code is synchronous, that means the work will stop—the CPU core that is running the code can’t do anything else until the request comes in. But if you code asynchronously, then the function that’s waiting for the response can go on hold while the CPU core takes care of running other functions.

Coding asynchronous Rust is possible even with Rust 2015. And there are lots of upsides to this. On the large scale, for things like server applications, it means that your code can handle many more connections per server. On the small scale, for things like embedded applications that are running on tiny, single threaded CPUs, it means you can make better use of your single thread.

But these upsides came with a major downside—you couldn’t use the borrow checker for that code, and you would have to write unidiomatic (and somewhat confusing) Rust. This is where async/await comes in. It gives the compiler the information it needs to borrow check across asynchronous function calls.

The keywords for async/await were introduced in 1.31, although they aren’t currently backed by an implementation. Much of that work is done, and you can expect the feature to be available in an upcoming release.

What’s next?

Beyond just enabling productive low-level development for networking applications, Rust could enable more productive development at a higher level.

Many servers need to do the same kinds of tasks. They need to parse URLs or work with HTTP. If these were turned into components—common abstractions that could be shared as crates—then it would be easy to plug them together to form all sorts of different servers and frameworks.

To drive the component development process, the Tide framework is providing a test bed for, and eventually example usage of, these components.

Command line tools

For command line tools, the working group needed to bring together smaller, low-level libraries into higher level abstractions, and polish some existing tools.

For some CLI scripts, you really want to use bash. For example, if you just need to call out to other shell tools and pipe data between them, then bash is best.

But Rust is a great fit for a lot of other kinds of CLI tools. For example, it’s great if you are building a complex tool like ripgrep or building a CLI tool on top of an existing library’s functionality.

Rust doesn’t require a runtime and allows you to compile to a single static binary, which makes it easy to distribute. And you get high-level abstractions that you don’t get with other languages like C and C++, so that already makes Rust CLI developers productive.

What did the working group need to make this better still? Even higher-level abstractions.

With these higher-level abstractions, it’s quick and easy to assemble a production ready CLI.

An example of one of these abstractions is the human panic library. Without this library, if your CLI code panics, it probably outputs the entire back trace. But that’s not very helpful for your end users. You could add custom error handling, but that requires effort.

If you use human panic, then the output will be automatically routed to an error dump file. What the user will see is a helpful message suggesting that they report the issue and upload the error dump file.

A cli tool with friendly output from human-panic

The working group also made it easier to get started with CLI development. For example, the confy library will automate a lot of setup for a new CLI tool. It only asks you two things:

  • What’s the name of your application?
  • What are configuration options you want to expose (which you define as a struct that can be serialized and deserialized)?

From that, confy will figure out the rest for you.

What’s next?

The working group abstracted away a lot of different tasks that are common between CLIs. But there’s still more that could be abstracted away. The working group will be making more of these high level libraries, and fixing more paper cuts as they go.

Rust tooling

Tooling icon

When you experience a language, you experience it through tools. This starts with the editor that you use. It continues through every stage of the development process, and through maintenance.

This means that a productive language depends on productive tooling.

Here are some tools (and improvements to Rust’s existing tooling) that were introduced as part of Rust 2018.

IDE support

Of course, productivity hinges on fluidly getting code from your mind to the screen quickly. IDE support is critical to this. To support IDEs, we need tools that can tell the IDE what Rust code actually means — for example, to tell the IDE what strings make sense for code completion.

In the Rust 2018 push, the community focused on the features that IDEs needed. With Rust Language Server and IntelliJ Rust, many IDEs now have fluid Rust support.

Faster compilation

With compilation, faster means more productive. So we’ve made the compiler faster.

Before, when you would compile a Rust crate, the compiler would recompile every single file in the crate. But now, with incremental compilation, the compiler is smart and only recompiles the parts that have changed. This, along with other optimizations, has made the Rust compiler much faster.

rustfmt

Productivity also means not having to fix style nits (and never having to argue over formatting rules).

The rustfmt tool helps with this by automatically reformatting your code using a default code style (which the community reached consensus on). Using rustfmt ensures that all of your Rust code conforms to the same style, like clang format does for C++ and Prettier does for JavaScript.

Clippy

Sometimes it’s nice to have an experienced advisor by your side… giving you tips on best practices as you code. That’s what Clippy does —it reviews your code as you go and tells you how to make that code more idiomatic.

rustfix

But if you have an older code base that uses outmoded idioms, then just getting tips and correcting the code yourself can be tedious. You just want someone to go into your code base an make the corrections.

For these cases, rustfix will automate the process. It will both apply lints from tools like Clippy and update older code to match Rust 2018 idioms.

Changes to Rust itself

These changes in the ecosystem have brought lots of productivity wins. But some productivity issues could only be fixed with changes to the language itself.

As I talked about in the intro, most of the language changes are completely compatible with existing Rust code. These changes are all part of Rust 2018. But because they don’t break any code, they also work in any Rust code… even if that code doesn’t use Rust 2018.

Let’s look at a few of the big language features that were added to all editions. Then we can look at the small list of Rust 2018-specific features.

New language features for all editions

Here’s a small sample of the big new language features that are (or will be) in all language editions.

More precise borrow checking (e.g. Non-Lexical Lifetimes)

One big selling point for Rust is the borrow checker. The borrow checker helps ensure that your code is memory safe. But it has also been a pain point for new Rust developers.

Part of that is learning new concepts. But there was another big part… the borrow checker would sometimes reject code that seemed like it should work, even to those who understood the concepts.

borrow checker telling a programmer that they can't borrow a variable because it's already borrowed

This is because the lifetime of a borrow was assumed to go all the way to the end of its scope — for example, to the end of the function that the variable is in.

This meant that even though the variable was done with the value and wouldn’t try to access it anymore, other variables were still denied access to it until the end of the function.

To fix this, we’ve made the borrow checker smarter. Now it can see when a variable is actually done using a value. If it is done, then it doesn’t block other borrowers from using the data.

borrow checker saying, Oh, now I see

While this is only available in Rust 2018 as of today, it will be available in all editions in the near future. I’ll be writing more about all of this soon.

Procedural macros on stable Rust

Macros in Rust have been around since before Rust 1.0. But with Rust 2018, we’ve made some big improvements, like introducing procedural macros.

With procedural macros, it’s kind of like you can add your own syntax to Rust.

Rust 2018 brings two kinds of procedural macros:

Function-like macros

Function-like macros allow you to have things that look like regular function calls, but that are actually run during compilation. They take in some code and spit out different code, which the compiler then inserts into the binary.

They’ve been around for a while, but what you could do with them was limited. Your macro could only take the input code and run a match statement on it. It didn’t have access to look at all of the tokens in that input code.

But with procedural macros, you get the same input that a parser gets — a token stream. This means can create much more powerful function-like macros.

Attribute-like macros

If you’re familiar with decorators in languages like JavaScript, attribute macros are pretty similar. They allow you to annotate bits of code in Rust that should be preprocessed and turned into something else.

The derive macro does exactly this kind of thing. When you put derive above a struct, the compiler will take that struct in (after it has been parsed as a list of tokens) and fiddle with it. Specifically, it will add a basic implementation of functions from a trait.

More ergonomic borrowing in matching

This change is pretty straight-forward.

Before, if you wanted to borrow something and tried to match on it, you had to add some weird looking syntax:

Old version of the code with &Some(ref s) next to new version with Some(s)

But now, you don’t need the &Some(ref s) anymore. You can just write Some(s), and Rust will figure it out from there.

New features specific to Rust 2018

The smallest part of Rust 2018 are the features specific to it. Here are the small handful of changes that using the Rust 2018 edition unlocks.

Keywords

There are a few keywords that have been added to Rust 2018.

  • try keyword
  • async/await keyword

These features haven’t been fully implemented yet, but the keywords are being added in Rust 1.31. This means we don’t have to introduce new keywords (which would be a breaking change) in the future, once the features behind these keywords are implemented.

The module system

One big pain point for developers learning Rust is the module system. And we could see why. It was hard to reason about how Rust would choose which module to use.

To fix this, we made a few changes to the way paths work in Rust.

For example, if you imported a crate, you could use it in a path at the top level. But if you moved any of the code to a submodule, then it wouldn’t work anymore.

// top level module
extern crate serde; // this works fine at the top level
impl serde::Serialize for MyType { ... } mod foo { // but it does *not* work in a sub-module impl serde::Serialize for OtherType { ... }
}

Another example is the prefix ::, which used to refer to either the crate root or an external crate. It could be hard to tell which.

We’ve made this more explicit. Now, if you want to refer to the crate root, you use the prefix crate:: instead. And this is just one of the path clarity improvements we’ve made.

If you have existing Rust code and you want it to use Rust 2018, you’ll very likely need to update it for these new module paths. But that doesn’t mean that you’ll need to manually update your code. Run cargo fix before you add the edition specifier to Cargo.toml and rustfix will make all the changes for you.

Learn More

Learn all about this edition in the Rust 2018 edition guide.

The post Rust 2018 is here… but what is it? appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Decentralizing Social Interactions with ActivityPub

In the Dweb series, we are covering projects that explore what is possible when the web becomes decentralized or distributed. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source and open for participation, and they share Mozilla’s mission to keep the web open and accessible for all.

Social websites first got us talking and sharing with our friends online, then turned into echo-chamber content silos, and finally emerged in their mature state as surveillance capitalist juggernauts, powered by the effluent of our daily lives online. The tail isn’t just wagging the dog, it’s strangling it. However, there just might be a way forward that puts users back in the driver seat: A new set of specifications for decentralizing social activity on the web. Today you’ll get a helping hand into that world from from Darius Kazemi, renowned bot-smith and Mozilla Fellow.

– Dietrich Ayala

ActivityPub logo

Introducing ActivityPub

Hi, I’m Darius Kazemi. I’m a Mozilla Fellow and decentralized web enthusiast. In the last year I’ve become really excited about ActivityPub, a W3C standard protocol that describes ways for different social network sites (loosely defined) to talk to and interact with one another. You might remember the heyday of RSS, when a user could subscribe to almost any content feed in the world from any number of independently developed feed readers. ActivityPub aims to do for social network interactions what RSS did for content.

Architecture

ActivityPub enables a decentralized social web, where a network of servers interact with each other on behalf of individual users/clients, very much like email operates at a macro level. On an ActivityPub compliant server, individual user accounts have an inbox and an outbox that accept HTTP GET and POST requests via API endpoints. They usually live somewhere like https://social.example/users/dariusk/inbox and https://social.example/users/dariusk/outbox, but they can really be anywhere as long as they are at a valid URI. Individual users are represented by an Actor object, which is just a JSON-LD file that gives information like username and where the inbox and outbox are located so you can talk to the Actor. Every message sent on behalf of an Actor has the link to the Actor’s JSON-LD file so anyone receiving the message can look up all the relevant information and start interacting with them.

A simple server to send ActivityPub messages

One of the most popular social network sites that uses ActivityPub is Mastodon, an open source community-owned and ad-free alternative to social network services like Twitter. But Mastodon is a huge, complex project and not the best introduction to the ActivityPub spec as a developer. So I started with a tutorial written by Eugen Rochko (the principal developer of Mastodon) and created a partial reference implementation written in Node.js and Express.js called the Express ActivityPub server.

The purpose of the software is to serve as the simplest possible starting point for developers who want to build their own ActivityPub applications. I picked what seemed to me like the smallest useful subset of ActivityPub features: the ability to publish an ActivityPub-compliant feed of posts that any ActivityPub user can subscribe to. Specifically, this is useful for non-interactive bots that publish feeds of information.

To get started with Express ActivityPub server in a local development environment, install

git clone https://github.com/dariusk/express-activitypub.git
cd express-activitypub/
npm i

In order to truly test the server it needs to be associated with a valid, https-enabled domain or subdomain. For local testing I like to use ngrok to test things out on one of the temporary domains that they provide. First, install ngrok using their instructions (you have to sign in but there is a free tier that is sufficient for local debugging). Next run:

ngrok http 3000

This will show a screen on your console that includes a domain like abcdef.ngrok.io. Make sure to note that down, as it will serve as your temporary test domain as long as ngrok is running. Keep this running in its own terminal session while you do everything else.

Then go to your config.json in the express-activitypub directory and update the DOMAIN field to whatever abcdef.ngrok.io domain that ngrok gave you (don’t include the http://), and update USER to some username and PASS to some password. These will be the administrative password required for creating new users on the server. When testing locally via ngrok you don’t need to specify the PRIVKEY_PATH or CERT_PATH.

Next run your server:

node index.js

Go to https://abcdef.ngrok.io/admin (again, replace the subdomain) and you should see an admin page. You can create an account here by giving it a name and then entering the admin user/pass when prompted. Try making an account called “test” — it will give you a long API key that you should save somewhere. Then open an ActivityPub client like Mastodon’s web interface and try following @test@abcdef.ngrok.io. It should find the account and let you follow!

Back on the admin page, you’ll notice another section called “Send message to followers” — fill this in with “test” as the username, the hex key you just noted down as the password, and then enter a message. It should look like this:

Screenshot of form

Screenshot of form

Hit “Send Message” and then check your ActivityPub client. In the home timeline you should see your message from your account, like so:

Post in Mastodon mobile web view

Post in Mastodon mobile web view

And that’s it! It’s not incredibly useful on its own but you can fork the repository and use it as a starting point to build your own services. For example, I used it as the foundation of an RSS-to-ActivityPub conversion service that I wrote (source code here). There are of course other services that could be built using this. For example, imagine a replacement for something like MailChimp where you can subscribe to updates for your favorite band, but instead of getting an email, everyone who follows an ActivityPub account will get a direct message with album release info. Also it’s worth browsing the predefined Activity Streams Vocabulary to see what kind of events the spec supports by default.

Learn More

There is a whole lot more to ActivityPub than what I’ve laid out here, and unfortunately there aren’t a lot of learning resources beyond the specs themselves and conversations on various issue trackers.

If you’d like to know more about ActivityPub, you can of course read the ActivityPub spec. It’s important to know that while the ActivityPub spec lays out how messages are sent and received, the different types of messages are specified in the Activity Streams 2.0 spec, and the actual formatting of the messages that are sent is specified in the Activity Streams Vocabulary spec. It’s important to familiarize yourself with all three.

You can join the Social Web Incubator Community Group, a W3C Community Group, to participate in discussions around ActivityPub and other social web tech standards. They have monthly meetings that you can dial into that are listed on the wiki page.

And of course if you’re on an ActivityPub social network service like Mastodon or Pleroma, the #ActivityPub hashtag there is always active.

The post Decentralizing Social Interactions with ActivityPub appeared first on Mozilla Hacks – the Web developer blog.

Original Link

The Power of Web Components

A video player with a corner folded over, revealing the code that powers it.

Background

Ever since the first animated DHTML cursor trails and “Site of the Week” badges graced the web, re-usable code has been a temptation for web developers. And ever since those heady days, integrating third-party UI into your site has been, well, a semi-brittle headache.

Using other people’s clever code has required buckets of boilerplate JavaScript or CSS conflicts involving the dreaded !important. Things are a bit better in the world of React and other modern frameworks, but it’s a bit of a tall order to require the overhead of a full framework just to re-use a widget. HTML5 introduced a few new elements like <video> and <input type="date">, which added some much-needed common UI widgets to the web platform. But adding new standard elements for every sufficiently common web UI pattern isn’t a sustainable option.

In response, a handful of web standards were drafted. Each standard has some independent utility, but when used together, they enable something that was previously impossible to do natively, and tremendously difficult to fake: the capability to create user-defined HTML elements that can go in all the same places as traditional HTML. These elements can even hide their inner complexity from the site where they are used, much like a rich form control or video player.

The standards evolve

As a group, the standards are known as Web Components. In the year 2018 it’s easy to think of Web Components as old news. Indeed, early versions of the standards have been around in one form or another in Chrome since 2014, and polyfills have been clumsily filling the gaps in other browsers.

After some quality time in the standards committees, the Web Components standards were refined from their early form, now called version 0, to a more mature version 1 that is seeing implementation across all the major browsers. Firefox 63 added support for two of the tent pole standards, Custom Elements and Shadow DOM, so I figured it’s time to take a closer look at how you can play HTML inventor!

Given that Web Components have been around for a while, there are lots of other resources available. This article is meant as a primer, introducing a range of new capabilities and resources. If you’d like to go deeper (and you definitely should), you’d do well to read more about Web Components on MDN Web Docs and the Google Developers site.

Defining your own working HTML elements requires new powers the browser didn’t previously give developers. I’ll be calling out these previously-impossible bits in each section, as well as what other newer web technologies they draw upon.

The <template> element: a refresher

This first element isn’t quite as new as the others, as the need it addresses predates the Web Components effort. Sometimes you just need to store some HTML. Maybe it’s some markup you’ll need to duplicate multiple times, maybe it’s some UI you don’t need to create quite yet. The <template> element takes HTML and parses it without adding the parsed DOM to the current document.

<template> <h1>This won't display!</h1> <script>alert("this won't alert!");</script>
</template>

Where does that parsed HTML go, if not to the document? It’s added to a “document fragment”, which is best understood as a thin wrapper that contains a portion of an HTML document. Document fragments dissolve when appended to other DOM, so they’re useful for holding a bunch of elements you want later, in a container you don’t need to keep.

“Well okay, now I have some DOM in a dissolving container, how do I use it when I need it?”

You could simply insert the template’s document fragment into the current document:

let template = document.querySelector('template');
document.body.appendChild(template.content);

This works just fine, except you just dissolved the document fragment! If you run the above code twice you’ll get an error, as the second time template.content is gone. Instead, we want to make a copy of the fragment prior to inserting it:

document.body.appendChild(template.content.cloneNode(true));

The cloneNode method does what it sounds like, and it takes an argument specifying whether to copy just the node itself or include all its children.

The template tag is ideal for any situation where you need to repeat an HTML structure. It particularly comes in handy when defining the inner structure of a component, and thus <template> is inducted into the Web Components club.

New Powers:

  • An element that holds HTML but doesn’t add it to the current document.

Review Topics:

Custom Elements

Custom Elements is the poster child for the Web Components standards. It does what it says on the tin – allowing developers to define their own custom HTML elements. Making this possible and pleasant builds fairly heavily on top of ES6’s class syntax, where the v0 syntax was much more cumbersome. If you’re familiar with classes in JavaScript or other languages, you can define classes that inherit from or “extend” other classes:

class MyClass extends BaseClass {
// class definition goes here
}

Well, what if we were to try this?

class MyElement extends HTMLElement {}

Until recently that would have been an error. Browsers didn’t allow the built-in HTMLElement class or its subclasses to be extended. Custom Elements unlocks this restriction.

The browser knows that a <p> tag maps to the HTMLParagraphElement class, but how does it know what tag to map to a custom element class? In addition to extending built-in classes, there’s now a “Custom Element Registry” for declaring this mapping:

customElements.define('my-element', MyElement);

Now every <my-element> on the page is associated with a new instance of MyElement. The constructor for MyElement will be run whenever the browser parses a <my-element> tag.

What’s with that dash in the tag name? Well, the standards bodies want the freedom to create new HTML tags in the future, and that means that developers can’t just go creating an <h7> or <vr> tag. To avoid future conflicts, all custom elements must contain a dash, and standards bodies promise to never make a new HTML tag containing a dash. Collision avoided!

In addition to having your constructor called whenever your custom element is created, there are a number of additional “lifecycle” methods that are called on a custom element at various moments:

  • connectedCallback is called when an element is appended to a document. This can happen more than once, e.g. if the element is moved or removed and re-added.
  • disconnectedCallback is the counterpart to connectedCallback.
  • attributeChangeCallback fires when attributes from a whitelist are modified on the element.

A slightly richer example looks like this:

class GreetingElement extends HTMLElement { constructor() { super(); this.name = 'Stranger'; } connectedCallback() { this.addEventListener('click', e => alert(`Hello, ${this.name}!`)); } attributeChangedCallback(attrName, oldValue, newValue) { if (attrName === 'name') { if (newValue) { this.name = newValue; } else { this.name = 'Stranger'; } } }
}
GreetingElement.observedAttributes = ['name'];
customElements.define('hey-there', GreetingElement);

Using this on a page will look like the following:

<hey-there>Greeting</hey-there>
<hey-there name="Potch">Personalized Greeting</hey-there>

But what if you want to extend an existing HTML element? You definitely can and should, but using them within markup looks fairly different. Let’s say we want our greeting to be a button:

class GreetingElement extends HTMLButtonElement

We’ll also need to tell the registry we’re extending an existing tag:

customElements.define('hey-there', GreetingElement, { extends: 'button' });

Because we’re extending an existing tag, we actually use the existing tag instead of our custom tag name. We use the new special is attribute to tell the browser what kind of button we’re using:

<button is="hey-there" name="World">Howdy</button>

It may seem a bit clunky at first, but assistive technologies and other scripts wouldn’t know our custom element is a kind of button without this special markup.

From here, all the classic web widget techniques apply. We can set up a bunch of event handlers, add custom styling, and even stamp out an inner structure using <template>. People can use your custom element alongside their own code, via HTML templating, DOM calls, or even new-fangled frameworks, several of which support custom tag names in their virtual DOM implementations. Because the interface is the standard DOM interface, Custom Elements allows for truly portable widgets.

New Powers

  • The ability to extend the built-in ‘HTMLElement’ class and its subclasses
  • A custom element registry, available via customElements.define()
  • Special lifecycle callbacks for detecting element creation, insertion to the DOM, attribute changes, and more.

Review Topics

Shadow DOM

We’ve made our friendly custom element, we’ve even thrown on some snazzy styling. We want to use it on all our sites, and share the code with others so they can use it on theirs. How do we prevent the nightmare of conflicts when our customized <button> element runs face-first into the CSS of other sites? Shadow DOM provides a solution.

The Shadow DOM standard introduces the concept of a shadow root. Superficially, a shadow root has standard DOM methods, and can be appended to as if it was any other DOM node. Shadow roots shine in that their contents don’t appear to the document that contains their parent node:

// attachShadow creates a shadow root.
let shadow = div.attachShadow({ mode: 'open' });
let inner = document.createElement('b');
inner.appendChild(document.createTextNode('Hiding in the shadows'));
// shadow root supports the normal appendChild method.
shadow.appendChild(inner);
div.querySelector('b'); // empty

In the above example, the <div> “contains” the <b> and the <b> is rendered to the page, but the traditional DOM methods can’t see it. Not only that, but the styles of the containing page can’t see it either. This means that styles outside of a shadow root can’t get in, and styles inside the shadow root don’t leak out. This boundary is not meant to be a security feature, as another script on the page could detect the shadow root’s creation, and if you have a reference to a shadow root, you can query it directly for its contents.

The contents of a shadow root are styled by adding a <style> (or <link>) to the root:

let style = document.createElement('style');
style.innerText = 'b { font-weight: bolder; color: red; }';
shadowRoot.appendChild(style);
let inner = document.createElement('b');
inner.innerHTML = "I'm bolder in the shadows";
shadowRoot.appendChild(inner);

Whew, we could really use a <template> right about now! Either way, the <b> will be affected by the stylesheet in the root, but any outer styles matching a <b> tag will not.

What if a custom element has non-shadow content? We can make them play nicely together using a new special element called <slot>:

<template> Hello, <slot></slot>!
</template>

If that template is attached to a shadow root, then the following markup:

<hey-there>World</hey-there>

Will render as:

Hello, World!

This ability to composite shadow roots with non-shadow content allows you to make rich custom elements with complex inner structures that look simple to the outer environment. Slots are more powerful than I’ve shown here, with multiple slots and named slots and special CSS pseudo-classes to target slotted content. You’ll have to read more!

New Powers:

  • A quasi-obscured DOM structure called a “shadow root”
  • DOM APIs for creating and accessing shadow roots
  • Scoped styles within shadow roots
  • New CSS pseudo-classes for working with shadow roots and scoped styles
  • The <slot> element

Putting it all together

Let’s make a fancy button! We’ll be creative and call the element <fancy-button>. What makes it fancy? It will have a custom style, and it will also allow us to supply an icon and make that look snazzy as well. We’d like our button’s styles to stay fancy no matter what site you use them on, so we’re going to encapsulate the styles in a shadow root.

You can see the completed custom element in the interactive example below. Be sure to take a look at both the JS definition of the custom element and the HTML <template> for the style and structure of the element.

Conclusion

The standards that make up Web Components are built on the philosophy that by providing multiple low-level capabilities, people will combine them in ways that nobody expected at the time the specs were written. Custom Elements have already been used to make it easier to build VR content on the web, spawned multiple UI toolkits, and much more. Despite the long standardization process, the emerging promise of Web Components puts more power in the hand of creators. Now that the technology is available in browsers, the future of Web Components is in your hands. What will you build?

The post The Power of Web Components appeared first on Mozilla Hacks – the Web developer blog.

Original Link

New & Experimental Web Design Tools: Feedback Requested

The state of design tools in Firefox

Screenshot of Grid Inspector in action
A year ago, the Firefox DevTools team formed a subgroup to focus on new tools for working in web design, CSS, and HTML. Motivated by the success of the Grid Inspector, and with help from the Developer Outreach, Gecko Platform, and Accessibility teams, we launched the Variable Fonts Editor and the Shape Path Editor, added an Accessibility Inspector, and revamped our Responsive Design Mode.

Our goal: To build empowering new tools that integrate smartly with your modern web design workflow.

Screenshot of Flexbox Inspector

We’re currently hard at work on a comprehensive Flexbox Inspector as well as CSS change-tracking. Early versions of each of these can be tried out in Firefox Nightly. (The Changes panel is hidden behind a flag in about:config: devtools.inspector.changes.enabled)

Please share your input

We’re just getting started, and now we want to learn more about you. Tell us about your biggest CSS and web design issues in the first-ever Design Tools survey! We want to hear from both web developers and designers, and not just Firefox users—Chrome, Safari, Edge, and IE users are greatly encouraged to submit your thoughts!

Survey Banner: We need your input! Tell us about your day-to-day work in web design and CSS (image of otter with wrench)

In early 2019, we’ll post an update with the results in order to share our data with the greater community and continue our experiment in open design.

Get in touch

Feel free to chat with the Firefox DevTools team at any time via @FirefoxDevTools on Twitter or Discourse. You can also learn more about getting involved.

The post New & Experimental Web Design Tools: Feedback Requested appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Private by Design: How we built Firefox Sync

What is Firefox Sync and why would you use it

That shopping rabbit hole you started on your laptop this morning? Pick up where you left off on your phone tonight. That dinner recipe you discovered at lunchtime? Open it on your kitchen tablet, instantly. Connect your personal devices, securely. – Firefox Sync

Firefox Sync lets you share your bookmarks, browsing history, passwords and other browser data between different devices, and send tabs from one device to another. It’s a feature that millions of our users take advantage of to streamline their lives and how they interact with the web.

But on an Internet where sharing your data with a provider is the norm, we think it’s important to highlight the privacy aspects of Firefox Sync.

Firefox Sync by default protects all your synced data so Mozilla can’t read it. We built Sync this way because we put user privacy first. In this post, we take a closer look at some of the technical design choices we made and why.

When building a browser and implementing a sync service, we think it’s important to look at what one might call ‘Total Cost of Ownership’.  Not just what users get from a feature, but what they give up in exchange for ease of use.

We believe that by making the right choices to protect your privacy, we’ve also lowered the barrier to trying out Sync. When you sign up and choose a strong passphrase, your data is protected from both attackers and from Mozilla, so you can try out Sync without worry. Give it a shot, it’s right up there in the menu bar!

Sign in to Sync Button in the Firefox Menu

Why Firefox Sync is safe

Encryption allows one to protect data so that it is entirely unreadable without the key used to encrypt it. The math behind encryption is strong, has been tested for decades, and every government in the world uses it to protect its most valuable secrets.

The hard part of encryption is that key. What key do you encrypt with, where does it come from, where is it stored, and how does it move between places? Lots of cloud providers claim they encrypt your data, and they do. But they also have the key! While the encryption is not meaningless, it is a small measure, and does not protect the data against the most concerning threats.

The encryption key is the essential element. The service provider must never receive it – even temporarily – and must never know it. When you sign into your Firefox Account, you enter a username and passphrase, which are sent to the server. How is it that we can claim to never know your encryption key if that’s all you ever provide us?  The difference is in how we handle your passphrase.

A typical login flow for an internet service is to send your username and passphrase up to the server, where they hash it, compare it to a stored hash, and if correct, the server sends you your data. (Hashing refers to the activity of converting passwords into unreadable strings of characters impossible to revert.)

Typical Web Provider Login Flow

The crux of the difference in how we designed Firefox Accounts, and Firefox Sync (our underlying syncing service), is that you never send us your passphrase. We transform your passphrase on your computer into two different, unrelated values. With one value, you cannot derive the other0. We send an authentication token, derived from your passphrase, to the server as the password-equivalent. And the encryption key derived from your passphrase never leaves your computer.

Firefox Sync Login Flow

Interested in the technical details?  We use 1000 rounds of PBKDF2 to derive your passphrase into the authentication token1. On the server, we additionally hash this token with scrypt (parameters N=65536, r=8, p=1)2 to make sure our database of authentication tokens is even more difficult to crack.

We derive your passphrase into an encryption key using the same 1000 rounds of PBKDF2. It is domain-separated from your authentication token by using HKDF with separate info values. We use this key to unwrap an encryption key (which you generated during setup and which we never see unwrapped), and that encryption key is used to protect your data.  We use the key to encrypt your data using AES-256 in CBC mode, protected with an HMAC3.

This cryptographic design is solid – but the constants need to be updated. One thousand rounds of PBKDF can be improved, and we intend to do so in the future (Bug 1320222). This token is only ever sent over a HTTPS connection (with preloaded HPKP pins) and is not stored, so when we initially developed this and needed to support low-power, low-resources devices, a trade-off was made. AES-CBC + HMAC is acceptable – it would be nice to upgrade this to an authenticated mode sometime in the future.

Other approaches

This isn’t the only approach to building a browser sync feature. There are at least three other options:

Option 1: Share your data with the browser maker

In this approach, the browser maker is able to read your data, and use it to provide services to you. For example,  when you sync your browser history in Chrome it will automatically go into your Web & App Activity unless you’ve changed the default settings. As Google Chrome Help explains, “Your activity may be used to personalize your experience on other Google products, like Search or ads. For example, you may see a news story recommended in your feed based on your Chrome history.”4

Option 2: Use a separate password for sign-in and encryption

We developed Firefox Sync to be as easy to use as possible, so we designed it from the ground up to derive an authentication token and an encryption key – and we never see the passphrase or the encryption key. One cannot safely derive an encryption key from a passphrase if the passphrase is sent to the server.

One could, however, add a second passphrase that is never sent to the server, and encrypt the data using that. Chrome provides this as a non-default option5. You can sign in to sync with your Google Account credentials; but you choose a separate passphrase to encrypt your data. It’s imperative you choose a separate passphrase though.

All-in-all, we don’t care for the design that requires a second passphrase. This approach is confusing to users. It’s very easy to choose the same (or similar) passphrase and negate the security of the design. It’s hard to determine which is more confusing: to require a second passphrase or to make it optional! Making it optional means it will be used very rarely.  We don’t believe users should have to opt-in to privacy.

Option 3: Manual key synchronization

The key (pun intended) to auditing a cryptographic design is to ask about the key: “Where does it come from? Where does it go?” With the Firefox Sync design, you enter a passphrase of your choosing and it is used to derive an encryption key that never leaves your computer.

Another option for Sync is to remove user choice, and provide a passphrase for you (that never leaves your computer). This passphrase would be secure and unguessable – which is an advantage, but it would be near-impossible to remember – which is a disadvantage.

When you want to add a new device to sync to, you’d need your existing device nearby in order to manually read and type the passphrase into the new device. (You could also scan a QR code if your new device has a camera).

Other Browsers

Overall, Sync works the way it does because we feel it’s the best design choice. Options 1 and 2 don’t provide thorough user privacy protections by default. Option 3 results in lower user adoption and thus reduces the number of people we can help (more on this below).

As noted above, Chrome implements Option 1 by default, which means unless you change the settings before you enable sync, Google will see all of your browsing history and other data, and use it to market services to you. Chrome also implements Option 2 as an opt-in feature.

Both Opera and Vivaldi follow Chrome’s lead, implementing Option 1 by default and Option 2 as an opt-in feature.

Brave, also a privacy-focused browser, has implemented Option 3. And, in fact, Firefox also implemented a form of Option 3 in its original Sync Protocol, but we changed our design in April 2014 (Firefox 29) in response to user feedback6. For example, our original design (and Brave’s current design) makes it much harder to regain access to your data if you lose your device or it gets stolen. Passwords or passphrases make that experience substantially easier for the average user, and significantly increased Sync adoption by users.

Brave’s sync protocol has some interesting wrinkles7. One distinct minus is that you can’t change your passphrase, if it were to be stolen by malware. Another interesting wrinkle is that Brave does not keep track of how many or what types of devices you have. This is a nuanced security trade-off: having less information about the user is always desirable… The downside is that Brave can’t allow you to detect when a new device begins receiving your sync data or allow you to deauthorize it. We respect Brave’s decision. In Firefox, however, we have chosen to provide this additional security feature for users (at the cost of knowing more about their devices).

Conclusion

We designed Firefox Sync to protect your data – by default – so Mozilla can’t read it. We built it this way – despite trade-offs that make development and offering features more difficult – because we put user privacy first. At Mozilla, this priority is a core part of our mission to “ensure the Internet is a global public resource… where individuals can shape their own experience and are empowered, safe and independent.”


0 It is possible to use one to guess the other, but only if you choose a weak password.

1 You can find more details in the full protocol specification or by following the code starting at this point. There are a few details we have omitted to simplify this blog post, including the difference between kA and kB keys, and application-specific subkeys.

2 Server hashing code is located here.

3 The encryption code can be seen here.

4 https://support.google.com/chrome/answer/165139 Section “Use your Chrome history to personalize Google”

5 Chrome 71 says “For added security, Google Chrome will encrypt your data” and describes these two options as “Encrypt synced passwords with your Google username and password” and “Encrypt synced data with your own sync passphrase”.  Despite this wording, only the sync passphrase option protects your data from Google.

6 One of the original engineers of Sync has written two blog posts about the transition to the new sync protocol, and why we did it. If you’re interested in the usability aspects of cryptography, we highly recommend you read them to see what we learned.

7 You can read more about Brave sync on Brave’s Design page.

The post Private by Design: How we built Firefox Sync appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Into the Depths: The Technical Details Behind AV1

AV1, the next generation royalty-free video codec from the Alliance for Open Media, is making waves in the broadcasting industry.

Firefox, Coney & Mozilla's AV1 team at IBC 2018

Since AOMedia officially cemented the AV1 v1.0.0 specification earlier this year, we’ve seen increasing interest from the broadcasting industry. Starting with the NAB Show (National Association of Broadcasters) in Las Vegas earlier this year, and gaining momentum through IBC (International Broadcasting Convention) in Amsterdam, and more recently the NAB East Show in New York, AV1 keeps picking up steam. Each of these industry events attract over 100,000 media professionals. Mozilla attended these shows to demonstrate AV1 playback in Firefox, and showed that AV1 is well on its way to being broadly adopted in web browsers.

Continuing to advocate for AV1 in the broadcast space, Nathan Egge from Mozilla dives into the depths of AV1 at the Mile High Video Workshop in Denver, sponsored by Comcast.

AV1 leapfrogs the performance of VP9 and HEVC, making it a next-generation codec. The AV1 format is and will always be royalty-free with a permissive FOSS license.

The post Into the Depths: The Technical Details Behind AV1 appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Cross-language Performance Profile Exploration with speedscope

speedscope

speedscope is a fast, interactive, web-based viewer for large performance profiles (www.speedscope.app, github.com/jlfwong/speedscope). It’s inspired by the performance panel of Chrome developer tools and by Brendan Gregg’s FlameGraphs. If you’ve never heard of flamegraphs before or if you’ve heard of them but never understood how to read them, the guide “Using flamegraphs” from rbspy’s documentation is wonderful.

The goal of speedscope is to provide a 60fps way of interactively exploring large performance profiles from a variety of profilers for a variety of programming languages. It runs totally in-browser, and does not send any profiling data to any servers. Because it runs totally in-browser, it should work in Firefox and Chrome on Mac, Windows, and Linux. It can be downloaded to run offline, either from npm, or just as a totally standalone zip file.

In doing performance work across many language environments at Figma, I noticed that every community tends to create its own tools for visualizing performance issues. With speedscope, I hoped to de-duplicate those efforts. To meet this goal, speedscope supports import of profiles from a growing list of profilers:

JavaScript

Ruby

Go

Haskell

Native code

speedscope also has a stable documented file format, making it appropriate as a tool to target for visualization of totally custom profiles. This allows new profilers to support import into speedscope without needing to modify speedscope’s code at all (though contributions are welcome!). This is how I added support for visualizing rbspy profiles: rbspy#161. Firefox & Chrome both have capable profile visualizers, but the file formats they use change frequently.

Also unlike other similar tools, speedscope is designed to make it easy to host inside your own infrastructure. This allows you to integrate speedscope to view backend performance profiles with a single click. At Figma, we have a ruby backend, so I made an opinionated fork of rack-mini-profiler to do exactly this. If you support access to performance profiles across domains, you can even load them directly into https://www.speedscope.app via a #profileUrl=… hash parameter.

What can it do?

speedscope is broken down into three primary views: Time Order, Left Heavy, and Sandwich.

🕰Time Order

speedscope time order view

In the “Time Order” view (the default), call stacks are ordered left-to-right in the same order as they occurred in the input file, which is usually the chronological order they were recorded in. This view is most helpful for understanding the behavior of an application over time, e.g. “first the data is fetched from the database, then the data is prepared for serialization, then the data is serialized to JSON”.

The horizontal axis represents the “weight” of each stack (most commonly CPU time), and the vertical axis shows you the stack active at the time of the sample. If you click on one of the frames, you’ll be able to see summary statistics about it.

⬅Left Heavy

speedscope left heavy view

In the “Left Heavy” view, identical stacks are grouped together, regardless of whether they were recorded sequentially. Then, the stacks are sorted so that the heaviest stack for each parent is on the left — hence “left heavy”. This view is useful for understanding where all the time is going in situations where there are hundreds or thousands of function calls interleaved between other call stacks.

🥪 Sandwich

speedscope sandwich view

The “Sandwich” view is a table view in which you can find a list of all functions and their associated times. You can sort by self time or total time.

It’s called the “Sandwich” view because if you select one of the rows in the table, you can see flamegraphs for all the callers and callees of the selected row.

Learn more

Up-to-date information about speedscope can be found on the GitHub page: github.com/jlfwong/speedscope. If you run into problems, please let me know by filing an issue!

The post Cross-language Performance Profile Exploration with speedscope appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Dweb: Identity for the Decentralized Web with IndieAuth

In the Dweb series, we are covering projects that explore what is possible when the web becomes decentralized or distributed. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source and open for participation, and they share Mozilla’s mission to keep the web open and accessible for all.

We’ve covered a number of projects so far in this series that require foundation-level changes to the network architecture of the web. But sometimes big things can come from just changing how we use the web we have today.

Imagine if you never had to remember a password to log into a website or app ever again. IndieAuth is a simple but powerful way to manage and verify identity using the decentralization already built into the web itself. We’re happy to introduce Aaron Parecki, co-founder of the IndieWeb movement, who will show you how to set up your own independent identity on the web with IndieAuth.

– Dietrich Ayala

Introducing IndieAuth

IndieAuth is a decentralized login protocol that enables users of your software to log in to other apps.

From the user perspective, it lets you use an existing account to log in to various apps without having to create a new password everywhere.

IndieAuth builds on existing web technologies, using URLs as identifiers. This makes it broadly applicable to the web today, and it can be quickly integrated into existing websites and web platforms.

IndieAuth has been developed over several years in the IndieWeb community, a loosely connected group of people working to enable individuals to own their online presence, and was published as a W3C Note in 2018.

IndieAuth Architecture

IndieAuth is an extension to OAuth 2.0 that enables any website to become its own identity provider. It builds on OAuth 2.0, taking advantage of all the existing security considerations and best practices in the industry around authorization and authentication.

IndieAuth starts with the assumption that every identifier is a URL. Users as well as applications are identified and represented by a URL.

When a user logs in to an application, they start by entering their personal home page URL. The application fetches that URL and finds where to send the user to authenticate, then sends the user there, and can later verify that the authentication was successful. The flow diagram below walks through each step of the exchange:

Diagram showing IndieAuth work-flow, from browser to client, to user URL to endpoint

Get Started with IndieAuth

The quickest way to use your existing website as your IndieAuth identity is to let an existing service handle the protocol bits and tell apps where to find the service you’re using.

If your website is using WordPress, you can easily get started by installing the IndieAuth plugin! After you install and activate the plugin, your website will be a full-featured IndieAuth provider and you can log in to websites like https://indieweb.org right away!

To set up your website manually, you’ll need to choose an IndieAuth server such as https://indieauth.com and add a few links to your home page. Add a link to the indieauth.com authorization endpoint in an HTML <link> tag so that apps will know where to send you to log in.

<link rel="authorization_endpoint" href="https://indieauth.com/auth">

Then tell indieauth.com how to authenticate you by linking to either a GitHub account or email address.

<a href="https://github.com/username" rel="me">GitHub</a>
<a href="mailto:you@example.com" rel="me">Email</a>

Note: This last step is unique to indieauth.com and isn’t part of the IndieAuth spec. This is how indieauth.com can authenticate you without you creating a password there. It lets you switch out the mechanism you use to authenticate, for example in case you decide to stop using GitHub, without changing your identity at the site you’re logging in to.

If you don’t want to rely on any third party services at all, then you can host your own IndieAuth authorization endpoint using an existing open source solution or build your own. In any case, it’s fine to start using a service for this today, because you can always swap it out later without your identity changing.

Now you’re ready! When logging in to a website like https://indieweb.org, you’ll be asked to enter your URL, then you’ll be sent to your chosen IndieAuth server to authenticate!

Learn More

If you’d like to learn more, OAuth for the Open Web talks about more of the technical details and motivations behind the IndieAuth spec.

You can learn how to build your own IndieAuth server at the links below:

You can find the latest spec at indieauth.spec.indieweb.org. If you have any questions, feel free to drop by the #indieweb-dev channel in the IndieWeb chat, or you can find me on Twitter, or my website.

The post Dweb: Identity for the Decentralized Web with IndieAuth appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Firefox 63 – Tricks and Treats!

It’s that time of the year again- when we put on costumes and pass out goodies to all. It’s Firefox release week! Join me for a spook-tacular1 look at the latest goodies shipping this release.

Web Components, Oh My!

After a rather long gestation, I’m pleased to announce that support for modern Web Components APIs has shipped in Firefox! Expect a more thorough write-up, but let’s cover what these new APIs make possible.

Custom Elements

To put it simply, Custom Elements makes it possible to define new HTML tags outside the standard set included in the web platform. It does this by letting JS classes extend the built-in HTMLElement object, adding an API for registering new elements, and by adding special “lifecycle” methods to detect when a custom element is appended, removed, or attributes are updated:

class FancyList extends HTMLElement { constructor () { super(); this.style.fontFamily = 'cursive'; // very fancy } connectedCallback() { console.log('Make Way!'); } disconnectedCallback() { console.log('I Bid You Adieu.'); }
} customElements.define('fancy-list', FancyList);

Shadow DOM

The web has long had reusable widgets people can use when building a site. One of the most common challenges when using third-party widgets on a page is making sure that the styles of the page don’t mess up the appearance of the widget and vice-versa. This can be frustrating (to put it mildly), and leads to lots of long, overly specific CSS selectors, or the use of complex third-party tools to re-write all the styles on the page to not conflict.

Cue frustrated developer:

There has to be a better way…

Now, there is!

The Shadow DOM is not a secretive underground society of web developers, but instead a foundational web technology that lets developers create encapsulated HTML trees that aren’t affected by outside styles, can have their own styles that don’t leak out, and in fact can be made unreachable from normal DOM traversal methods (querySelector, .childNodes, etc.).

let shadow = div.attachShadow({ mode: 'open' });
let inner = document.createElement('b');
inner.appendChild(document.createTextNode('I was born in the shadows'));
shadow.appendChild(inner);
div.querySelector('b'); // empty

Custom elements and shadow roots can be used independently of one another, but they really shine when used together. For instance, imagine you have a <media-player> element with playback controls. You can put the controls in a shadow root and keep the page’s DOM clean! In fact, Both Firefox and Chrome now use Shadow DOM for the implementation of the <video> element.

Expect a deeper dive on building full-fledged components here on Hacks soon! In the meantime, you can plunge into the Web Components docs on MDN as well as see the code for a bunch of sample custom elements on GitHub.

Fonts Editor

a screenshot of the fonts panel being used to adjust a variable font

The Inspector’s Fonts panel is a handy way to see what local and web fonts are being used on a page. Already useful for debugging webfonts, in Firefox 63 the Fonts panel gains new powers! You can adjust the parameters of the font on the currently selected element, and if the current font supports Font Variations, you can view and fine-tune those paramaters as well. The syntax for adjusting variable fonts can be a little unfamiliar and it’s not otherwise possible to discover all the variations built into a font, so this tool can be a life saver.

Read all about how to use the new Fonts panel on MDN Web Docs.

Reduced motion preferences for CSS

Slick animations can give a polished and unique feel to a digital experience. However, for some people, animated effects like parallax and sliding/zooming transitions can cause vertigo and headaches. In addition, some older/less powerful devices can struggle to render animations smoothly. To respond to this, some devices and operating systems offer a “reduce motion” option. In Firefox 63, you can now detect this preference using CSS media queries and adjust/reduce your use of transitions and animations to ensure more people have a pleasant experience using your site. CSS Tricks has a great overview of both how to detect reduced motion and why you should care.

Conclusion

There is, as always, a bunch more in this release of Firefox. MDN Web Docs has the full run-down of developer-facing changes, and more highlights can be found in the official release notes. Happy Browsing!

1. not particularly spook-tacular

The post Firefox 63 – Tricks and Treats! appeared first on Mozilla Hacks – the Web developer blog.

Original Link

WebAssembly’s post-MVP future: A cartoon skill tree

/* temporary fix until theme is updated */.article dl{max-width:65ch}.article dt{font-weight:bold}.article dd{margin-top:.75rem;margin-bottom:.75rem}

People have a misconception about WebAssembly. They think that the WebAssembly that landed in browsers back in 2017—which we called the minimum viable product (or MVP) of WebAssembly—is the final version of WebAssembly.

I can understand where that misconception comes from. The WebAssembly community group is really committed to backwards compatibility. This means that the WebAssembly that you create today will continue working on browsers into the future.

But that doesn’t mean that WebAssembly is feature complete. In fact, that’s far from the case. There are many features that are coming to WebAssembly which will fundamentally alter what you can do with WebAssembly.

I think of these future features kind of like the skill tree in a videogame. We’ve fully filled in the top few of these skills, but there is still this whole skill tree below that we need to fill-in to unlock all of the applications.

Skill tree showing WebAssembly skills which will be filled in through the course of the post.

So let’s look at what’s been filled in already, and then we can see what’s yet to come.

Minimum Viable Product (MVP)

The very beginning of WebAssembly’s story starts with Emscripten, which made it possible to run C++ code on the web by transpiling it to JavaScript. This made it possible to bring large existing C++ code bases, for things like games and desktop applications, to the web.

The JS it automatically generated was still significantly slower than the comparable native code, though. But Mozilla engineers found a type system hiding inside the generated JavaScript, and figured out how to make this JavaScript run really fast. This subset of JavaScript was named asm.js.

Once the other browser vendors saw how fast asm.js was, they started adding the same optimizations to their engines, too.

But that wasn’t the end of the story. It was just the beginning. There were still things that engines could do to make this faster.

But they couldn’t do it in JavaScript itself. Instead, they needed a new language—one that was designed specifically to be compiled to. And that was WebAssembly.

So what skills were needed for the first version of WebAssembly? What did we need to get to a minimum viable product that could actually run C and C++ efficiently on the web?

Skill: Compile target

The folks working on WebAssembly knew they didn’t want to just support C and C++. They wanted many different languages to be able to compile to WebAssembly. So they needed a language-agnostic compile target.

They needed something like the assembly language that things like desktop applications are compiled to—like x86. But this assembly language wouldn’t be for an actual, physical machine. It would be for a conceptual machine.

Skill: Fast execution

That compiler target had to be designed so that it could run very fast. Otherwise, WebAssembly applications running on the web wouldn’t keep up with users’ expectations for smooth interactions and game play.

Skill: Compact

In addition to execution time, load time needed to be fast, too. Users have certain expectations about how quickly something will load. For desktop applications, that expectation is that they will load quickly because the application is already installed on your computer. For web apps, the expectation is also that load times will be fast, because web apps usually don’t have to load nearly as much code as desktop apps.

When you combine these two things, though, it gets tricky. Desktop applications are usually pretty large code bases. So if they are on the web, there’s a lot to download and compile when the user first goes to the URL.

To meet these expectations, we needed our compiler target to be compact. That way, it could go over the web quickly.

Skill: Linear memory

These languages also needed to be able to use memory differently from how JavaScript uses memory. They needed to be able to directly manage their memory—to say which bytes go together.

This is because languages like C and C++ have a low-level feature called pointers. You can have a variable that doesn’t have a value in it, but instead has the memory address of the value. So if you’re going to support pointers, the program needs to be able to write and read from particular addresses.

But you can’t have a program you downloaded from the web just accessing bytes in memory willy-nilly, using whatever addresses they want. So in order to create a secure way of giving access to memory, like a native program is used to, we had to create something that could give access to a very specific part of memory and nothing else.

To do this, WebAssembly uses a linear memory model. This is implemented using TypedArrays. It’s basically just like a JavaScript array, except this array only contains bytes of memory. When you access data in it, you just use array indexes, which you can treat as though they were memory addresses. This means you can pretend this array is C++ memory.

Achievement unlocked

So with all of these skills in place, people could run desktop applications and games in your browser as if they were running natively on their computer.

And that was pretty much the skill set that WebAssembly had when it was released as an MVP. It was truly an MVP—a minimum viable product.

This allowed certain kinds of applications to work, but there were still a whole host of others to unlock.

Heavy-weight Desktop Applications

The next achievement to unlock is heavier weight desktop applications.

Can you imagine if something like Photoshop were running in your browser? If you could instantaneously load it on any device like you do with Gmail?

We’ve already started seeing things like this. For example, Autodesk’s AutoCAD team has made their CAD software available the browser. And Adobe has made Lightroom available through the browser using WebAssembly.

But there are still a few features that we need to put in place to make sure that all of these applications—even the heaviest of heavy weight—can run well in the browser.

Skill: Threading

First, we need support for multithreading. Modern-day computers have multiple cores. These are basically multiple brains that can all be working at the same time on your problem. That can make things go much faster, but to make use of these cores, you need support for threading.

Skill: SIMD

Alongside threading, there’s another technique that utilizes modern hardware, and which enables you to process things in parallel.

That is SIMD: single instruction multiple data. With SIMD, it’s possible to take a chunk of memory and split up across different execution units, which are kind of like cores. Then you have the same bit of code—the same instruction—run across all of those execution units, but they each apply that instruction to their own bit of the data.

Skill: 64-bit addressing

Another hardware capability that WebAssembly needs to take full advantage of is 64-bit addressing.

Memory addresses are just numbers, so if your memory addresses are only 32 bits long, you can only have so many memory addresses—enough for 4 gigabytes of linear memory.

But with 64-bit addressing, you have 16 exabytes. Of course, you don’t have 16 exabytes of actual memory in your computer. So the maximum is subject to however much memory the system can actually give you. But this will take the artificial limitation on address space out of WebAssembly.

Skill: Streaming compilation

For these applications, we don’t just need them to run fast. We needed load times to be even faster than they already were. There are a few skills that we need specifically to improve load times.

One big step is to do streaming compilation—to compile a WebAssembly file while it’s still being downloaded. WebAssembly was designed specifically to enable easy streaming compilation. In Firefox, we actually compile it so fast—faster than it is coming in over the network— that it’s pretty much done compiling by the time you download the file. And other browsers are adding streaming, too.

Another thing that helps is having a tiered compiler.

For us in Firefox, that means having two compilers. The first one—the baseline compiler—kicks in as soon as the file starts downloading. It compiles the code really quickly so that it starts up quickly.

The code it generates is fast, but not 100% as fast as it could be. To get that extra bit of performance, we run another compiler—the optimizing compiler—on several threads in the background. This one takes longer to compile, but generates extremely fast code. Once it’s done, we swap out the baseline version with the fully optimized version.

This way, we get quick start up times with the baseline compiler, and fast execution times with the optimizing compiler.

In addition, we’re working on a new optimizing compiler called Cranelift. Cranelift is designed to compile code quickly, in parallel at a function by function level. At the same time, the code it generates gets even better performance than our current optimizing compiler.

Cranelift is in the development version of Firefox right now, but disabled by default. Once we enable it, we’ll get to the fully optimized code even quicker, and that code will run even faster.

But there’s an even better trick we can use to make it so we don’t have to compile at all most of the time…

Skill: Implicit HTTP caching

With WebAssembly, if you load the same code on two page loads, it will compile to the same machine code. It doesn’t need to change based on what data is flowing through it, like the JS JIT compiler needs to.

This means that we can store the compiled code in the HTTP cache. Then when the page is loading and goes to fetch the .wasm file, it will instead just pull out the precompiled machine code from the cache. This skips compiling completely for any page that you’ve already visited that’s in cache.

Skill: Other improvements

Many discussions are currently percolating around other ways to improve this, skipping even more work, so stay tuned for other load-time improvements.

Where are we with this?

Where are we with supporting these heavyweight applications right now?

Threading
For the threading, we have a proposal that’s pretty much done, but a key piece of that—SharedArrayBuffers—had to be turned off in browsers earlier this year.
They will be turned on again. Turning them off was just a temporary measure to reduce the impact of the Spectre security issue that was discovered in CPUs and disclosed earlier this year, but progress is being made, so stay tuned.
SIMD
SIMD is under very active development at the moment.
64-bit addressing
For wasm-64, we have a good picture of how we will add this, and it is pretty similar to how x86 or ARM got support for 64 bit addressing.
Streaming compilation
We added streaming compilation in late 2017, and other browsers are working on it too.
Tiered compilation
We added our baseline compiler in late 2017 as well, and other browsers have been adding the same kind of architecture over the past year.
Implicit HTTP caching
In Firefox, we’re getting close to landing support for implicit HTTP caching.
Other improvements
Other improvements are currently in discussion.

Even though this is all still in progress, you already see some of these heavyweight applications coming out today, because WebAssembly already gives these apps the performance that they need.

But once these features are all in place, that’s going to be another achievement unlocked, and more of these heavyweight applications will be able to come to the browser.

Small modules interoperating with JavaScript

But WebAssembly isn’t just for games and for heavyweight applications. It’s also meant for regular web development… for the kind of web development folks are used to: the small modules kind of web development.

Sometimes you have little corners of your app that do a lot of heavy processing, and in some cases, this processing can be faster with WebAssembly. We want to make it easy to port these bits to WebAssembly.

Again, this is a case where some of it’s already happening. Developers are already incorporating WebAssembly modules in places where there are tiny modules doing lots of heavy lifting.

One example is the parser in the source map library that’s used in Firefox’s DevTools and webpack. It was rewritten in Rust, compiled to WebAssembly, which made it 11x faster. And WordPress’s Gutenberg parser is on average 86x faster after doing the same kind of rewrite.

But for this kind of use to really be widespread—for people to be really comfortable doing it—we need to have a few more things in place.

Skill: Fast calls between JS and WebAssembly

First, we need fast calls between JS and WebAssembly, because if you’re integrating a small module into an existing JS system, there’s a good chance you’ll need to call between the two a lot. So you’ll need those calls to be fast.

But when WebAssembly first came out, these calls weren’t fast. This is where we get back to that whole MVP thing—the engines had the minimum support for calls between the two. They just made the calls work, they didn’t make them fast. So engines need to optimize these.

We’ve recently finished our work on this in Firefox. Now some of these calls are actually faster than non-inlined JavaScript to JavaScript calls. And others engines are also working on this.

Skill: Fast and easy data exchange

That brings us to another thing, though. When you’re calling between JavaScript and WebAssembly, you often need to pass data between them.

You need to pass values into the WebAssembly function or return a value from it. This can also be slow, and it can be difficult too.

There are a couple of reasons it’s hard. One is because, at the moment, WebAssembly only understands numbers. This means that you can’t pass more complex values, like objects, in as parameters. You need to convert that object into numbers and put it in the linear memory. Then you pass WebAssembly the location in the linear memory.

That’s kind of complicated. And it takes some time to convert the data into linear memory. So we need this to be easier and faster.

Skill: ES module integration

Another thing we need is integration with the browser’s built in ES module support. Right now, you instantiate a WebAssembly module using an imperative API. You call a function and it gives you back a module.

But that means that the WebAssembly module isn’t really part of the JS module graph. In order to use import and export like you do with JS modules, you need to have ES module integration.

Skill: Toolchain integration

Just being able to import and export doesn’t get us all the way there, though. We need a place to distribute these modules, and download them from, and tools to bundle them up.

What’s the npm for WebAssembly? Well, what about npm?

And what’s the webpack or Parcel for WebAssembly? Well, what about webpack and Parcel?

These modules shouldn’t look any different to the people who are using them, so no reason to create a separate ecosystem. We just need tools to integrate with them.

Skill: Backwards compatibility

There’s one more thing that we need to really do well in existing JS applications—support older versions of browsers, even those browsers that don’t know what WebAssembly is. We need to make sure that you don’t have to write a whole second implementation of your module in JavaScript just so that you can support IE11.

Where are we on this?

So where are we on this?

 

Fast calls between JS and WebAssembly
Calls between JS and WebAssembly are fast in Firefox now, and other browsers are also working on this.
Easy and fast data exchange
For easy and fast data exchange, there are a few proposals that will help with this.

As I mentioned before, one reason you have to use linear memory for more complex kinds of data is because WebAssembly only understands numbers. The only types it has are ints and floats.

With the reference types proposal, this will change. This proposal adds a new type that WebAssembly functions can take as arguments and return. And this type is a reference to an object from outside WebAssembly—for example, a JavaScript object.

But WebAssembly can’t operate directly on this object. To actually do things like call a method on it, it will still need to use some JavaScript glue. This means it works, but it’s slower than it needs to be.

To speed things up, there’s a proposal that we’ve been calling the host bindings proposal. It let’s a wasm module declare what glue must be applied to its imports and exports, so that the glue doesn’t need to be written in JS. By pulling glue from JS into wasm, the glue can be optimized away completely when calling builtin Web APIs.

There’s one more part of the interaction that we can make easier. And that has to do with keeping track of how long data needs to stay in memory. If you have some data in linear memory that JS needs access to, then you have to leave it there until the JS reads the data. But if you leave it in there forever, you have what’s called a memory leak. How do you know when you can delete the data? How do you know when JS is done with it? Currently, you have to manage this yourself.

Once the JS is done with the data, the JS code has to call something like a free function to free the memory. But this is tedious and error prone. To make this process easier, we’re adding WeakRefs to JavaScript. With this, you will be able to observe objects on the JS side. Then you can do cleanup on the WebAssembly side when that object is garbage collected.

So these proposals are all in flight. In the meantime, the Rust ecosystem has created tools that automate this all for you, and that polyfill the proposals that are in flight.

One tool in particular is worth mentioning, because other languages can use it too. It’s called wasm-bindgen. When it sees that your Rust code should do something like receive or return certain kinds of JS values or DOM objects, it will automatically create JavaScript glue code that does this for you, so you don’t need to think about it. And because it’s written in a language independent way, other language toolchains can adopt it.

ES module integration
For ES module integration, the proposal is pretty far along. We are starting work with the browser vendors to implement it.
Toolchain support
For toolchain support, there are tools like wasm-pack in the Rust ecosystem which automatically runs everything you need to package your code for npm. And the bundlers are also actively working on support.
Backwards compatibility
Finally, for backwards compatibility, there’s the wasm2js tool. That takes a wasm file and spits out the equivalent JS. That JS isn’t going to be fast, but at least that means it will work in older versions of browsers that don’t understand WebAssembly.

So we’re getting close to unlocking this achievement. And once we unlock it, we open the path to another two.

JS frameworks and compile-to-JS languages

One is rewriting large parts of things like JavaScript frameworks in WebAssembly.

The other is making it possible for statically-typed compile-to-js languages to compile to WebAssembly instead—for example, having languages like Scala.js, or Reason, or Elm compile to WebAssembly.

For both of these use cases, WebAssembly needs to support high-level language features.

Skill: GC

We need integration with the browser’s garbage collector for a couple of reasons.

First, let’s look at rewriting parts of JS frameworks. This could be good for a couple of reasons. For example, in React, one thing you could do is rewrite the DOM diffing algorithm in Rust, which has very ergonomic multithreading support, and parallelize that algorithm.

You could also speed things up by allocating memory differently. In the virtual DOM, instead of creating a bunch of objects that need to be garbage collected, you could used a special memory allocation scheme. For example, you could use a bump allocator scheme which has extremely cheap allocation and all-at-once deallocation. That could potentially help speed things up and reduce memory usage.

But you’d still need to interact with JS objects—things like components—from that code. You can’t just continually copy everything in and out of linear memory, because that would be difficult and inefficient.

So you need to be able to integrate with the browser’s GC so you can work with components that are managed by the JavaScript VM. Some of these JS objects need to point to data in linear memory, and sometimes the data in linear memory will need to point out to JS objects.

If this ends up creating cycles, it can mean trouble for the garbage collector. It means the garbage collector won’t be able to tell if the objects are used anymore, so they will never be collected. WebAssembly needs integration with the GC to make sure these kinds of cross-language data dependencies work.

This will also help statically-typed languages that compile to JS, like Scala.js, Reason, Kotlin or Elm. These language use JavaScript’s garbage collector when they compile to JS. Because WebAssembly can use that same GC—the one that’s built into the engine—these languages will be able to compile to WebAssembly instead and use that same garbage collector. They won’t need to change how GC works for them.

Skill: Exception handling

We also need better support for handling exceptions.

Some languages, like Rust, do without exceptions. But in other languages, like C++, JS or C#, exception handling is sometimes used extensively.

You can polyfill exception handling currently, but the polyfill makes the code run really slowly. So the default when compiling to WebAssembly is currently to compile without exception handling.

However, since JavaScript has exceptions, even if you’ve compiled your code to not use them, JS may throw one into the works. If your WebAssembly function calls a JS function that throws, then the WebAssembly module won’t be able to correctly handle the exception. So languages like Rust choose to abort in this case. We need to make this work better.

Skill: Debugging

Another thing that people working with JS and compile-to-JS languages are used to having is good debugging support. Devtools in all of the major browsers make it easy to step through JS. We need this same level of support for debugging WebAssembly in browsers.

Skill: Tail calls

And finally, for many functional languages, you need to have support for something called tail calls. I’m not going to get too into the details on this, but basically it lets you call a new function without adding a new stack frame to the stack. So for functional languages that support this, we want WebAssembly to support it too.

Where are we on this?

So where are we on this?

Garbage collection
For garbage collection, there are two proposals currently underway:

The Typed Objects proposal for JS, and the GC proposal for WebAssembly. Typed Objects will make it possible to describe an object’s fixed structure. There is an explainer for this, and the proposal will be discussed at an upcoming TC39 meeting.

The WebAssembly GC proposal will make it possible to directly access that structure. This proposal is under active development.

With both of these in place, both JS and WebAssembly know what an object looks like and can share that object and efficiently access the data stored on it. Our team actually already has a prototype of this working. However, it still will take some time for these to go through standardization so we’re probably looking at sometime next year.

Exception handling
Exception handling is still in the research and development phase, and there’s work now to see if it can take advantage of other proposals like the reference types proposal I mentioned before.
Debugging
For debugging, there is currently some support in browser devtools. For example, you can step through the text format of WebAssembly in Firefox debugger.But it’s still not ideal. We want to be able to show you where you are in your actual source code, not in the assembly. The thing that we need to do for that is figure out how source maps—or a source maps type thing—work for WebAssembly. So there’s a subgroup of the WebAssembly CG working on specifying that.
Tail calls
The tail calls proposal is also underway.

Once those are all in place, we’ll have unlocked JS frameworks and many compile-to-JS languages.

So, those are all achievements we can unlock inside the browser. But what about outside the browser?

Outside the Browser

Now, you may be confused when I talk about “outside the browser”. Because isn’t the browser what you use to view the web? And isn’t that right in the name—WebAssembly.

But the truth is the things you see in the browser—the HTML and CSS and JavaScript—are only part of what makes the web. They are the visible part—they are what you use to create a user interface—so they are the most obvious.

But there’s another really important part of the web which has properties that aren’t as visible.

That is the link. And it is a very special kind of link.

The innovation of this link is that I can link to your page without having to put it in a central registry, and without having to ask you or even know who you are. I can just put that link there.

It’s this ease of linking, without any oversight or approval bottlenecks, that enabled our web. That’s what enabled us to form these global communities with people we didn’t know.

But if all we have is the link, there are two problems here that we haven’t addressed.

The first one is… you go visit this site and it delivers some code to you. How does it know what kind of code it should deliver to you? Because if you’re running on a Mac, then you need different machine code than you do on Windows. That’s why you have different versions of programs for different operating systems.

Then should a web site have a different version of the code for every possible device? No.

Instead, the site has one version of the code—the source code. This is what’s delivered to the user. Then it gets translated to machine code on the user’s device.

The name for this concept is portability.

So that’s great, you can load code from people who don’t know you and don’t know what kind of device you’re running.

But that brings us to a second problem. If you don’t know these people who’s web pages you’re loading, how do you know what kind of code they’re giving you? It could be malicious code. It could be trying to take over your system.

Doesn’t this vision of the web—running code from anybody who’s link you follow—mean that you have to blindly trust anyone who’s on the web?

This is where the other key concept from the web comes in.

That’s the security model. I’m going to call it the sandbox.

Basically, the browser takes the page—that other person’s code—and instead of letting it run around willy-nilly in your system, it puts it in a sandbox. It puts a couple of toys that aren’t dangerous into that sandbox so that the code can do some things, but it leaves the dangerous things outside of the sandbox.

So the utility of the link is based on these two things:

  • Portability—the ability to deliver code to users and have it run on any type of device that can run a browser.
  • And the sandbox—the security model that lets you run that code without risking the integrity of your machine.

So why does this distinction matter? Why does it make a difference if we think of the web as something that the browser shows us using HTML, CSS, and JS, or if we think of the web in terms of portability and the sandbox?

Because it changes how you think about WebAssembly.

You can think about WebAssembly as just another tool in the browser’s toolbox… which it is.

It is another tool in the browser’s toolbox. But it’s not just that. It also gives us a way to take these other two capabilities of the web—the portability and the security model—and take them to other use cases that need them too.

We can expand the web past the boundaries of the browser. Now let’s look at where these attributes of the web would be useful.

Node.js

How could WebAssembly help Node? It could bring full portability to Node.

Node gives you most of the portability of JavaScript on the web. But there are lots of cases where Node’s JS modules aren’t quite enough—where you need to improve performance or reuse existing code that’s not written in JS.

In these cases, you need Node’s native modules. These modules are written in languages like C, and they need to be compiled for the specific kind of machine that the user is running on.

Native modules are either compiled when the user installs, or precompiled into binaries for a wide matrix of different systems. One of these approaches is a pain for the user, the other is a pain for the package maintainer.

Now, if these native modules were written in WebAssembly instead, then they wouldn’t need to be compiled specifically for the target architecture. Instead, they’d just run like the JavaScript in Node runs. But they’d do it at nearly native performance.

So we get to full portability for the code running in Node. You could take the exact same Node app and run it across all different kinds of devices without having to compile anything.

But WebAssembly doesn’t have direct access to the system’s resources. Native modules in Node aren’t sandboxed—they have full access to all of the dangerous toys that the browser keeps out of the sandbox. In Node, JS modules also have access to these dangerous toys because Node makes them available. For example, Node provides methods for reading from and writing files to the system.

For Node’s use case, it makes a certain amount of sense for modules to have this kind access to dangerous system APIs. So if WebAssembly modules don’t have that kind of access by default (like Node’s current modules do), how could we give WebAssembly modules the access they need? We’d need to pass in functions so that the WebAssembly module can work with the operating system, just as Node does with JS.

For Node, this will probably include a lot of the functionality that’s in things like the C standard library. It would also likely include things that are part of POSIX—the Portable Operating System Interface—which is an older standard that helps with compatibility. It provides one API for interacting with the system across a bunch of different Unix-like OSs. Modules would definitely need a bunch of POSIX-like functions.

Skill: Portable interface

What the Node core folks would need to do is figure out the set of functions to expose and the API to use.

But wouldn’t it be nice if that were actually something standard? Not something that was specific to just Node, but could be used across other runtimes and use cases too?

A POSIX for WebAssembly if you will. A PWSIX? A portable WebAssembly system interface.

And if that were done in the right way, you could even implement the same API for the web. These standard APIs could be polyfilled onto existing Web APIs.

These functions wouldn’t be part of the WebAssembly spec. And there would be WebAssembly hosts that wouldn’t have them available. But for those platforms that could make use of them, there would be a unified API for calling these functions, no matter which platform the code was running on. And this would make universal modules—ones that run across both the web and Node—so much easier.

Where are we with this?

So, is this something that could actually happen?

A few things are working in this idea’s favor. There’s a proposal called package name maps that will provide a mechanism for mapping a module name to a path to load the module from. And that will likely be supported by both browsers and Node, which can use it to provide different paths, and thus load entirely different modules, but with the same API. This way, the .wasm module itself can specify a single (module-name, function-name) import pair that Just Works on different environments, even the web.

With that mechanism in place, what’s left to do is actually figure out what functions make sense and what their interface should be.

There’s no active work on this at the moment. But a lot of discussions are heading in this direction right now. And it looks likely to happen, in one form or another.

Which is good, because unlocking this gets us halfway to unlocking some other use cases outside the browser. And with this in place, we can accelerate the pace.

So, what are some examples of these other use cases?

CDNs, Serverless, and Edge Computing

One example is things like CDNs, and Serverless, and Edge Computing. These are cases where you’re putting your code on someone else’s server, and they make sure that the server is maintained and that the code is close to all of your users.

Why would you want to use WebAssembly in these cases? There was a great talk explaining exactly this at a conference recently.

Fastly is a company that provides CDNs and edge computing. And their CTO, Tyler McMullen, explained it this way (and I’m paraphrasing here):

If you look at how a process works, code in that process doesn’t have boundaries. Functions have access to whatever memory in that process they want, and they can call whatever other functions they want.

When you’re running a bunch of different people’s services in the same process, this is an issue. Sandboxing could be a way to get around this. But then you get to a scale problem.

For example, if you use a JavaScript VM like Firefox’s SpiderMonkey or Chrome’s V8, you get a sandbox and you can put hundreds of instances into a process. But with the numbers of requests that Fastly is servicing, you don’t just need hundreds per process—you need tens of thousands.

Tyler does a better job of explaining all of it in his talk, so you should go watch that. But the point is that WebAssembly gives Fastly the safety, speed, and scale needed for this use case.

So what did they need to make this work?

Skill: Runtime

They needed to create their own runtime. That means taking a WebAssembly compiler—something that can compile WebAssembly down to machine code—and combining it with the functions for interacting with the system that I mentioned before.

For the WebAssembly compiler, Fastly used Cranelift, the compiler that we’re also building into Firefox. It’s designed to be very fast and doesn’t use much memory.

Now, for the functions that interact with the rest of the system, they had to create their own, because we don’t have that portable interface available yet.

So it’s possible to create your own runtime today, but it takes some effort. And it’s effort that will have to be duplicated across different companies.

What if we didn’t just have the portable interface, but we also had a common runtime that could be used across all of these companies and other use cases? That would definitely speed up development.

Then other companies could just use that runtime—like they do Node today—instead of creating their own from scratch.

Where are we on this?

So what’s the status of this?

Even though there’s no standard runtime yet, there are a few runtime projects in flight right now. These include WAVM, which is built on top of LLVM, and wasmjit.

In addition, we’re planning a runtime that’s built on top of Cranelift, called wasmtime.

And once we have a common runtime, that speeds up development for a bunch of different use cases. For example…

Portable CLI tools


WebAssembly can also be used in more traditional operating systems. Now to be clear, I’m not talking about in the kernel (although brave souls are trying that, too) but WebAssembly running in Ring 3—in user mode.

Then you could do things like have portable CLI tools that could be used across all different kinds of operating systems.

And this is pretty close to another use case…

Internet of Things


The internet of things includes devices like wearable technology, and smart home appliances.

These devices are usually resource constrained—they don’t pack much computing power and they don’t have much memory. And this is exactly the kind of situation where a compiler like Cranelift and a runtime like wasmtime would shine, because they would be efficient and low-memory. And in the extremely-resource-constrained case, WebAssembly makes it possible to fully compile to machine code before loading the application on the device.

There’s also the fact that there are so many of these different devices, and they are all slightly different. WebAssembly’s portability would really help with that.

So that’s one more place where WebAssembly has a future.

Conclusion

Now let’s zoom back out and look at this skill tree.

Skill tree showing WebAssembly skills which will be filled in through the course of the post.I said at the beginning of this post that people have a misconception about WebAssembly—this idea that the WebAssembly that landed in the MVP was the final version of WebAssembly.

I think you can see now why this is a misconception.

Yes, the MVP opened up a lot of opportunities. It made it possible to bring a lot of desktop applications to the web. But we still have many use cases to unlock, from heavy-weight desktop applications, to small modules, to JS frameworks, to all the things outside the browser… Node.js, and serverless, and the blockchain, and portable CLI tools, and the internet of things.

So the WebAssembly that we have today is not the end of this story—it’s just the beginning.

The post WebAssembly’s post-MVP future: A cartoon skill tree appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Introducing Opus 1.3

The Opus Audio Codec gets another major update with the release of version 1.3 (demo).

Opus is a totally open, royalty-free audio codec that can be used for all audio applications, from music streaming and storage to high-quality video-conferencing and VoIP. Six years after its standardization by the IETF, Opus is now included in all major browsers and mobile operating systems. It has been adopted for a wide range of applications, and is the default WebRTC codec.

This release brings quality improvements to both speech and music compression, while remaining fully compatible with RFC 6716. Here’s a few of the upgrades that users and implementers will care about the most.

speech vs music probabiities graph with opus 1.3

Opus 1.3 includes a brand new speech/music detector. It is based on a recurrent neural network and is both simpler and more reliable than the detector that has been used since version 1.1. The new detector should improve the Opus performance on mixed content encoding, especially at bitrates below 48 kb/s.

There are also many improvements for speech encoding at lower bitrates, both for mono and stereo. The demo contains many more details, as well as some audio samples. This new release also includes a cool new feature: ambisonics support. Ambisonics can be used to encode 3D audio soundtracks for VR and 360 videos.

You can read all the details of The Release of Opus 1.3.

Read more

The post Introducing Opus 1.3 appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Home Monitoring with Things Gateway 0.6

When it comes to smart home devices, protecting the safety and security of your home when you aren’t there is a popular area of adoption. Traditional home security systems are either completely offline (an alarm sounds in the house, but nobody is notified) or professionally monitored (with costly subscription services). Self monitoring of your connected home therefore makes sense, but many current smart home solutions still require ongoing service fees and send your private data to a centralised cloud service.

A floor plan style diagram describes uses of autonomous home monitoring with Project Things

The latest version of the Things Gateway rolls out today with new home monitoring features that let you directly monitor your home over the web, without a middleman. That means no monthly fees, your private data stays in your home by default, and you can choose from a variety of sensors from different brands.

Version 0.6 adds support for door sensors, motion sensors and customisable push notifications. Other enhancements include support for push buttons and a wider range of Apple HomeKit devices, as well as general robustness improvements and better error reporting.

Sensors

The latest update comes with support for door/window sensors and motion sensors, including the SmartThings Motion Sensor and SmartThings Multipurpose Sensor.An illustration with icons of various sensors used in home monitoringThese sensors make great triggers for a home monitoring system and also report temperature, battery level and tamper detection.

Push Notifications

You can now create rules which trigger a push notification to your desktop, laptop, tablet or smartphone. An example use case for this is to notify you when a door has been opened or motion is detected in your home, but you can use notifications for whatever you like!

To create a rule which triggers a push notification, simply drag and drop the notification output and customize it with your own message.

A diagram showing how the Intruder Alarm is triggered by the interaction of the sensors.Thanks to the power of Progressive Web Apps, if you’ve installed the gateway’s web app on your smartphone or tablet you’ll receive notifications even if the web app is closed.

Push Buttons

We’ve also added support for push buttons, like the SmartThings Button, which you can program to trigger any action you like using the rules engine. Use a button to simply turn a light on, or set a whole scene with multiple outputs.

Diagram of writing rules that trigger actions by the the Push Button

Error Reporting

0.6 also comes with a range of robustness improvements including connection detection and error reporting. That means it will be easier to tell whether you have lost connectivity to the gateway, or one of your devices has dropped offline, and if something goes wrong with an add-on, you’ll be informed about it inside the gateway UI.

If a device has dropped offline, its icon is displayed as translucent until it comes back online. If your web app loses connectivity with the gateway, you’ll see a message appear at the bottom of the screen.A diagram of all the sensors showing their status.

HomeKit

The HomeKit adapter add-on now supports a wider range of Apple HomeKit compatible devices including:

Smart plugs

Bridges

Light bulbs

Sensors

These devices use the built-in Bluetooth or WiFi support of your Raspberry Pi-based gateway, so you don’t even need a USB dongle.

Download

You can download version 0.6 today from the website. If you’ve already built your own Things Gateway with a Raspberry Pi and have it connected to the Internet, it should automatically update itself soon.

We can’t wait to see what creative things you do with all these new features. Be sure to let us know on Discourse and Twitter!

The post Home Monitoring with Things Gateway 0.6 appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Calls between JavaScript and WebAssembly are finally fast 🎉

At Mozilla, we want WebAssembly to be as fast as it can be.

This started with its design, which gives it great throughput. Then we improved load times with a streaming baseline compiler. With this, we compile code faster than it comes over the network.

So what’s next?

One of our big priorities is making it easy to combine JS and WebAssembly. But function calls between the two languages haven’t always been fast. In fact, they’ve had a reputation for being slow, as I talked about in my first series on WebAssembly.

That’s changing, as you can see

This means that in the latest version of Firefox Beta, calls between JS and WebAssembly are faster than non-inlined JS to JS function calls. Hooray! 🎉

Performance chart showing time for 100 million calls. wasm-to-js before: about 750ms. wasm-to-js after: about 450ms. JS-to-wasm before: about 5500ms. JS-to-wasm after: about 450ms. monomorphic JS-to-wasm before: about 5250ms. monomorphic JS-to-wasm before: about 250ms. wasm-to-builtin before: about 6000ms. wasm-to-builtin before: about 650ms.

So these calls are fast in Firefox now. But, as always, I don’t just want to tell you that these calls are fast. I want to explain how we made them fast. So let’s look at how we improved each of the different kinds of calls in Firefox (and by how much).

But first, let’s look at how engines do these calls in the first place. (And if you already know how the engine handles function calls, you can skip to the optimizations.)

How do function calls work?

Functions are a big part of JavaScript code. A function can do lots of things, such as:

  • assign variables which are scoped to the function (called local variables)
  • use functions that are built-in to the browser, like Math.random
  • call other functions you’ve defined in your code
  • return a value

A function with 4 lines of code: assigning a local variable with let w = 8; calling a built-in function with Math.random(); calling a user-defined function named randGrid(); and returning a value.

But how does this actually work? How does writing this function make the machine do what you actually want? 

As I explained in my first WebAssembly article series, the languages that programmers use — like JavaScript — are very different than the language the computer understands. To run the code, the JavaScript we download in the .js file needs to be translated to the machine language that the machine understands. 

Each browser has a built-in translator. This translator is sometimes called the JavaScript engine or JS runtime. However, these engines now handle WebAssembly too, so that terminology can be confusing. In this article, I’ll just call it the engine.

Each browser has its own engine:

  • Chrome has V8
  • Safari has JavaScriptCore (JSC)
  • Edge has Chakra
  • and in Firefox, we have SpiderMonkey

Even though each engine is different, many of the general ideas apply to all of them. 

When the browser comes across some JavaScript code, it will fire up the engine to run that code. The engine needs to work its way through the code, going to all of the functions that need to be called until it gets to the end.

I think of this like a character going on a quest in a videogame.

Let’s say we want to play Conway’s Game of Life. The engine’s quest is to render the Game of Life board for us. But it turns out that it’s not so simple…

Engine asking Sir Conway function to explain life. Sir Conway sends the engine to the Universum Neu function to get a Universe.

So the engine goes over to the next function. But the next function will send the engine on more quests by calling more functions.

Engine going to Universum Neu to ask for a universe. Universum Neu sends the engine to Randgrid.

The engine keeps having to go on these nested quests until it gets to a function that just gives it a result. 

Rnadgrid giving the engine a grid.

Then it can come back to each of the functions that it spoke to, in reverse order.

The engine returning through all of the functions.

If the engine is going to do this correctly — if it’s going to give the right parameters to the right function and be able to make its way all the way back to the starting function — it needs to keep track of some information. 

It does this using something called a stack frame (or a call frame). It’s basically like a sheet of paper that has the arguments to go into the function, says where the return value should go, and also keeps track of any of the local variables that the function creates. 

A stack frame, which is basically a form with lines for arguments, locals, a return value, and more.

The way it keeps track of all of these slips of paper is by putting them in a stack. The slip of paper for the function that it is currently working with is on top. When it finishes that quest, it throws out the slip of paper. Because it’s a stack, there’s a slip of paper underneath (which has now been revealed by throwing away the old one). That’s where we need to return to. 

This stack of frames is called the call stack.

a stack of stack frames, which is basically a pile of papers

The engine builds up this call stack as it goes. As functions are called, frames are added to the stack. As functions return, frames are popped off of the stack. This keeps happening until we get all the way back down and have popped everything out of the stack.

So that’s the basics of how function calls work. Now, let’s look at what made function calls between JavaScript and WebAssembly slow, and talk about how we’ve made this faster in Firefox.

How we made WebAssembly function calls fast

With recent work in Firefox Nightly, we’ve optimized calls in both directions — both JavaScript to WebAssembly and WebAssembly to JavaScript. We’ve also made calls from WebAssembly to built-ins faster.

All of the optimizations that we’ve done are about making the engine’s work easier. The improvements fall into two groups:

  • Reducing bookkeeping —which means getting rid of unnecessary work to organize stack frames
  • Cutting out intermediaries — which means taking the most direct path between functions

Let’s look at where each of these came into play.

Optimizing WebAssembly » JavaScript calls

When the engine is going through your code, it has to deal with functions that are speaking two different kinds of language—even if your code is all written in JavaScript. 

Some of them—the ones that are running in the interpreter—have been turned into something called byte code. This is closer to machine code than JavaScript source code, but it isn’t quite machine code (and the interpreter does the work). This is pretty fast to run, but not as fast as it can possibly be.

Other functions — those which are being called a lot — are turned into machine code directly by the just-in-time compiler (JIT). When this happens, the code doesn’t run through the interpreter anymore.

So we have functions speaking two languages; byte code and machine code.

I think of these different functions which speak these different languages as being on different continents in our videogame. 

A game map with two continents—One with a country called The Interpreter Kingdom, and the other with a country called JITland

The engine needs to be able to go back and forth between these continents. But when it does this jump between the different continents, it needs to have some information, like the place it left from on the other continent (which it will need to go back to). The engine also wants to separate the frames that it needs. 

To organize its work, the engine gets a folder and puts the information it needs for its trip in one pocket — for example, where it entered the continent from. 

 It will use the other pocket to store the stack frames. That pocket will expand as the engine accrues more and more stack frames on this continent.

A folder with a map on the left side, and the stack of frames on the right.

Sidenote: if you’re looking through the code in SpiderMonkey, these “folders” are called activations.

Each time it switches to a different continent, the engine will start a new folder. The only problem is that to start a folder, it has to go through C++. And going through C++ adds significant cost.

This is the trampolining that I talked about in my first series on WebAssembly. 

Every time you have to use one of these trampolines, you lose time. 

In our continent metaphor, it would be like having to do a mandatory layover on Trampoline Point for every single trip between two continents.

Same map as before, with a new Trampoline country on the same continent as The Interpreter Kingdom. An arrow goes from The Interpreter Kingdom, to Trampoline, to JITland.

So how did this make things slower when working with WebAssembly? 

When we first added WebAssembly support, we had a different type of folder for it. So even though JIT-ed JavaScript code and WebAssembly code were both compiled and speaking machine language, we treated them as if they were speaking different languages. We were treating them as if they were on separate continents.

Same map with Wasmania island next to JITland. There is an arrow going from JITland to Trampoline to Wasmania. On Trampoline, the engine asks a shopkeeper for folders.

This was unnecessarily costly in two ways:

  • it creates an unnecessary folder, with the setup and teardown costs that come from that
  • it requires that trampolining through C++ (to create the folder and do other setup)

We fixed this by generalizing the code to use the same folder for both JIT-ed JavaScript and WebAssembly. It’s kind of like we pushed the two continents together, making it so you don’t need to leave the continent at all.

SpiderMonkey engineer Benjamin Bouvier pushing Wasmania and JITland together

With this, calls from WebAssembly to JS were almost as fast as JS to JS calls.

Same perf graph as above with wasm-to-JS circled.

We still had a little work to do to speed up calls going the other way, though.

Optimizing JavaScript » WebAssembly calls

Even though JIT-ed JavaScript and WebAssembly speak the same language, they have different customs. They have different ways of doing things.

Even in the case of JIT-ed JavaScript code, where JavaScript and WebAssembly are speaking the same language, they still use different customs. 

For example, to handle dynamic types, JavaScript uses something called boxing.

Because JavaScript doesn’t have explicit types, types need to be figured out at runtime. The engine keeps track of the types of values by attaching a tag to the value. 

It’s as if the JS engine put a box around this value. The box contains that tag indicating what type this value is. For example, the zero at the end would mean integer.

Two binary numbers with a box around them, with a 0 label on the box.

In order to compute the sum of these two integers, the system needs to remove that box. It removes the box for a and then removes the box for b.

Two lines, the first with boxed numbers from the last image. The second with unboxed numbers.

Then it adds the unboxed values together.

Three lines, with the third line being the two numbers added together

Then it needs to add that box back around the results so that the system knows the result’s type.

Four lines, with the fourth line being the numbers added together with a box around it.

This turns what you expect to be 1 operation into 4 operations… so in cases where you don’t need to box (like statically typed languages) you don’t want to add this overhead.

Sidenote: JavaScript JITs can avoid these extra boxing/unboxing operations in many cases, but in the general case, like function calls, JS needs to fall back to boxing.

This is why WebAssembly expects parameters to be unboxed, and why it doesn’t box its return values. WebAssembly is statically typed, so it doesn’t need to add this overhead. WebAssembly also expects values to be passed in at a certain place — in registers rather than the stack that JavaScript usually uses. 

If the engine takes a parameter that it got from JavaScript, wrapped inside of a box, and gives it to a WebAssembly function, the WebAssembly function wouldn’t know how to use it. 

Engine giving a wasm function boxed values, and the wasm function being confused.

So, before it gives the parameters to the WebAssembly function, the engine needs to unbox the values and put them in registers.

To do this, it would go through C++ again. So even though we didn’t need to trampoline through C++ to set up the activation, we still needed to do it to prepare the values (when going from JS to WebAssembly).

The engine going to Trampoline to get the numbers unboxed before going to Wasmania

Going to this intermediary is a huge cost, especially for something that’s not that complicated. So it would be better if we could cut the middleman out altogether.

That’s what we did. We took the code that C++ was running — the entry stub — and made it directly callable from JIT code. When the engine goes from JavaScript to WebAssembly, the entry stub un-boxes the values and places them in the right place. With this, we got rid of the C++ trampolining.

I think of this as a cheat sheet. The engine uses it so that it doesn’t have to go to the C++. Instead, it can unbox the values when it’s right there, going between the calling JavaScript function and the WebAssembly callee.

The engine looking at a cheat sheet for how to unbox values on its way from JITland to Wasmania.

So that makes calls from JavaScript to WebAssembly fast. 

Perf chart with JS to wasm circled.

But in some cases, we can make it even faster. In fact, we can make these calls even faster than JavaScript » JavaScript calls in many cases.

Even faster JavaScript » WebAssembly: Monomorphic calls

When a JavaScript function calls another function, it doesn’t know what the other function expects. So it defaults to putting things in boxes.

But what about when the JS function knows that it is calling a particular function with the same types of arguments every single time? Then that calling function can know in advance how to package up the arguments in the way that the callee wants them. 

JS function not boxing values

This is an instance of the general JS JIT optimization known as “type specialization”. When a function is specialized, it knows exactly what the function it is calling expects. This means it can prepare the arguments exactly how that other function wants them… which means that the engine doesn’t need that cheat sheet and spend extra work on unboxing.

This kind of call — where you call the same function every time — is called a monomorphic call. In JavaScript, for a call to be monomorphic, you need to call the function with the exact same types of arguments each time. But because WebAssembly functions have explicit types, calling code doesn’t need to worry about whether the types are exactly the same — they will be coerced on the way in.

If you can write your code so that JavaScript is always passing the same types to the same WebAssembly exported function, then your calls are going to be very fast. In fact, these calls are faster than many JavaScript to JavaScript calls.

Perf chart with monomorphic JS to wasm circled

Future work

There’s only one case where an optimized call from JavaScript » WebAssembly is not faster than JavaScript » JavaScript. That is when JavaScript has in-lined a function.

The basic idea behind in-lining is that when you have a function that calls the same function over and over again, you can take an even bigger shortcut. Instead of having the engine go off to talk to that other function, the compiler can just copy that function into the calling function. This means that the engine doesn’t have to go anywhere — it can just stay in place and keep computing. 

I think of this as the callee function teaching its skills to the calling function.

Wasm function teaching the JS function how to do what it does.

This is an optimization that JavaScript engines make when a function is being run a lot — when it’s “hot” — and when the function it’s calling is relatively small. 

We can definitely add support for in-lining WebAssembly into JavaScript at some point in the future, and this is a reason why it’s nice to have both of these languages working in the same engine. This means that they can use the same JIT backend and the same compiler intermediate representation, so it’s possible for them to interoperate in a way that wouldn’t be possible if they were split across different engines. 

Optimizing WebAssembly » Built-in function calls

There was one more kind of call that was slower than it needed to be: when WebAssembly functions were calling built-ins. 

Built-ins are functions that the browser gives you, like Math.random. It’s easy to forget that these are just functions that are called like any other function.

Sometimes the built-ins are implemented in JavaScript itself, in which case they are called self-hosted. This can make them faster because it means that you don’t have to go through C++: everything is just running in JavaScript. But some functions are just faster when they’re implemented in C++.

Different engines have made different decisions about which built-ins should be written in self-hosted JavaScript and which should be written in C++. And engines often use a mix of both for a single built-in.

In the case where a built-in is written in JavaScript, it will benefit from all of the optimizations that we have talked about above. But when that function is written in C++, we are back to having to trampoline.

Engine going from wasmania to trampoline to built-in

These functions are called a lot, so you do want calls to them to be optimized. To make it faster, we’ve added a fast path specific to built-ins. When you pass a built-in into WebAssembly, the engine sees that what you’ve passed it is one of the built-ins, at which point it knows how to take the fast-path. This means you don’t have to go through that trampoline that you would otherwise.

It’s kind of like we built a bridge over to the built-in continent. You can use that bridge if you’re going from WebAssembly to the built-in. (Sidenote: The JIT already did have optimizations for this case, even though it’s not shown in the drawing.)

A bridge added between wasmania and built-in

With this, calls to these built-ins are much faster than they used to be.

Perf chart with wasm to built-in circled.

Future work

Currently the only built-ins that we support this for are mostly limited to the math built-ins. That’s because WebAssembly currently only has support for integers and floats as value types. 

That works well for the math functions because they work with numbers, but it doesn’t work out so well for other things like the DOM built-ins. So currently when you want to call one of those functions, you have to go through JavaScript. That’s what wasm-bindgen does for you.

Engine going from wasmania to the JS Data Marshall Islands to built-in

But WebAssembly is getting more flexible types very soon. Experimental support for the current proposal is already landed in Firefox Nightly behind the pref javascript.options.wasm_gc. Once these types are in place, you will be able to call these other built-ins directly from WebAssembly without having to go through JS.

The infrastructure we’ve put in place to optimize the Math built-ins can be extended to work for these other built-ins, too. This will ensure many built-ins are as fast as they can be.

But there are still a couple of built-ins where you will need to go through JavaScript. For example, if those built-ins are called as if they were using new or if they’re using a getter or setter. These remaining built-ins will be addressed with the host-bindings proposal.

Conclusion

So that’s how we’ve made calls between JavaScript and WebAssembly fast in Firefox, and you can expect other browsers to do the same soon.

Performance chart showing time for 100 million calls. wasm-to-js before: about 750ms. wasm-to-js after: about 450ms. JS-to-wasm before: about 5500ms. JS-to-wasm after: about 450ms. monomorphic JS-to-wasm before: about 5250ms. monomorphic JS-to-wasm before: about 250ms. wasm-to-builtin before: about 6000ms. wasm-to-builtin before: about 650ms.

Thank you

Thank you to Benjamin Bouvier, Luke Wagner, and Till Schneidereit for their input and feedback.

The post Calls between JavaScript and WebAssembly are finally fast 🎉 appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Hack on MDN: Better accessibility for MDN Web Docs

From Saturday, September 22 to Monday, September 24, more than twenty people met in London to work on improving accessibility on MDN Web Docs — both the content about accessibility and the accessibility of the site itself. While much remains to be done, the result was a considerable refresh in both respects.

Attendees at Hack on MDN listen to a lightning talk by Eva Ferreira

Attendees at Hack on MDN listen to a lightning talk by Eva Ferreira. Photo by Adrian Roselli.

Hack on MDN events

Hack on MDN events evolved from the documentation sprints for MDN that were held from 2010 to 2013, which brought together staff members and volunteers to write and localize content on MDN over a weekend. As implied by the name, “Hack on MDN” events expand the range of participants to include those with programming and design skills. In its current incarnation, each Hack on MDN event has a thematic focus. One in March of this year focused on browser compatibility data.

The Hack on MDN format is a combination of hackathon and unconference; participants pitch projects and commit to working on concrete tasks (rather than meetings or long discussions) that can be completed in three days or less. People self-organize to work on projects in which a group can make significant progress over a long weekend. Lightning talks provide an unconference break from projects.

Accessibility on MDN Web Docs

Making websites accessible to a wide range of users, including those with physical or cognitive limitations, is a vital topic for creators on the web. Yet information about accessibility on MDN Web Docs was sparse and often outdated. Similarly, the accessibility of the site had eroded over time. Therefore, accessibility was chosen as the theme for the September 2018 Hack on MDN.

Hack on MDN Accessibility in London

The people who gathered at Campus London (thanks to Google for the space), included writers, developers, and accessibility experts, from within and outside of Mozilla. After a round of introductions, there was a “pitch” session presenting ideas of projects to work on. Participants rearranged themselves into project groups, and the hacking began. Adrian Roselli gave a brief crash course on accessibility for non-experts in the room, which he fortunately had up his sleeve and was able to present while jet-lagged.

At the end of each morning and afternoon, we did a status check-in to see how work was progressing. On Sunday and Monday, there were also lightning talks, where anyone could present anything that they wanted to share. Late Sunday afternoon, some of us took some time out to explore some of the offerings of the Shoreditch Design Triangle, including playing with a “font” comprised of (more or less sit-able) chairs.

Glenda Sims, Estelle Weyl, Janet Swisher and Adrian Roselli pose with metal letter-shaped chairs spelling "HACK" and "MdN"

Glenda Sims, Estelle Weyl, Janet Swisher and Adrian Roselli pose with metal letter-shaped chairs spelling “HACK” and “MdN”. Photo by Dan Rubin.

Outcomes

One project focused on updating the WAI-ARIA documentation on MDN Web Docs, using a new ARIA reference page template created by Estelle Weyl. Eric Bailey, Eric Eggert, and several others completed documentation on 27 ARIA roles, including recommending appropriate semantic HTML elements to use in preference to an ARIA role. The team even had remote contributors, with Shane Hudson writing about the ARIA alert role.

A number of participants worked on adding sections on “Accessibility concerns” to relevant HTML, CSS, and JavaScript pages, such as the <canvas> element, display property, and the Animation API.

Other efforts included:

Also, a fun time was had and the group enjoyed working together. Check the #HackOnMDN tag on Twitter for photos, “overheard” quotes, nail art by @ninjanails and more. Also see blog posts by Adrian Roselli and Hidde de Vries for their perspectives and more details.

What’s next?

There is plenty of work left to make MDN’s accessibility content up-to-date and useful. The list of ARIA roles, states, and properties is far from complete. More reference pages need “accessibility concerns” information added. The accessibility of the MDN Web Docs site still can be improved. As a result of the enthusiasm from this event, discussions are starting about doing a mini-hack in connection with an upcoming accessibility conference.

If you find issues that need to be addressed, please file a bug against the site or the content. Better yet, get involved in improving MDN Web Docs. If you’re not sure where to begin, visit the MDN community forum to ask any questions you might have about how to make MDN more awesome (and accessible). We’d love to have your help!

The post Hack on MDN: Better accessibility for MDN Web Docs appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Dweb: Creating Decentralized Organizations with Aragon

In the Dweb series, we are covering projects that explore what is possible when the web becomes decentralized or distributed. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source and open for participation, and they share Mozilla’s mission to keep the web open and accessible for all.

While many of the projects we’ve covered build on the web as we know it or operate like the browsers we’re familiar with, the Aragon project has a broader vision: Give people the tools to build their own autonomous organizations with social mores codified in smart contracts. I hope you enjoy this introduction to Aragon from project co-founder Luis Cuende.

– Dietrich Ayala

Introducing Aragon

I’m Luis. I cofounded Aragon, which allows for the creation of decentralized organizations. The principles of Aragon are embodied in the Aragon Manifesto, and its format was inspired by the Mozilla Manifesto!

Here’s a quick summary.

  • We are in a key moment in history: Technology either oppresses or liberates us.
  • That outcome will depend on common goods being governed by the community, and not just nation states or corporate conglomerates.
  • For that to happen, we need technology that allows for decentralized governance.
  • Thanks to crypto, decentralized governance can provide new means of organization that don’t entail violence or surveillance, therefore providing more freedom to the individual and increasing fairness.

Aragon logo

With Aragon, developers can create new apps, such as voting mechanisms, that use smart contracts to leverage decentralized governance and allow peers to control resources like funds, membership, and code repos.

Aragon is built on Ethereum, which is a blockchain for smart contracts. Smart contracts are software that is executed in a trust-less and transparent way, without having to rely on a third-party server or any single point of failure.

Aragon is at the intersection of social, app platform, and blockchain.

Architecture

The Aragon app is one of few truly decentralized apps. Its smart contracts and front end are upgrade-able thanks to aragonOS and Aragon Package Manager (APM). You can think of APM as a fully decentralized and community-governed NPM. The smart contracts live on the Ethereum blockchain, and APM takes care of storing a log of their versions. APM also keeps a record of arbitrary data blobs hosted on decentralized storage platforms like IPFS, which in our case we use for storing the front end for the apps.

Aragon architecture diagram

The Aragon app allows users to install new apps into their organization, and those apps are embedded using sandboxed iframes. All the apps use Aragon UI, therefore users don’t even know they are interacting with apps made by different developers. Aragon has a very rich permission system that allows users to set what each app can do inside their organization. An example would be: Up to $1 can be withdrawn from the funds if there’s a vote with 51% support.

Aragon tech stack diagram

Hello World

To create an Aragon app, you can go to the Aragon Developer portal. Getting started is very easy.

First, install IPFS if you don’t have it already installed.

Second, run the following commands:

$ npm i -g @aragon/cli
$ aragon init foo.aragonpm.eth
$ cd foo
$ aragon run

Here we will show a basic counter app, which allows members of an organization to count up or down if a democratic vote happens, for example.

This would be the smart contract (in Solidity) that keeps track of the counter in Ethereum:

contract Counter is AragonApp {
/**
* @notice Increment the counter by 1
*/
function increment() auth(INCREMENT_ROLE) external {
// ...
} /**
* @notice Decrement the counter by 1
*/
function decrement() auth(DECREMENT_ROLE) external {
// ...
}
}

This code runs in a web worker, keeping track of events in the smart contract and caching the state in the background:

// app/script.js
import Aragon from '@aragon/client' // Initialize the app
const app = new Aragon() // Listen for events and reduce them to a state
const state$ = app.store((state, event) => {
// Initial state
if (state === null) state = 0 // Build state
switch (event.event) {
case 'Decrement':
state--
break
case 'Increment':
state++
break
} return state
})

Some basic HTML (not using Aragon UI, for simplicity):

<!-- app/index.html !-->
<!doctype html> <button id="decrement">-</button>
<div id="view">...</div>
<button id="increment">+</button>
<script src="app.js"></script>

And the JavaScript that updates the UI:

// app/app.js
import Aragon, { providers } from '@aragon/client' const app = new Aragon(
new providers.WindowMessage(window.parent)
)
const view = document.getElementById('view') app.state().subscribe(
function(state) {
view.innerHTML = `The counter is ${state || 0}`
},
function(err) {
view.innerHTML = 'An error occurred, check the console'
console.log(err)
}
)

aragon run takes care of updating your app on APM and uploading your local webapp to IPFS, so you don’t need to worry about it!

Learn More

You can go to Aragon’s website or the Developer Portal to learn more about Aragon. If you are interested in decentralized governance, you can also check out our research forum.

If you would like to contribute, you can look at our good first issues.

If you have any questions, please join the Aragon community chat!

The post Dweb: Creating Decentralized Organizations with Aragon appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Streaming RNNs in TensorFlow

The Machine Learning team at Mozilla Research continues to work on an automatic speech recognition engine as part of Project DeepSpeech, which aims to make speech technologies and trained models openly available to developers. We’re hard at work improving performance and ease-of-use for our open source speech-to-text engine. The upcoming 0.2 release will include a much-requested feature: the ability to do speech recognition live, as the audio is being recorded. This blog post describes how we changed the STT engine’s architecture to allow for this, achieving real-time transcription performance. Soon, you’ll be able to transcribe audio at least as fast as it’s coming in.

When applying neural networks to sequential data like audio or text, it’s important to capture patterns that emerge over time. Recurrent neural networks (RNNs) are neural networks that “remember” — they take as input not just the next element in the data, but also a state that evolves over time, and use this state to capture time-dependent patterns. Sometimes, you may want to capture patterns that depend on future data as well. One of the ways to solve this is by using two RNNs, one that goes forward in time and one that goes backward, starting from the last element in the data and going to the first element. You can learn more about RNNs (and about the specific type of RNN used in DeepSpeech) in this article by Chris Olah.

Using a bidirectional RNN

The current release of DeepSpeech (previously covered on Hacks) uses a bidirectional RNN implemented with TensorFlow, which means it needs to have the entire input available before it can begin to do any useful work. One way to improve this situation is by implementing a streaming model: Do the work in chunks, as the data is arriving, so when the end of the input is reached, the model is already working on it and can give you results more quickly. You could also try to look at partial results midway through the input.

This animation shows how the data flows through the network. Data flows from the audio input to feature computation, through three fully connected layers. Then it goes through a bidirectional RNN layer, and finally through a final fully connected layer, where a prediction is made for a single time step.

This animation shows how the data flows through the network. Data flows from the audio input to feature computation, through three fully connected layers. Then it goes through a bidirectional RNN layer, and finally through a final fully connected layer, where a prediction is made for a single time step.

In order to do this, you need to have a model that lets you do the work in chunks. Here’s the diagram of the current model, showing how data flows through it.

As you can see, on the bidirectional RNN layer, the data for the very last step is required for the computation of the second-to-last step, which is required for the computation of the third-to-last step, and so on. These are the red arrows in the diagram that go from right to left.

We could implement partial streaming in this model by doing the computation up to layer three as the data is fed in. The problem with this approach is that it wouldn’t gain us much in terms of latency: Layers four and five are responsible for almost half of the computational cost of the model.

Using a unidirectional RNN for streaming

Instead, we can replace the bidirectional layer with a unidirectional layer, which does not have a dependency on future time steps. That lets us do the computation all the way to the final layer as soon as we have enough audio input.

With a unidirectional model, instead of feeding the entire input in at once and getting the entire output, you can feed the input piecewise. Meaning, you can input 100ms of audio at a time, get those outputs right away, and save the final state so you can use it as the initial state for the next 100ms of audio.

An alternative architecture that uses a unidirectional RNN in which each time step only depends on the input at that time and the state from the previous step.

An alternative architecture that uses a unidirectional RNN in which each time step only depends on the input at that time and the state from the previous step.

Here’s code for creating an inference graph that can keep track of the state between each input window:

import tensorflow as tf def create_inference_graph(batch_size=1, n_steps=16, n_features=26, width=64): input_ph = tf.placeholder(dtype=tf.float32, shape=[batch_size, n_steps, n_features], name='input') sequence_lengths = tf.placeholder(dtype=tf.int32, shape=[batch_size], name='input_lengths') previous_state_c = tf.get_variable(dtype=tf.float32, shape=[batch_size, width], name='previous_state_c') previous_state_h = tf.get_variable(dtype=tf.float32, shape=[batch_size, width], name='previous_state_h') previous_state = tf.contrib.rnn.LSTMStateTuple(previous_state_c, previous_state_h) # Transpose from batch major to time major input_ = tf.transpose(input_ph, [1, 0, 2]) # Flatten time and batch dimensions for feed forward layers input_ = tf.reshape(input_, [batch_size*n_steps, n_features]) # Three ReLU hidden layers layer1 = tf.contrib.layers.fully_connected(input_, width) layer2 = tf.contrib.layers.fully_connected(layer1, width) layer3 = tf.contrib.layers.fully_connected(layer2, width) # Unidirectional LSTM rnn_cell = tf.contrib.rnn.LSTMBlockFusedCell(width) rnn, new_state = rnn_cell(layer3, initial_state=previous_state) new_state_c, new_state_h = new_state # Final hidden layer layer5 = tf.contrib.layers.fully_connected(rnn, width) # Output layer output = tf.contrib.layers.fully_connected(layer5, ALPHABET_SIZE+1, activation_fn=None) # Automatically update previous state with new state state_update_ops = [ tf.assign(previous_state_c, new_state_c), tf.assign(previous_state_h, new_state_h) ] with tf.control_dependencies(state_update_ops): logits = tf.identity(logits, name='logits') # Create state initialization operations zero_state = tf.zeros([batch_size, n_cell_dim], tf.float32) initialize_c = tf.assign(previous_state_c, zero_state) initialize_h = tf.assign(previous_state_h, zero_state) initialize_state = tf.group(initialize_c, initialize_h, name='initialize_state') return { 'inputs': { 'input': input_ph, 'input_lengths': sequence_lengths, }, 'outputs': { 'output': logits, 'initialize_state': initialize_state, } }

The graph created by the code above has two inputs and two outputs. The inputs are the sequences and their lengths. The outputs are the logits and a special “initialize_state” node that needs to be run at the beginning of a new sequence. When freezing the graph, make sure you don’t freeze the state variables previous_state_h and previous_state_c.

Here’s code for freezing the graph:

from tensorflow.python.tools import freeze_graph freeze_graph.freeze_graph_with_def_protos( input_graph_def=session.graph_def, input_saver_def=saver.as_saver_def(), input_checkpoint=checkpoint_path, output_node_names='logits,initialize_state', restore_op_name=None, filename_tensor_name=None, output_graph=output_graph_path, initializer_nodes='', variable_names_blacklist='previous_state_c,previous_state_h')

With these changes to the model, we can use the following approach on the client side:

  1. Run the “initialize_state” node.
  2. Accumulate audio samples until there’s enough data to feed to the model (16 time steps in our case, or 320ms).
  3. Feed through the model, accumulate outputs somewhere.
  4. Repeat 2 and 3 until data is over.

It wouldn’t make sense to drown readers with hundreds of lines of the client-side code here, but if you’re interested, it’s all MPL 2.0 licensed and available on GitHub. We actually have two different implementations, one in Python that we use for generating test reports, and one in C++ which is behind our official client API.

Performance improvements

What does this all mean for our STT engine? Well, here are some numbers, compared with our current stable release:

  • Model size down from 468MB to 180MB
  • Time to transcribe: 3s file on a laptop CPU, down from 9s to 1.5s
  • Peak heap usage down from 4GB to 20MB (model is now memory-mapped)
  • Total heap allocations down from 12GB to 264MB

Of particular importance to me is that we’re now faster than real time without using a GPU, which, together with streaming inference, opens up lots of new usage possibilities like live captioning of radio programs, Twitch streams, and keynote presentations; home automation; voice-based UIs; and so on. If you’re looking to integrate speech recognition in your next project, consider using our engine!

Here’s a small Python program that demonstrates how to use libSoX to record from the microphone and feed it into the engine as the audio is being recorded.

import argparse
import deepspeech as ds
import numpy as np
import shlex
import subprocess
import sys parser = argparse.ArgumentParser(description='DeepSpeech speech-to-text from microphone')
parser.add_argument('--model', required=True, help='Path to the model (protocol buffer binary file)')
parser.add_argument('--alphabet', required=True, help='Path to the configuration file specifying the alphabet used by the network')
parser.add_argument('--lm', nargs='?', help='Path to the language model binary file')
parser.add_argument('--trie', nargs='?', help='Path to the language model trie file created with native_client/generate_trie')
args = parser.parse_args() LM_WEIGHT = 1.50
VALID_WORD_COUNT_WEIGHT = 2.25
N_FEATURES = 26
N_CONTEXT = 9
BEAM_WIDTH = 512 print('Initializing model...') model = ds.Model(args.model, N_FEATURES, N_CONTEXT, args.alphabet, BEAM_WIDTH)
if args.lm and args.trie: model.enableDecoderWithLM(args.alphabet, args.lm, args.trie, LM_WEIGHT, VALID_WORD_COUNT_WEIGHT)
sctx = model.setupStream() subproc = subprocess.Popen(shlex.split('rec -q -V0 -e signed -L -c 1 -b 16 -r 16k -t raw - gain -2'), stdout=subprocess.PIPE, bufsize=0)
print('You can start speaking now. Press Control-C to stop recording.') try: while True: data = subproc.stdout.read(512) model.feedAudioContent(sctx, np.frombuffer(data, np.int16))
except KeyboardInterrupt: print('Transcription:', model.finishStream(sctx)) subproc.terminate() subproc.wait()

Finally, if you’re looking to contribute to Project DeepSpeech itself, we have plenty of opportunities. The codebase is written in Python and C++, and we would love to add iOS and Windows support, for example. Reach out to us via our IRC channel or our Discourse forum.

The post Streaming RNNs in TensorFlow appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Firefox Focus with GeckoView

Firefox Focus is private browsing as an app: It automatically blocks ads and trackers, so you can surf the web in peace. When you’re done, a single tap completely erases your history, cookies, and other local data.

Protecting you from invasive tracking is part of Mozilla’s non-profit mission, and Focus’s built-in tracking protection helps keep you safe. It also makes websites load faster!

A screenshot of Firefox Focus, showing the main menu open with the heading "26 Trackers Blocked"

With Focus, you don’t have to worry about your browsing history coming back to haunt you in retargeted ads on other websites.

Bringing Gecko to Focus

In the next weeks, we’ll release a new version of Focus for Android, and for the first time, Focus will come bundled with Gecko, the browser engine that powers Firefox Quantum. This is a major architectural change, so while every copy of Focus will include Gecko—hence the larger download size—we plan on enabling it gradually to ensure a smooth transition. You can help us test Gecko in Focus today by installing the Focus Beta.

Diagram of Firefox Focus 7, showing how the app now contains GeckoView, instead of just relying on the WebView component provided by Android

Note: At time of publishing, Focus Beta is conducting an A/B test between the old and new engines. Look for “Gecko/62.0” in your User-Agent String to determine if your copy is using Gecko or not.

Up until this point, Focus has been powered exclusively by Android’s built-in WebView. This made sense for initial development, since WebView was already on every Android device, but we quickly ran into limitations. Foremost, it isn’t designed for building browsers. Despite being based on Chromium, WebView only supports a subset of web standards, as Google expects app developers to use native Android APIs, and not the Web, for advanced functionality. Instead, we’d prefer if apps had access to the entirety of the open, standards-based web platform.

In Focus’s case, we can only build next-generation privacy features if we have deep access to the browser internals, and that means we need our own engine. We need Gecko. Fortunately, Firefox for Android already uses Gecko, just not in a way that’s easy to reuse in other applications. That’s where GeckoView comes in.

GeckoView: Making Gecko Reusable

GeckoView is Gecko packaged as a reusable Android library. We’ve worked to decouple the engine itself from its user interface, and made it easy to embed in other applications. Thanks to GeckoView’s clean architecture, our initial benchmarks of the new Focus show a median page load improvement of 20% compared to Firefox for Android, making GeckoView our fastest version of Gecko on Android yet.

Screenshot of the GeckoView AAR (Android Library) file. It is about 37 MB large.

We first put GeckoView into production last year, powering both Progressive Web Apps (PWAs) and Custom Tabs in Firefox for Android. These minimal, self-contained features were good initial projects, but with Focus we’re going much further. Focus will be our first time using GeckoView to completely power an existing, successful, and standalone product.

We’re also using GeckoView in entirely new products like Firefox Reality, a browser designed exclusively for virtual and augmented reality headsets. We’ll be sharing more about it later this year.

Building Browsers with Android Components

To build a web browser, you need more than just an engine. You also need common functionality like tabs, auto-complete, search suggestions, and so on. To avoid unnecessary duplication of effort, we’ve also created Android Components, a collection of independent, ready-to-use libraries for building browsers and browser-like applications on Android.

For Mozilla, GeckoView means we can leverage all of our Firefox expertise in building more compelling, safe, and robust online experiences, while Android Components ensures that we can continue experimenting with new projects (like Focus and Firefox Reality) without reinventing wheels. In many ways, these projects set the stage for the next generation of the Firefox family of browsers on Android.

For Android developers, GeckoView means control. It’s a production-grade engine with a stable and expansive API, usable either on its own or through Android Components. Because GeckoView is a self-contained library, you don’t have to compile it yourself. Furthermore, powering your app with GeckoView gives you a specific web engine version you can work with. Compare that to WebView, which tends to have quite a bit of variance between versions depending on the OS and Chrome version available on the device. With GeckoView, you always know what you’re getting — and you benefit from Gecko’s excellent, cross-platform support for web standards.

Get Involved

We’re really excited about what GeckoView means for the future of browsers on Android, and we’d love for you to get involved:

Let us know what you think of GeckoView and the new Focus in the comments below!

The post Firefox Focus with GeckoView appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Converting a WebGL application to WebVR

A couple months ago I ported the Pathfinder demo app to WebVR. It was an interesting experience, and I feel like I learned a bunch of things about porting WebGL applications to WebVR that would be generally useful to folks, especially folks coming to WebVR from non-web programming backgrounds.

Pathfinder is a GPU-based font rasterizer in Rust, and it comes with a demo app that runs the Rust code on the server side but does all the GPU work in WebGL in a TypeScript website.

We had a 3D demo showing a representation of the Mozilla Monument as a way to demo text rasterization in 3D. What I was hoping to do was to convert this to a WebVR application that would let you view the monument by moving your head instead of using arrow keys.

I started working on this problem with a decent understanding of OpenGL and WebGL, but almost zero background in VR or WebVR. I’d written an Android Cardboard app three years previously and that was about it.

I’m hoping this article may be useful for others from similar backgrounds.

The converted triangle demo running in WebVR

What is WebVR?

WebVR is a set of APIs for writing VR applications on the web. It lets us request jumping into VR mode, at which point we can render things directly to the eyes of a VR display, rather than rendering to a flat surface browser within the display. When the user is on a device like the Cardboard or Daydream where a regular phone substitutes for the VR display, this is the point where the user puts their phone within the headset.

WebVR APIs help with transitioning to/from VR mode, obtaining pose information, rendering in VR, and dealing with device input. Some of these things are being improved in the work in progress on the new WebXR Device API specification.

Do I need any devices to work with WebVR?

Ideally, a good VR device will make it easier to test your work in progress, but depending on how much resolution you need, a Daydream or Cardboard (where you use your phone in a headset casing) is enough. You can even test stuff without the headset casing, though stuff will look weird and distorted.

For local testing Chrome has a WebVR API emulation extension that’s pretty useful. You can use the devtools panel in it to tweak the pose, and you get a non-distorted display of what the eyes see.

Firefox supports WebVR, and Chrome Canary supports it if you enable some flags. There’s also a polyfill which should work for more browsers.

How does it work under the hood?

I think not understanding this part was the source of a lot of confusion and bugs for me when I was getting started. The core of the API is basically “render something to a canvas and then magic happens”, and I had trouble figuring how that magic worked.

Essentially, there’s a bunch of work we’re supposed to do, and then there’s extra work the browser (or polyfill) does.

Once we enter VR mode, there’s a callback triggered whenever the device requests a frame. Within this callback we have access to pose information.

Using this pose information, we can figure out what each eye should see, and provide this to the WebVR API in some form.

What the WebVR API expects is that we render each eye’s view to a canvas, split horizontally (this canvas will have been passed to the API when we initialize it).

That’s it from our side, the browser (or polyfill) does the rest. It uses our rendered canvas as a texture, and for each eye, it distorts the rendered half to appropriately work with the lenses used in your device. For example, the distortion for Daydream and Cardboard follows this code in the polyfill.

It’s important to note that, as application developers, we don’t have to worry about this — the WebVR API is handling it for us! We need to render undistorted views from each eye to the canvas — the left view on the left half and the right view on the right half, and the browser handles the rest!

Porting WebGL applications

I’m going to try and keep this self contained, however I’ll mention off the bat that some really good resources for learning this stuff can be found at webvr.info and MDN. webvr.info has a bunch of neat samples if, like me, you learn better by looking at code and playing around with it.

Entering VR mode

First up, we need to be able to get access to a VR display and enter VR mode.

let vrDisplay;
navigator.getVRDisplays().then(displays => { if (displays.length === 0) { return; } vrDisplay = displays[displays.length - 1]; // optional, but recommended vrDisplay.depthNear = /* near clip plane distance */; vrDisplay.depthFar = /* far clip plane distance */;
}

We need to add an event handler for when we enter/exit VR:

let canvas = document.getElementById(/* canvas id */);
let inVR = false; window.addEventListener('vrdisplaypresentchange', () => { // no VR display, exit if (vrDisplay == null) return; // are we entering or exiting VR? if (vrDisplay.isPresenting) { // We should make our canvas the size expected // by WebVR const eye = vrDisplay.getEyeParameters("left"); // multiply by two since we're rendering both eyes side // by side canvas.width = eye.renderWidth * 2; canvas.height = eye.renderHeight; const vrCallback = () => { if (vrDisplay == null || !inVR) { return; } // reregister callback if we're still in VR vrDisplay.requestAnimationFrame(vrCallback); // render scene render(); }; // register callback vrDisplay.requestAnimationFrame(vrCallback); } else { inVR = false; // resize canvas to regular non-VR size if necessary }
});

And, to enter VR itself:

if (vrDisplay != null) { inVR = true; // hand the canvas to the WebVR API vrDisplay.requestPresent([{ source: canvas }]); // requestPresent() will request permission to enter VR mode, // and once the user has done this our `vrdisplaypresentchange` // callback will be triggered
}

Rendering in VR

Well, we’ve entered VR, now what? In the above code snippets we had a render() call which was doing most of the hard work.

Since we’re starting with an existing WebGL application, we’ll have some function like this already.

let width = canvas.width;
let height = canvas.height; function render() { let gl = canvas.getContext("gl"); gl.viewport(0, 0, width, height); gl.clearColor(/* .. */); gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT); gl.useProgram(program); gl.bindBuffer(/* .. */); // ... let uProjection = gl.getUniformLocation(program, "uProjection"); let uModelView = gl.getUniformLocation(program, "uModelview"); gl.uniformMatrix4fv(uProjection, false, /* .. */); gl.uniformMatrix4fv(uModelView, false, /* .. */); // set more parameters // run gl.drawElements()
} 

So first we’re going to have to split this up a bit further, to handle rendering the two eyes:


// entry point for WebVR, called by vrCallback() function renderVR() { let gl = canvas.getContext("gl"); // set clearColor and call gl.clear() clear(gl); renderEye(true); renderEye(false); vrDisplay.submitFrame(); // Send the rendered frame over to the VR display } // entry point for non-WebVR rendering // called by whatever mechanism (likely keyboard/mouse events) // you used before to trigger redraws function render() { let gl = canvas.getContext("gl"); // set clearColor and call gl.clear() clear(gl); renderSceneOnce(); } function renderEye(isLeft) { // choose which half of the canvas to draw on if (isLeft) { gl.viewport(0, 0, width / 2, height); } else { gl.viewport(width / 2, 0, width / 2, height); } renderSceneOnce(); } function renderSceneOnce() { // the actual GL program and draw calls go here }

This looks like a good step forward, but notice that we’re rendering the same thing to both eyes, and not handling movement of the head at all.

To implement this we need to use the perspective and view matrices provided by WebVR from the VRFrameData object.

The VRFrameData object contains a pose member with all of the head pose information (its position, orientation, and even velocity and acceleration for devices that support these). However, for the purpose of correctly positioning the camera whilst rendering, VRFrameData provides projection and view matrices which we can directly use.

We can do this like so:

let frameData = new VRFrameData();
vrDisplay.getFrameData(frameData); // use frameData.leftViewMatrix / framedata.leftProjectionMatrix
// for the left eye, and
// frameData.rightViewMatrix / framedata.rightProjectionMatrix for the right

In graphics, we often find ourselves dealing with the model, view, and projection matrices. The model matrix defines the position of the object we wish to render in the coordinates of our space, the view matrix defines the transformation between the camera space and the world space, and the projection matrix handles the transformation between clip space and camera space (also potentially dealing with perspective). Sometimes we’ll deal with the combination of some of these, like the “model-view” matrix.

One can see these matrices in use in the cubesea code in the stereo rendering example from webvr.info.

There’s a good chance our application has some concept of a model/view/projection matrix already. If not, we can pre-multiply our positions with the view matrix in our vertex shaders.

So now our code will look something like this:

// entry point for non-WebVR rendering
// called by whatever mechanism (likely keyboard/mouse events)
// we used before to trigger redraws
function render() { let gl = canvas.getContext("gl"); // set clearColor and call gl.clear() clear(gl); let projection = /* calculate projection using something like glmatrix.mat4.perspective() (we should be doing this already in the normal WebGL app) */; let view = /* use our view matrix if we have one, or an identity matrix */; renderSceneOnce(projection, view);
} function renderEye(isLeft) { // choose which half of the canvas to draw on let projection, view; let frameData = new VRFrameData(); vrDisplay.getFrameData(frameData); if (isLeft) { gl.viewport(0, 0, width / 2, height); projection = frameData.leftProjectionMatrix; view = frameData.leftViewMatrix; } else { gl.viewport(width / 2, 0, width / 2, height); projection = frameData.rightProjectionMatrix; view = frameData.rightViewMatrix; } renderSceneOnce(projection, view);
} function renderSceneOnce(projection, view) { let model = /* obtain model matrix if we have one */; let modelview = glmatrix.mat4.create(); glmatrix.mat4.mul(modelview, view, model); gl.useProgram(program); gl.bindBuffer(/* .. */); // ... let uProjection = gl.getUniformLocation(program, "uProjection"); let uModelView = gl.getUniformLocation(program, "uModelview"); gl.uniformMatrix4fv(uProjection, false, projection); gl.uniformMatrix4fv(uModelView, false, modelview); // set more parameters // run gl.drawElements()
}

This should be it! Moving your head around should now trigger movement in the scene to match it! You can see the code at work in this demo app that takes a spinning triangle WebGL application and turns it into a WebVR-capable triangle-viewing application using the techniques from this blog post.

If we had further input we might need to use the Gamepad API to design a good VR interface that works with typical VR controllers, but that’s out of scope for this post.

The post Converting a WebGL application to WebVR appeared first on Mozilla Hacks – the Web developer blog.

Original Link

New API to Bring Augmented Reality to the Web

We’re entering a new phase of work on JavaScript APIs here at Mozilla, that will help everyone create and share virtual reality (VR) and augmented reality (AR) projects on the open web.

As you might know, we formally launched this work last year with the release of Firefox desktop support for the WebVR 1.1 API. Using that draft API, early adopters like WITHIN were able to distribute 3D experiences on the web and have them work well on a range of devices, from mobile phones and cardboard viewers to full-fledged, immersive VR headsets.

AR demo app on iOS

The Expansion of WebVR

WebVR has been instrumental in democratizing VR, so more people can experience 3D content without expensive headsets. It’s also been a huge time-saver for content creators, who need to test and verify that their work renders well on every viewing platform. Having a stable API to work with means 3D content can find a wider audience, and it cuts down on the rework creators have to do to deliver great web experiences to a range of devices.

Mozilla has been pushing the boundaries of VR in the browser, getting people together across the industry to support a standard way of rendering 3D content. That work has created a fast lane for artists and programmers to share web-based VR experiences with a growing user base. And with WebVR support in browsers like Firefox, we’ve started the work of liberating VR and AR content from silos and headset stores, and making them accessible on the open web.

The Promise of Mixed Reality

Mixed Reality is going to be a powerful platform, bringing highly engaging and emotionally evocative immersive content to the web. Like any new creative medium, we want it to be widely accessible, so curious viewers can experience the next generation of digital media without having to shell out hundreds of dollars for a high-end viewer.

Today, the industry is taking another step toward these goals. We have ambitions to broaden the number of platforms and devices that can display VR and AR content. For instance, the camera on most mobile phones can be used to overlay information on physical reality – if it has a set of instructions on how to do that.

Experimentation continues with a new JavaScript API called the WebXR Device API. We expect this specification will replace WebVR in time and offer a smooth path forward for folks using WebVR today.

What’s New in WebXR

The new WebXR Device API has two new goals that differentiate it from WebVR. They are:

  • To support a wider variety of user inputs, such as voice and gestures, giving users options for navigating and interacting in virtual spaces
  • To establish a technical foundation for development of AR experiences, letting creators integrate real-world media with contextual overlays that elevate the experience.

You can find details about WebXR Device API by visiting the Immersive Web Community Group. We expect that many of the same crew that worked on WebVR – talented engineers from Mozilla, Google, Samsung, Amazon and other companies – will continue to work on the WebXR Device API, along with new contributors like Magic Leap.

AR Comes to the Web

AR and VR both are at the cutting edge of creative expression. Some museums offer AR experiences to give depth and context to exhibits. Other projects include educational content, from geology lessons to what it’s like to walk the streets in war-torn Syria.

What can augmented reality do on the web? Already there are examples that demonstrate powerful use cases. For instance, want to know how that new sofa will fit in your living room, before you buy it? Or how an espresso machine would look in your kitchen? Augmented reality can make online shopping a more sensory experience, so you can test-drive new products in your home in a way that preserves size and scale. It’s a great complement to online shopping, especially as companies start offering online visualizations of physical products.

Mozilla has some key tenets for how we’d like this next-generation media to work on behalf of users.

  • We want to ensure user privacy. You shouldn’t have to give an art store website access to pictures of your home and everything in it in order to see how a poster would look on your wall.
  • We want to make AR and VR accessible to the widest possible audience. We’re committed to removing barriers for people.
  • We want to help creators make content that works on all devices, so users can access mixed reality experiences with the device they have, or want to use.
  • We want to enable the long tail of creators, not just big studios and well-known brands. Everyone who wants to should be able to augment the world, not just those who can get an app into a store.

The WebXR community is working on draft specifications that target some of the constraints of today’s wireless devices. For instance, creating a skybox setting you can use to can change the background image of a web page. We’re also working on a way to expose the world-sensing capabilities of early AR platforms to the web, so developers can determine where surfaces are without needing to run complex computer vision code on a battery-powered device.

Support in Firefox

We’re proud that Firefox supports WebVR today, so people can use current technology while we’re working to implement the next-generation specification. We have begun work to add WebXR support to Firefox. An early implementation will be available in Firefox Nightly in the coming months, so developers and early adopters can turn it on and give it a test-drive.

Some parts of the WebXR specification are still in motion. Rather than waiting for a final version of the spec, we’re going to move forward with what we have now and adjust to any changes along the way. The roadmap for the upcoming Firefox Reality browser will be similar to the Firefox desktop version, with initial support for immersive browsing using WebVR, and WebXR support to follow.

In time, we plan to support WebXR everywhere that we support WebVR currently, including Windows, Linux, macOS, and Android/GeckoView platforms. We will continue supporting WebVR until most popular sites and engines have completed the transition to WebXR. Want more technical details? Check out this WebXR explainer.

Today’s AR Experiments

If you can’t wait to dive into augmented reality, here’s something you can try today: Mozilla’s WebXR Viewer for iOS. It’s a way you can get a glimpse of the future right on your iPhone (6s or newer) or iPad. To be clear: this app is an experiment based on a proposed interim API we created last year. We are currently converting it to use the WebXR Device API.

We created this app as a way to experiment with AR and to find out how easy it was to get it working on iOS using Apple’s ARKit.  If you want to have a look at the code for the iOS app, it’s posted on GitHub. For Android users, Google has a similar experiment going with early support for the immersive web.

Want to keep up with the progress of WebXR and the new WebXR Device API? Follow @mozillareality on twitter, or subscribe to the Mozilla Mixed Reality blog for our weekly roundup of XR news.

The post New API to Bring Augmented Reality to the Web appeared first on Mozilla Hacks – the Web developer blog.

Original Link

Firefox 62 – Tools Cool for School!

Hello there! It’s been six-odd weeks, and the march of progress continues to, uh… march… progressingly. That means we have a brand new Firefox to share, with an abundance of bug fixes, performance improvements, and (in particular) sweet developer tool treats! So tuck in your napkin and enjoy this tasting menu of some of what’s new in Firefox 62.

Shape Up Your Floats

A CSS shape around some grapes

CSS Shapes lets a floated element sculpt the flow of content around it beyond the classic rectangular bounding box we’ve been constrained to. For instance, in the above screenshot and linked demo, the text is wrapping to the shape of the grapes vs the image’s border. There are properties for basic shapes all the way up to complex polygons. There are of course great docs on all of this, but Firefox 62 also includes new tooling to both inspect and visually manipulate CSS Shapes values.

You can learn more in Josh Marinacci’s post on the new CSS Shapes tooling from yesterday.

Variable Fonts Are Here!

Screenshot of the new font tool in the Firefox DevTools

No punny title, I’m just excited! OpenType Font Variations allow a single font file to contain multiple instances of the same font, encoding the differences between instances. In addition to being in one file, font creators can expose any number of variation axes that give developers fine-grained control on how a font is rendered. These can be standard variations like font weight (font weight 536 looks right? no problem!) or things that were never previously available via CSS (x-height! serif-size!). In addition to the candy-store possibilities for typography nerds, being able to serve a single file with multiple variants is a major page weight savings. Dan Callahan goes much deeper on the grooviness to be found and how Firefox makes it easy to tweak these new custom values.

Devtools Commands

The Developer Toolbar was an alternate command repl input in the Firefox Developer tools, apart from the Web Console. I say “was” because as of Firefox 62, it has been removed. It was always a bit hard to find and not as well-advertised as it could be, but did encapsulate some powerful commands. Most of these commands have been progressively migrated elsewhere in the devtools, and this is wrapped up in Firefox 62, so we’ve removed the toolbar altogether.

One of the last commands to be migrated is screenshot, which is a power-user version of the “take a screenshot” button available in the devtools UI. The screenshot command is now available as :screenshot in the Web Console. For example, have you ever needed a high-res screenshot of a page for print? You can specify a higher pixel density for a screenshot via the command:

:screenshot --dpr 4

There are a bunch of other options as well, such as specifying output filenames, capture delays, and selector-cropped screenshots. Eric Meyer wrote a great primer on the power of :screenshot on his blog, and it will change your page capture game!

🌠 Did You Know: In addition to :screenshot, there are a bunch of other helpful commands and magic variables available from within the Web Console? You can learn about them on MDN Web Docs.

Mo’ Pixels, Mo’ Panels

Do you have a 4k monitor? Do your browser windows bathe in a wash of ample screen real-estate? Let your devtools stretch their legs with a new 3-column mode in the Page Inspector. You can now pop the CSS Rules view into its own column, to let you view style information and the excellent Grid tooling or Animations panel side-by-side.

The Three-Column View toggle can be found in the top-left of the Inspector side panel.

Streamlining MediaStream

If you’ve worked with WebRTC’s getUserMedia API, you may be familiar with a bit of branching logic when attaching a MediaStream object to a <video> or <audio> tag:

navigator.mediaDevices.getUserMedia({ audio: true, video: true })
.then(function(stream) { if ("srcObject" in video) { videoEl.srcObject = stream; } else { videoEl.src = URL.createObjectURL(stream); }
});

It’s true that earlier support for WebRTC required the use of the URL API, but this was non-standard and is no longer necessary. Firefox 62 removes support for passing a MediaStream to createObjectURL, so be sure you’re using a proper capability check as above.

Why stop here?

I’ve shown you a glimpse of what’s new and exciting in Firefox 62, but there’s more to learn and love! Be sure to check out the product release notes for user-facing features as well as a more complete list of developer facing changes on MDN.

Happy building!

Original Link

Make your web layouts bust out of the rectangle with the Firefox Shape Path Editor

The web doesn’t have to be boxy. Historically, every element in a page is rendered as a rectangle of some kind, but it doesn’t have to be this way. With CSS Shapes you can create web layouts every bit as stylish as print magazines, but with all of the advantages of the web.

CSS Shapes let your web designs break out of the rectangular grid. All of those classic magazine design elements like non-rectangular text flow and shaped images can be yours, for the low low price of using a new CSS standard. Text can flow, images can be rounded, even just a few non-parallel lines can make your site stand out and make your brand distinctive. Standing out is the biggest challenge most sites face today. Shapes can help!

Save The Trees mockup with leaf-shaped icon, and flowed lorem ipsum text

Image by Sara Soueidan

The Standard

The shape of your elements can be controlled with just two CSS properties: shape-outside and clip-path.

The shape-outside property changes the way content flows outside of a floated DOM element. It affects layout, not drawing. The clip-path property changes the clipping boundary of how the DOM element is drawn. It affects drawing, not layout.

clipping the image of a kitten into a circular shape

The clip-path and shape-outside properties.

Because these two properties are separate, you can use one, or both, or none — to get just exactly the effect you are looking for. The good news is that both of these use the same basic-shape syntax.

Want to clip your image to be in a circle? Just use clip-path: circle(50%). Want to make text wrap around your image as if it were a circle, just use shape-outside: circle(50%). The shape syntax supports rectangles, circles, ellipses, and full polygons. Of course, manually positioning polygons with numbers is slow and painful. Fortunately there is a better way.

The Shape Path Editor

With the Shape Path Editor in Firefox 62, you can visually edit the shape directly from the CSS inspector. Open your page in Firefox, and use Firefox Developer Tools to select the element whose shape you want to modify. Once you select the element there will be a little icon next to the shape-outside and clip-path properties if you have used one of them. If not, add shape-outside and clip-path to that element first. Click on that little icon to start the visual editor. Then you can directly manipulate the shape with your mouse.

Using the shape editor in Firefox Dev Tools

Image courtesy of placekitten, text courtesy of catipsum.

Open the Inspector and select the element you want to modify:

using the inspector to modify a kitten photo

Click the icon next to clip-path or shape-outside. If the element doesn’t have one of these properties, add it, then select it.

modifying the image element with the shape editor

Edit the clip path:

editing the clip path

Edit the outside shape:

editing the outside shape

Check out this live demo on glitch.

 

To learn more about how to use the CSS Shape Editor read the full documentation.

Progressive Enhancement

CSS shapes are here and they work today in most browsers, and most importantly they degrade gracefully. Readers with current browsers will get a beautiful experience and readers with non-compliant browsers will never know they are missing anything.

kitten with shape supportkitten image without shape support degrades progressively

Stunning Examples

Here are just a few examples of the amazing layouts you can do with CSS Shapes:

Page layout text effects with clip-path:

Codepen by Mandy Michael called "Create"

via Mandy Michael

Plants and background effect using clip-path:

Minion using shape-outside:

Break out of the Box

Shapes on the web are here today, thanks to shape-outside and clip-path. Using the Firefox Shape Path Editor makes them even easier to use.

How will you make your website break out of the box? Let us know how you’re using Shapes.

Original Link

Variable Fonts Arrive in Firefox 62

Firefox 62, which lands in general release this week, adds support for Variable Fonts, an exciting new technology that makes it possible to create beautiful typography with a single font file. Variable fonts are now supported in all major browsers.

What are Variable Fonts?

Font families can have dozens of variations: different weights, expanded or condensed widths, italics, etc. Traditionally, each variant required its own separate font file, which meant that Web designers had to balance typographic nuance with pragmatic concerns around page weight and network performance.

Compared to traditional fonts, variable fonts contain additional data, which make it possible to generate different styles of the font on demand. For one example, consider Jost*, an open-source, Futura-inspired typeface from indestructible type*. Jost* comes in nine weights, each with regular and italic styles, for a total of eighteen files.

Screenshot of 18 traditional TTF files next to to a single, variable TTF file that that can replace the 18 other files.

Jost* also comes as a single variable font file which is able to generate not only those same eighteen variations, but also any intermediate weight at any degree of italicization.

Design Axes

Jost* is an example of a “two-axis” variable font: it can vary in both weight and italics. Variable fonts can have any number of axes, and each axis can control any aspect of the design. Weight is the most common axis, but typographers are free to invent their own.

Illustration of the various weights and italics settings for the Jost* font

One typeface that invented its own axis is Slovic. Slovic is a Cyrillic variable font with a single axis, style, that effectively varies history. At one extreme, characters are drawn similarly to how they appear in 9th century manuscripts, while at the other, they adopt modern sans-serif forms. In between are several intermediate styles. Variable font technology allows the design to adapt and morph smoothly across the entire range of the axis.

Illustration of the Slovic font's letterforms morphing with different values of the "style" variable font axis

The sky’s the limit! To see other examples of variable fonts, check out v-fonts.com and Axis Praxis.

Better Tools for Better Typography on the Web

Great features deserve great tools, and that’s why we’re hard at work building an all new Font Editor into the Firefox DevTools. Here’s a sneak peek:

You can find the Font Editor as a panel inside the Page Inspector in the Dev Tools. If you have enough space on your screen, it’s helpful to enable 3-pane mode so you can see the DOM tree, CSS Rules, and Font Editor all side-by-side.

When you click on an element in the DOM tree, the Font Editor updates to show information about the selected element’s font, as well as tools for editing its properties. The Font Editor works on all fonts, but really shines with variable ones. For instance, the weight control subtly changes from stepped slider to continuous one in the presence of a variable font with a weight axis.

A comparison of the DevTools Font Editor when inspecting a variable font versus a traditional font, showing how the varible font axes appear as continuous, smooth sliders, while the traditional font has toggles or stepped sliders to adjust things like italic or weight

Similarly, each design axis in a variable font gets its own widget in the editor, allowing you to directly customize the font’s appearance and immediately see the results on your page.

The new Font Editor will arrive with Firefox 63 in October, but you can use it today by downloading Firefox Nightly. Let us know what you think! Your feedback is an essential guide as we continue to build and refine Firefox’s design tools.

Original Link

Dweb: Building Cooperation and Trust into the Web with IPFS

In this series we are covering projects that explore what is possible when the web becomes decentralized or distributed. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source, and open for participation, and share Mozilla’s mission to keep the web open and accessible for all.

Some projects start small, aiming for incremental improvements. Others start with a grand vision, leapfrogging today’s problems by architecting an idealized world. The InterPlanetary File System (IPFS) is definitely the latter – attempting to replace HTTP entirely, with a network layer that has scale, trust, and anti-DDOS measures all built into the protocol. It’s our pleasure to have an introduction to IPFS today from Kyle Drake, the founder of Neocities and Marcin Rataj, the creator of IPFS Companion, both on the IPFS team at Protocol Labs -Dietrich Ayala

IPFS – The InterPlanetary File System

We’re a team of people all over the world working on IPFS, an implementation of the distributed web that seeks to replace HTTP with a new protocol that is powered by individuals on the internet. The goal of IPFS is to “re-decentralize” the web by replacing the location-oriented HTTP with a content-oriented protocol that does not require trust of third parties. This allows for websites and web apps to be “served” by any computer on the internet with IPFS support, without requiring servers to be run by the original content creator. IPFS and the distributed web unmoor information from physical location and singular distribution, ultimately creating a more affordable, equal, available, faster, and less censorable web.

IPFS aims for a “distributed” or “logically decentralized” design. IPFS consists of a network of nodes, which help each other find data using a content hash via a Distributed Hash Table (DHT). The result is that all nodes help find and serve web sites, and even if the original provider of the site goes down, you can still load it as long as one other computer in the network has a copy of it. The web becomes empowered by individuals, rather than depending on the large organizations that can afford to build large content delivery networks and serve a lot of traffic.

The IPFS stack is an abstraction built on top of IPLD and libp2p:

The IPFS Stack: High Level Overview

Hello World

We have a reference implementation in Go (go-ipfs) and a constantly improving one in Javascript (js-ipfs). There is also a long list of API clients for other languages.

Thanks to the JS implementation, using IPFS in web development is extremely easy. The following code snippet…

  • Starts an IPFS node
  • Adds some data to IPFS
  • Obtains the Content IDentifier (CID) for it
  • Reads that data back from IPFS using the CID

<script src="https://unpkg.com/ipfs/dist/index.min.js"></script> Open Console (Ctrl+Shift+K) <script> const ipfs = new Ipfs() const data = 'Hello from IPFS, <YOUR NAME HERE>!' // Once the ipfs node is ready ipfs.once('ready', async () => { console.log('IPFS node is ready! Current version: ' + (await ipfs.id()).agentVersion) // convert your data to a Buffer and add it to IPFS console.log('Data to be published: ' + data) const files = await ipfs.files.add(ipfs.types.Buffer.from(data)) // 'hash', known as CID, is a string uniquely addressing the data // and can be used to get it again. 'files' is an array because // 'add' supports multiple additions, but we only added one entry const cid = files[0].hash console.log('Published under CID: ' + cid) // read data back from IPFS: CID is the only identifier you need! const dataFromIpfs = await ipfs.files.cat(cid) console.log('Read back from IPFS: ' + String(dataFromIpfs)) // Compatibility layer: HTTP gateway console.log('Bonus: open at one of public HTTP gateways: https://ipfs.io/ipfs/' + cid) }) </script>

That’s it!

Before diving deeper, let’s answer key questions:

Who else can access it?

Everyone with the CID can access it. Sensitive files should be encrypted before publishing.

How long will this content exist? Under what circumstances will it go away? How does one remove it?

The permanence of content-addressed data in IPFS is intrinsically bound to the active participation of peers interested in providing it to others. It is impossible to remove data from other peers but if no peer is keeping it alive, it will be “forgotten” by the swarm.

The public HTTP gateway will keep the data available for a few hours — if you want to ensure long term availability make sure to pin important data at nodes you control. Try IPFS Cluster: a stand-alone application and a CLI client to allocate, replicate and track pins across a cluster of IPFS daemons.

Developer Quick Start

You can experiment with js-ipfs to make simple browser apps. If you want to run an IPFS server you can install go-ipfs, or run a cluster, as we mentioned above.

There is a growing list of examples, and make sure to see the bi-directional file exchange demo built with js-ipfs.

You can add IPFS to the browser by installing the IPFS Companion extension for Firefox.

Learn More

Learn about IPFS concepts by visiting our documentation website at https://docs.ipfs.io.

Readers can participate by improving documentation, visiting https://ipfs.io, developing distributed web apps and sites with IPFS, and exploring and contributing to our git repos and various things built by the community.

A great place to ask questions is our friendly community forum: https://discuss.ipfs.io.
We also have an IRC channel, #ipfs on Freenode (or #freenode_#ipfs:matrix.org on Matrix). Join us!

Original Link

Share your favorite images and videos in VR with Mozilla Hubs

Last April we released Mozilla Hubs, a VR chat system that lets you walk and talk in VR with your friends, no matter where in the world they are. Now we have a game changing new feature: you can share virtually any kind of media with everyone in your Hubs room by just pasting in a URL. Anything you share becomes a virtual object that everyone can interact with.  From images to videos to 3D models, Hubs is the best way to collaborate across devices (laptops, phones, headsets) and OSes. Let’s look at a few details.

What can I share?

Hubs supports the common image formats: PNG, JPG, and even animated GIFs.

Hubs also supports streaming media files like MP3s and MP4s, as well as 3D models in GLB format (the compact binary form of GLTF).  And finally, Hubs has special support for content from certain websites.

If you paste in a URL to a model on Sketchfab (an online community and marketplace of 3D artists and makers), Hubs will fetch the model and import it into the room, no extra work required. As long as the model is less than 100MB and marked as downloadable, the Hubs server will process the webpage to find the link to the actual 3D model and import only that.  Hubs can also perform this same trick with URLs from Giphy, Imgur, Google Poly (only objects made in Blocks, not panoramas), and even YouTube videos. The media is cached into the Hubs server before sending to Hubs clients.

Is it secure?

Yes. When someone in your Hubs room pastes in a URL it does not immediately go to every client logged into the room. Instead the URL is processed by the Hubs server which hosts the encrypted media, sending only the validated data to the other clients.  This helps to protect from XSS attacks as well.

The Hubs server will host all content for up to 48 hours after the last time it is accessed, so once everyone is done interacting with it, the media file will eventually disappear. It is never copied anywhere else. The server also encrypts the data, so only clients in your Hubs room can see it. Additionally, Hubs does not track logins, so content has no user-identifiable data that could be used for tracking.

What can I do with content in Hubs?

You can pick up any kind of content object then move it, throw it, or leave it stuck in one place by holding still for a moment before releasing.  If you have a 6DoF controller then you can also resize the object.

For audio and video you can click to play and pause the media stream.  For PDFs you can advance one page at a time, making Hubs the best way to share presentations in VR, or just join your friends for some YouTube binging.

Are there any other requirements?

Nope.  Any content you have on the web you can share. Since the server acts as a proxy you don’t need to worry about CORS. If the content you want to share is not web accessible then you can upload the file directly to Hubs, which will then host and share it automatically. You can even use content from your public Dropbox folder. It all just works.

If you have content that you want to ensure users can share within Hubs, then just make it publicly accessible from the web in one of the file formats we support (as listed above).  For data you don’t want to stay public permanently, you can serve up a randomly generated URL which expires after 10 minutes. Once the data is imported into the Hubs server it no longer needs to be accessible on the web.

What now?

Play with it. Join your friends in a room to hang out, work on a presentation, or just watch some videos.  All with real-time voice communication as well. As always, Hubs will work on any device with a browser, from 2D desktop to standalone VR headset to smartphone. Hubs will adapt to the device you have available.  Just share a link to your room and go.

Original Link

Dweb: Serving the Web from the Browser with Beaker

In this series we are covering projects that explore what is possible when the web becomes decentralized or distributed. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source, and open for participation, and share Mozilla’s mission to keep the web open and accessible for all.

So far we’ve covered distributed social feeds and sharing files in a decentralized way with some new tools for developers. Today we’d like to introduce something a bit different: Imagine what an entire browser experience would be like if the web was distributed… Beaker browser does exactly this! Beaker is a big vision from a team who are proving out the distributed web from top to bottom. Please enjoy this post from Beaker co-creator Tara Vancil. – Dietrich Ayala


Blue Link Labs and Beaker

We’re Blue Link Labs, a team of three working to improve the Web with the Dat protocol and an experimental peer-to-peer browser called Beaker.

Blue Link Labs team

We work on Beaker because publishing and sharing is core to the Web’s ethos, yet to publish your own website or even just share a document, you need to know how to run a server, or be able to pay someone to do it for you.

So we asked ourselves, “What if you could share a website directly from your browser?”

Peer-to-peer protocols like dat:// make it possible for regular user devices to host content, so we use dat:// in Beaker to enable publishing from the browser, where instead of using a server, a website’s author and its visitors help host its files. It’s kind of like BitTorrent, but for websites!

Architecture

Beaker uses a distributed peer-to-peer network to publish websites and datasets (sometimes we call them “dats”).

dat:// websites are addressed with a public key as the URL, and each piece of
data added to a dat:// website is appended to a signed log. Visitors to a dat://
website find each other with a tracker or DHT, then sync the data between each other, acting both as downloaders and uploaders, and checking that the data hasn’t been tampered with in transit.

<img src="https://hacks.mozilla.org/files/2018/08/beaker-mozhacks-diagram.png" alt="a basic illustration of the dat:// network” width=”394″ height=”252″ class=”alignnone size-full wp-image-32540″ />

At its core, a dat:// website isn’t much different than an https:// website — it’s a collection of files and folders that a browser interprets according to Web standards. But dat:// websites are special in Beaker because we’ve added peer-to-peer Web APIs so developers can do things like read, write, and watch dat:// files and build peer-to-peer Web apps.

Create a P2P Website

Beaker makes it easy for anyone to create a new dat:// website with one click (see our tour). If you’re familiar with HTML, CSS, or JavaScript (even just a little bit!) then you’re ready to publish your first dat:// website.

Developers can get started by checking out our API documentation or reading through our guides.

This example shows a website editing itself to create and save a new JSON file. While this example is contrived, it demonstrates a common pattern for storing data, user profiles, etc. in a dat:// website—instead of application data being sent away to a server, it can be stored in the website itself!

// index.html Submit message <script src="index.js"></script> 
// index.js // first get an instance of the website's files
var files = new DatArchive(window.location) document.getElementById('create-json-button').addEventListener('click', saveMessage) async function saveMessage () {
var timestamp = Date.now()
var filename = timestamp + '.json'
var content = {
timestamp,
message: document.getElementById('message').value
} // write the message to a JSON file
// this file can be read later using the DatArchive.readFile API
await files.writeFile(filename, JSON.stringify(content))
}

Learn More

We’re always excited to see what people build with dat:// and Beaker. We especially love seeing when someone builds a personal site or blog, or when they experiment with Beaker’s APIs to build an app.

There’s lots to explore on the peer-to-peer Web!

Original Link

Dweb: Building a Resilient Web with WebTorrent

In this series we are covering projects that explore what is possible when the web becomes decentralized or distributed. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source, and open for participation, and share Mozilla’s mission to keep the web open and accessible for all.

The web is healthy when the financial cost of self-expression isn’t a barrier. In this installment of the Dweb series we’ll learn about WebTorrent – an implementation of the BitTorrent protocol that runs in web browsers. This approach to serving files means that websites can scale with as many users as are simultaneously viewing the website – removing the cost of running centralized servers at data centers. The post is written by Feross Aboukhadijeh, the creator of WebTorrent, co-founder of PeerCDN and a prolific NPM module author… 225 modules at last count! –Dietrich Ayala

What is WebTorrent?

WebTorrent is the first torrent client that works in the browser. It’s written completely in JavaScript – the language of the web – and uses WebRTC for true peer-to-peer transport. No browser plugin, extension, or installation is required.

Using open web standards, WebTorrent connects website users together to form a distributed, decentralized browser-to-browser network for efficient file transfer. The more people use a WebTorrent-powered website, the faster and more resilient it becomes.

Screenshot of the WebTorrent player interface

Architecture

The WebTorrent protocol works just like BitTorrent protocol, except it uses WebRTC instead of TCP or uTP as the transport protocol.

In order to support WebRTC’s connection model, we made a few changes to the tracker protocol. Therefore, a browser-based WebTorrent client or “web peer” can only connect to other clients that support WebTorrent/WebRTC.

Once peers are connected, the wire protocol used to communicate is exactly the same as in normal BitTorrent. This should make it easy for existing popular torrent clients like Transmission, and uTorrent to add support for WebTorrent. Vuze already has support for WebTorrent!

Diagram showing the decentralized P2P network of torrents

Getting Started

It only takes a few lines of code to download a torrent in the browser!

To start using WebTorrent, simply include the webtorrent.min.js script on your page. You can download the script from the WebTorrent website or link to the CDN copy.

<script src="webtorrent.min.js"></script>

This provides a WebTorrent function on the window object. There is also an
npm package available.

var client = new WebTorrent() // Sintel, a free, Creative Commons movie
var torrentId = 'magnet:...' // Real torrent ids are much longer. var torrent = client.add(torrentId) torrent.on('ready', () => {
// Torrents can contain many files. Let's use the .mp4 file
var file = torrent.files.find(file => file.name.endsWith('.mp4')) // Display the file by adding it to the DOM.
// Supports video, audio, image files, and more!
file.appendTo('body')
})

That’s it! Now you’ll see the torrent streaming into a <video width="300" height="150"> tag in the webpage!

Learn more

You can learn more at webtorrent.io, or by asking a question in #webtorrent on Freenode IRC or on Gitter. We’re looking for more people who can answer questions and help people with issues on the GitHub issue tracker. If you’re a friendly, helpful person and want an excuse to dig deeper into the torrent protocol or WebRTC, then this is your chance!

 

 

Original Link

The Video Wars of 2027

Author’s Note: This post imagines a dystopian future for web video, if we continue to rely on patented codecs to transmit media files. What if one company had a perpetual monopoly on those patents? How could it limit our access to media and culture? The premise of this cautionary tale is grounded in fact. However, the future scenario is fiction, and the entities and events portrayed are not intended to represent real people, companies, or events.

Illustration by James Dybvig

The year is 2029. It’s been two years since the start of the Video Wars, and there’s no end in sight. It’s hard to believe how deranged things have become on earth. People are going crazy because they can’t afford web video fees – and there’s not much else to do. The world’s media giants have irrevocably twisted laws and governments to protect their incredibly lucrative franchise: the right to own their intellectual property for all time.

It all started decades ago, with an arcane compression technology and a cartoon mouse. As if we needed any more proof that truth is stranger than fiction.

Adulteration of the U.S. Legal System

In 1998, the U.S. Congress passed the Sonny Bono Copyright Term Extension Act. This new law extended copyrights on corporate works to the author’s lifetime plus 95 years. The effort was driven by the Walt Disney Company, to protect its lucrative retail franchise around the animated character Mickey Mouse. Without this extension, Mickey would have entered the public domain, meaning anyone could create new cartoons and merchandise without fear of being sued by Disney. When the extension passed, it gave Disney another 20 years to profit from Mickey. The news sparked outrage from lawyers and academics at the time, but it was a dull and complex topic that most people didn’t understand or care about.

In 2020, Disney again lobbied to extend the law, so its copyright would last for 10,000 years. Its monopoly on our culture was complete. No art, music, video, or story would pass into the public domain for millennia. All copyrighted ideas would remain the private property of corporations. The quiet strangulation of our collective creativity had begun.

A small but powerful corporate collective called MalCorp took note of Disney’s success. Backed by deep-pocketed investors, MalCorp had quietly started buying the technology patents that made video streaming work over the internet. It revealed itself in 2021 as a protector of innovation. But its true goal was to create a monopoly on video streaming technology that would last forever, to shunt profits to its already wealthy investors. It was purely an instrument of greed.

Better Compression for Free

Now, there were some good guys in this story. As early as 2007, prescient tech companies wanted the web platform to remain free and open to all – especially for video. Companies like Cisco, Mozilla, Google, and others worked on new video codecs that could replace the patented, ubiquitous H.264 codec. They even combined their efforts in 2015 to create a royalty-free codec called AV1 that anyone could use free of charge.

AV1 was notable in that it offered better compression, and therefore better video quality, than any other codec of its time. But just as the free contender was getting off the ground, the video streaming industry was thrown into turmoil. Browser companies backed different codecs, and the market fragmented. Adoption stalled, and for years the streaming industry continued paying licensing fees for subpar codecs, even though better options were available.

The End of Shared Innovation

Meanwhile MalCorp found a way to tweak the law so its patents would never expire. It proposed a special amendment, just for patent pools, that said: Any time any part of any patent changes, the entire pool is treated as a new invention under U.S. law. With its deep pockets, MalCorp was able to buy the votes needed to get its law passed.

MalCorp’s patents would not expire. Not in 20 years. Not ever. And because patent law is about as interesting as copyright law, few protested the change.

Things went downhill quickly for advocates of the open web. MalCorp’s patents became broader, vaguer, ever-changing. With billions in its war chest, MalCorp was able to sue royalty-free codecs like AV1 out of existence. MalCorp had won. It had a monopoly on web streaming technology. It began, slowly at first, to raise licensing fees.

Gorgeous Video, Crushing Fees

For those who could afford it, web video got much better. MalCorp’s newest high-efficiency video codecs brought pixel-perfect 32K-Strato-Def images and 3D sound into people’s homes. Video and audio were clear and rich – better than real life. Downloads were fast. Images were crisp and spectacular. Fees were high.

Without access to any competing technologies, streaming companies had to pay billions instead of millions a year to MalCorp. Streaming services had to 100x their prices to cover their costs. Monthly fees rose to $4,500. Even students had to pay $50 a minute to watch a lecture on YouTube. Gradually, the world began to wake up to what MalCorp had done.

Life Indoors

By the mid-twenties, the Robotic Age had put most people out of work. The lucky ones lived on fixed incomes, paid by their governments. Humans were only needed for specialized service jobs, like nursery school teachers and style consultants. Even doctors were automated, using up-to-the-minute, crowd-sourced data to diagnose disease and track trends and outbreaks.

People were idle. Discontent was rising. Where once a retired workforce might have traveled or pursued hobbies, growing environmental problems rendered the outside world mostly uninhabitable. People hiked at home with their headsets on, enjoying stereoscopic birdsong and the idea of a fresh breeze. We lived indoors, in front of screens.

Locked In, Locked Out

It didn’t take long for MalCorp to become the most powerful corporation in the world. When video and mixed reality files made up 90 percent of all internet traffic, MalCorp was collecting on every transmission. Still, its greed kept growing.

Fed up with workarounds like piracy sites and peer-to-peer networks, MalCorp dismantled all legacy codecs. The slow, furry, lousy videos that were vaguely affordable ceased to function on modern networks and devices. People noticed when the signal went dark. Sure, there was still television and solid state media, but it wasn’t the same. Soon enough, all hell broke loose.

The Wars Begin

During Super Bowl LXII, football fans firebombed police stations in 70 cities, because listening to the game on radio just didn’t cut it. Thousands died in the riots and, later, in the crackdowns. Protesters picketed Disneyland, because the people had finally figured out what had happened to their democracy, and how it got started.

For the first time in years, people began to organize. They joined chat rooms and formed political parties like VidPeace and YouStream, vying for a majority. They had one demand: Give us back free video on the open web. They put banners on their vid-free Facebook feeds, advocating for the liberation of web video from greedy patent holders. They rallied around an inalienable right, once taken for granted, to be able to make and watch and share their own family movies, without paying MalCorp’s fees.

But it was too late. The opportunity to influence the chain of events had ended years before. Some say the tipping point was in 2019. Others blame the apathy and naiveté of early web users, who assumed tech companies and governments would always make decisions that served the common good. That capitalism would deliver the best services, in spite of powerful profit motives. And that the internet would always be free.

Original Link

Dweb: Social Feeds with Secure Scuttlebutt

In the series introduction, we highlighted the importance of putting people in control their social interactions online, instead of allowing for-profit companies be the arbiters of hate speech or harassment. Our first installment in the Dweb series introduces Secure Scuttlebutt, which envisions a world where users are in full control of their communities online.

In the weeks ahead we will cover a variety of projects that represent explorations of the decentralized/distributed space. These projects aren’t affiliated with Mozilla, and some of them rewrite the rules of how we think about a web browser. What they have in common: These projects are open source, and open for participation, and share Mozilla’s mission to keep the web open and accessible for all.

This post is written by André Staltz, who has written extensively on the fate of the web in the face of mass digital migration to corporate social networks, and is a core contributor to the Scuttlebutt project. –Dietrich Ayala

Getting started with Scuttlebutt

Scuttlebutt is a free and open source social network with unique offline-first and peer-to-peer properties. As a JavaScript open source programmer, I discovered Scuttlebutt two years ago as a promising foundation for a new “social web” that provides an alternative to proprietary platforms. The social metaphor of mainstream platforms is now a more popular way of creating and consuming content than the Web is. Instead of attempting to adapt existing Web technologies for the mobile social era, Scuttlebutt allows us to start from scratch the construction of a new ecosystem.

A local database, shared with friends

The central idea of the Secure Scuttlebutt (SSB) protocol is simple: your social account is just a cryptographic keypair (your identity) plus a log of messages (your feed) stored in a local database. So far, this has no relation to the Internet, it is just a local database where your posts are stored in an append-only sequence, and allows you to write status updates like you would with a personal diary. SSB becomes a social network when those local feeds are shared among computers through the internet or through local networks. The protocol supports peer-to-peer replication of feeds, so that you can have local (and full) copies of your friends’ feeds, and update them whenever you are online. One implementation of SSB, Scuttlebot, uses Node.js and allows UI applications to interact with the local database and the network stack.

Using Scuttlebot

While SSB is being implemented in multiple languages (Go, Rust, C), its main implementation at the moment is the npm package scuttlebot and Electron desktop apps that use Scuttlebot. To build your own UI application from scratch, you can setup Scuttlebot plus a localhost HTTP server to render the UI in your browser.

Run the following npm command to add Scuttlebot to your Node.js project:

npm install --save scuttlebot

You can use Scuttlebot locally using the command line interface, to post messages, view messages, connect with friends. First, start the server:

$(npm bin)/sbot server

In another terminal you can use the server to publish a message in your local feed:

$(npm bin)/sbot publish --type post --text "Hello world"

You can also consume invite codes to connect with friends and replicate their feeds. Invite codes are generated by pub servers
owned by friends in the community, which act as mirrors of feeds in the community. Using an invite code means the server will allow you to connect to it and will mirror your data too.

$(npm bin)/sbot invite.accept $INSERT_INVITE_CODE_HERE

To create a simple web app to render your local feed, you can start the scuttlebot server in a Node.js script (with dependencies ssb-config and pull-stream), and serve the feed through an HTTP server:

// server.js
const fs = require('fs');
const http = require('http');
const pull = require('pull-stream');
const sbot = require('scuttlebot/index').call(null, require('ssb-config')); http .createServer((request, response) => { if (request.url.endsWith('/feed')) { pull( sbot.createFeedStream({live: false, limit: 100}), pull.collect((err, messages) => { response.end(JSON.stringify(messages)); }), ); } else { response.end(fs.readFileSync('./index.html')); } }) .listen(9000);

Start the server with node server.js, and upon opening localhost:9000 in your browser, it should serve the index.html:

<html> <body> <script> fetch('/feed') .then(res => res.json()) .then(messages => { document.body.innerHTML = ` <h1>Feed</h1> <ul>${messages .filter(msg => msg.value.content.type === 'post') .map(msg => `<li>${msg.value.author} said: ${msg.value.content.text}</li>` ) }</ul> `; }); </script>
</body> </html>

Learn more

SSB applications can accomplish more than social messaging. Secure Scuttlebutt is being used for Git collaboration, chess games, and managing online gatherings.

You build your own applications on top of SSB by creating or using plug-ins for specialized APIs or different ways of querying the database. See secret-stack for details on how to build custom plugins. See flumedb for details on how to create custom indexes in the database. Also there are many useful repositories in our GitHub org.

To learn about the protocol that all of the implementations use, see the protocol guide, which explains the cryptographic primitives used, and data formats agreed on.

Finally, don’t miss the frontpage Scuttlebutt.nz, which explains the design decisions and principles we value. We highlight the important role that humans have in internet communities, which should not be delegated to computers.

Original Link

Things Gateway 0.5 packed full of new features, including experimental smart assistant

The Things Gateway from Mozilla lets you directly monitor and control your home over the web, without a middleman.

Today the Mozilla IoT team is excited to announce the 0.5 release of the Things Gateway, which is packed full of new features including customisable devices, a more powerful rules engine, an interactive floorplan and an experimental smart assistant you can talk to.

Customisable Things

Custom Capabilities

A powerful new “capabilities” system means that devices are no longer restricted to a predefined set of Web Thing Types, but can be assembled from an extensible schema-based system of “capabilities” through our new schema repository.

This means that developers have much more flexibility to create weird and wacky devices, and users have more control over how the device is used. So if you have a door sensor which also happens to be a temperature sensor, a smart plug which also has a multi-colour LED ring, or a whole bunch of sensors all in one device, you’re not limited by restrictive device types.

This also provides more flexibility to developers who want to build their own web things using the Things Framework, which now also has support for Rust, MicroPython and Arduino.

Custom Icons

When a user adds a device to the gateway they can now choose what main function they want to use it for and what icon is used to represent it.

Image showing the UI for choosing capabilities from a dropdown menu

You can even upload your own custom icon if you want to.

Image showing UI for selecting an image icon for different types of things

Custom Web Interface

In addition to the built-in UI the gateway generates for devices, web things can now provide a link to a custom web interface designed specifically for any given device. This is useful for complex or unusual devices like a robot or a “pixel wall” where a custom designed UI can be much more user friendly.

Image of UI showing examples of custom web interface icons you can create

Actions & Events

In addition to properties (like “on/off”, “level” and “color”), the gateway UI can now represent actions like “fade” which are triggered with a button and can accept input via a form.

Screenshot of UI for choosing different types of actions

Screenshot of UI for defining duration and level

The UI can also display an event log for a device.

Screenshot of event log UI

Powerful Rules Engine

The rules engine now supports rules with multiple inputs and multiple outputs. Simple rules are still just as easy to create, but more advanced rules can make use of “if”, “while”, “and”, “or” and “equals” operators to create more sophisticated automations through an intuitive drag and drop interface.

It’s also now possible to set colours and strings as outputs.

Interactive Floorplan

The floorplan view is even more useful now that you can view the status of devices and even control them from directly inside the floorplan. Simply tap things to turn them on and off, or long press to get to their detail view. This provides a helpful visual overview of the status of your whole smart home.

UI showing an interactive floorplan for monitoring your smart home

Smart Assistant Experiment

A feature we’re particularly excited about is a new smart assistant you can talk to via a chat style interface, either by typing or using your voice.

You can give it commands like “Turn the kitchen light on” and it will respond to you to confirm the action. So far it can understand a basic set of commands to turn devices on and off, set levels, set colours and set colour temperatures.

Image of the foxy smart assistant and examples of voice and text interactions

The smart assistant is still very experimental so it’s currently turned off by default, but you can enable it through Settings -> Smart Assistant UI.

UI for enabling the smart assistant

Other Changes

Other new features include developer settings which allow you to view system logs and enable/disable the gateway’s SSH server so you can log in via the command line.

UI showing developer settings panel

It’s also now much easier to rename devices and you can now also add devices that require a pin number to be entered during pairing.

How to Get Involved

To try out the latest version of the gateway, download the software image from our website to use on a Raspberry Pi. If you already have a gateway set up, you should notice it automatically update itself to the 0.5 release.

As always, we welcome your contributions to our open source project. You can provide feedback and ask questions on Discourse and file bugs and send pull requests on GitHub.

Happy hacking!

Original Link

Introducing the Dweb

Introducing the Dweb

The web is the most successful programming platform in history, resulting in the largest open and accessible collection of human knowledge ever created. So yeah, it’s pretty great. But there are a set of common problems that the web is not able to address.

Have you ever…

  • Had a website or app you love get updated to a new version, and you wished to go back to the old version?
  • Tried to share a file between your phone and laptop or tv or other device while not connected to the internet? And without using a cloud service?
  • Gone to a website or service that you depend on, only to find it’s been shut down? Whether it got bought and enveloped by some internet giant, or has gone out of business, or whatever, it was critical for you and now it’s gone.

Additionally, the web is facing critical internet health issues, seemingly intractable due to the centralization of power in the hands of a few large companies who have economic interests in not solving these problems:

  • Hate speech, harassment and other attacks on social networks
  • Repeated attacks on Net Neutrality by governments and corporations
  • Mass human communications compromised and manipulated for profit or political gain
  • Censorship and whole internet shutdowns by governments

These are some of the problems and use-cases addressed by a new wave of projects, products and platforms building on or with web technologies but with a twist: They’re using decentralized or distributed network architectures instead of the centralized networks we use now, in order to let the users control their online experience without intermediaries, whether government or corporate. This new structural approach gives rise to the idea of a ‘decentralized web’, often conveniently shortened to ‘dweb’.

You can read a number of perspectives on centralization, and why it’s an important issue for us to tackle, in Mozilla’s Internet Health Report, released earlier this year.

What’s the “D” in Dweb?!

The “d” in “dweb” usually stands for either decentralized or distributed.
What is the difference between distributed vs decentralized architectures? Here’s a visual illustration:

visual representation of centralized, decentralized, and distributed networks
(Image credit: Openclipart.org, your best source for technical clip art with animals)

In centralized systems, one entity has control over the participation of all other entities. In decentralized systems, power over participation is divided between more than one entity. In distributed systems, no one entity has control over the participation of any other entity.

Examples of centralization on the web today are the domain name system (DNS), servers run by a single company, and social networks designed for controlled communication.

A few examples of decentralized or distributed projects that became household names are Napster, BitTorrent and Bitcoin.

Some of these new dweb projects are decentralizing identity and social networking. Some are building distributed services in or on top of the existing centralized web, and others are distributed application protocols or platforms that run the web stack (HTML, JavaScript and CSS) on something other than HTTP. Also, there are blockchain-based platforms that run anything as long as it can be compiled into WebAssembly.

Here We Go

Mozilla’s mission is to put users in control of their experiences online. While some of these projects and technologies turn the familiar on its head (no servers! no DNS! no HTTP(S)!), it’s important for us to explore their potential for empowerment.

This is the first post in a series. We’ll introduce projects that cover social communication, online identity, file sharing, new economic models, as well as high-level application platforms. All of this work is either decentralized or distributed, minimizing or entirely removing centralized control.

You’ll meet the people behind these projects, and learn about their values and goals, the technical architectures used, and see basic code examples of using the project or platform.

So leave your assumptions at the door, and get ready to learn what a web more fully in users’ control could look like.

Original Link

9 Biggest Mistakes with CSS Grid

It’s easy to make lots of mistakes with a new technology, especially something that’s as big of a change from the past as CSS Grid. In this video, I explain the 9 Biggest Mistakes people are making, with advice and tips for avoiding these pitfalls and breaking old habits.

For more information:

Mistake 1: Thinking CSS Grid is Everything
Flexbox vs. CSS Grid — Which is Better?
Using Flexbox & Grid Together
Obliterate Boxiness with CSS Shapes

Mistake 2: Using Only Percents for Sizing
Min & Max Content Sizing in CSS Grid
FR Units in CSS Grid
MinMax in CSS Grid

Mistake 3 : Assuming You Need Breakpoints
Incredibly Easy Layouts with CSS Grid

Mistake 4: Getting Confused by Numbering
Innovative & Practical Graphic Design with CSS Grid
Basics of CSS Grid: The Big Picture

Mistake 5: Always Using 12-columns
I talk about this towards the end of “FR Units in CSS Grid”

Mistake 6: Ignoring the Power of Rows
Flexibility & The Fold
Whitespace on The Web

Mistake 7: Reaching for a Framework

Mistake 8: Waiting for IE11 to Die
Internet Explorer + CSS Grid?
7-part Series on Writing Resilient CSS that works in all browsers

Mistake 9: Hesitating, Instead of Playing
Responsive Mondrian
CSS Grid like you are Jan Tschichold

Jen Simmons is a Designer and Developer Advocate at Mozilla, specializing in CSS and layout design on the web. She is a member of the CSS Working Group, and speaks at many conferences including An Event Apart, SXSW, Fluent, Smashing Conf and more.

More articles by Jen Simmons…

Original Link

360° Images on the Web, the Easy Way

One of the most popular uses for VR today is 360° images and video. 360° images are easy to discover and share online, and you don’t need to learn any new interactions to explore the 360° experience.

Building 360° views is not as easy as exploring them, especially if you want to make an experience where the viewer can navigate from scene to scene. Here is the solution I came up with using A-Frame, a web framework for building virtual reality experiences and Glitch, a creative community platform for building, remixing and hosting web apps and sites.

I often teach students at my local public library. I have found the combination of A-Frame and Glitch to be ideal, especially for the younger learners. A-Frame lets you write markup that feels like HTML to produce 3D content. You don’t have to write any JS code if you don’t want to. And Glitch is wonderful because I can give my students a sample project that they then ‘remix’ to create their own version. Thinking about it, ‘remix’ is probably a better word for non-programmers than ‘fork’.

This is the project I start them with. The 360 Workshop. If you load it up you’ll see me in my office with a ridiculous grin on my face and a strange sphere on my forehead.

If you click the headset icon in the lower right you can enter VR. If you use the fish icon in the upper right you can create your remix. If you move the gaze cursor to the pale blue sphere you will jump to a new scene.

This is an example of what I call a 360° tour.

360 Tour

A 360° tour is a web-based experience where you view 3D imagery from a single vantage point. You can rotate around and look up, but you can’t move forward or back. The tour is made up of scenes. To navigate you click on hot spots to jump from scene to scene. You can think of this as the modern equivalent of the old multi-media CD-ROM edutainment apps from the mid-90s.

A Myst CD-ROM from the '90s

The Code

The code is pretty simple. It’s stock A-Frame with one custom component. You load your images in the a-assets section. To create a ‘scene’ make an entity with a screen component (I wanted to call it scene instead of screen but it would have been too confusing with the main a-scene component).

Inside the screen entity put an a-sky component for the 360° image, then add spheres, or cubes or whatever else you want for the hotspots. The hotspots should have a to-screen component which names the scene to jump to.

<a-entity screen="shed"> <a-sky src="#shed" rotation="0 90 0"></a-sky> <a-sphere color="yellow" radius="0.2" position="2 0 4" to-screen="patio" ></a-sphere>
</a-entity>

You can make as many screens as you want, then wrap them all in a single entity with a screen-switch component. Set the screen property to whichever scene you wish to load first.

<a-entity screen-switch="screen:scene3"> <a-entity screen="scene1">
... <a-entity screen="scene2">
... <a-entity screen="scene3">
...
</a-entity>

The underlying switcher component is included in the Glitch project. You can look at the switcher.js file, but the code is pretty boring. It just toggles the visibility of each screen.

That’s it. That is all you need for 360° experiences.

How about some of my own images?

Oh, you wanted to use your own images? Ah, well good news. That’s easy too.

You can find free reusable 360° images on the web by searching for equirectangular and creative commons. Here’s a bunch on Flickr.

If you want to make your own, you can get a 360° camera. There are several good brands. I’ve been using the Ricoh Theta V. It can connect to your computer via USB, but you can also control it via the Ricoh mobile app. The app is wonderful for checking the scene and adjusting the tripod before your shoot.

An image of the Ricoh Theta V

Ricoh Theta V for 360 images and video

The Theta V can shoot both still images and video.

For shooting images I first suggest you get a tripod. The camera essentially has two 180˚ lenses that are stitched with clever software into a single 360˚ equirectangular image. The stitching at the top and sides is quite good, as long as you don’t have anything closer than about two feet from the camera. For most directions this is fine, but since you have to hold the camera with your hands you’ll see this weird mush of thumbs and fingers at the bottom.

If you use a tripod the bottom mush will be almost invisible. Plus tripods are just a good idea in general to get crisp sharp photos. You can use the mobile app to view the scene and snap the shutter once you get out of the camera view (ie, hide behind a door).

Since these images are typically viewed in a headset you should place the camera at roughly head height, or 1 to 1.5 meters. On the other hand, if you want a fun ‘miniaturized’ effect put the camera on the floor (or a tiny tripod) and place LEGO buildings around it.

What about Video?

Ah yes. Video. The Theta V can do video, at surprisingly high resolution. Technically you can even stream the video over WiFi but the resolution is lower. If you really need to stream you’ll want to use a USB cable and do the encoding on a laptop.

For non-streaming video you can just upload the MP4 files to your web-server and include the resource in the 360° tour with a video tag. Then just add the video to a scene with the <a-videosphere> element. It will work.

However….

360° video files are big. Really big. Like a gigabyte per minute big. So while it’s possible to just upload mp4 files to a web-server, anyone who wants to view your experience is going to be in for a long and possibly expensive download. To do 360° video right you’ll need to use a service that specializes in streaming video, like Wowza or Vimeo. They can progressively stream the video and adjust the bitrate dynamically. If you do go with Vimeo check out this github project for how to do it.

360° It!

I hope you’ve enjoyed this introductory tour of what we can do with 360° images. You can build your own by remixing this Glitch. I have added a few creative commons images to my assets repo; these images are licensed for you to reuse and remix. When you want to go beyond spheres you can read the full A-Frame documentation, which explains how to use other geometry, add text, and build your own components.

If you want to learn more about how to make VR and AR on the web I stream periodically on Twitch and post lots of links to Twitter. Enjoy.

I am an author, researcher, and recovering engineer. Formerly on the Swing Team at Sun, the webOS team at Palm, and Nokia Research. I spread the word of good user experiences. I live in sunny Eugene Oregon with my wife and genius Lego builder child.

More articles by Josh Marinacci…

Original Link

AV1: next generation video – The Constrained Directional Enhancement Filter

AV1

For those just joining us….
AV1 is a new general-purpose video codec developed by the Alliance for Open Media. The alliance began development of the new codec using Google’s VPX codecs, Cisco’s Thor codec, and Mozilla’s/Xiph.Org’s Daala codec as a starting point. AV1 leapfrogs the performance of VP9 and HEVC, making it a next-next-generation codec . The AV1 format is and will always be royalty-free with a permissive FOSS license.

This post was written originally as the second in an in-depth series of posts exploring AV1 and the underlying new technologies deployed for the first time in a production codec. An earlier post on the Xiph.org website looked at the Chroma from Luma prediction feature. Today we cover the Constrained Directional Enhancement Filter. If you’ve always wondered what goes into writing a codec, buckle your seat-belts, and prepare to be educated!

Filtering in AV1

Virtually all video codecs use enhancement filters to improve subjective output quality.

By ‘enhancement filters’ I mean techniques that do not necessarily encode image information or improve objective coding efficiency, but make the output look better in some way. Enhancement filters must be used carefully because they tend to lose some information, and for that reason they’re occasionally dismissed as a deceptive cheat used to make the output quality look better than it really is.

But that’s not fair. Enhancement filters are designed to mitigate or eliminate specific artifacts to which objective metrics are blind, but are obvious to the human eye. And even if filtering is a form of cheating, a good video codec needs all the practical, effective cheats it can deploy.

Filters are divided into multiple categories. First, filters can be normative or non-normative. A normative filter is a required part of the codec; it’s not possible to decode the video correctly without it. A non-normative filter is optional.

Second, filters are divided according to where they’re applied. There are preprocessing filters, applied to the input before coding begins, postprocessing filters applied to the output after decoding is complete, and in-loop or just loop filters that are an integrated part of the encoding process in the encoding loop. Preprocessing and postprocessing filters are usually non-normative and external to a codec. Loop filters are normative almost by definition and part of the codec itself; they’re used in the coding optimization process, and applied to the reference frames stored or inter-frame coding.

a diagram of AV1 coding loop filters AV1 uses three normative enhancement filters in the coding loop. The first, the deblocking filter, does what it says; it removes obvious bordering artifacts at the edges of coded blocks. Although the DCT is relatively well suited to compacting energy in natural images, it still tends to concentrate error at block edges. Remember that eliminating this blocking tendency was a major reason Daala used a lapped transform, however AV1 is a more traditional codec with hard block edges. As a result, it needs a traditional deblocking filter to smooth the block edge artifacts away.

An example of blocking artifacts in a traditional DCT block-based codec. Errors at the edges of blocks are particularly noticeable as they form hard edges. Worse, the DCT (and other transforms in the DCT family) tend to concentrate error at block edges, compounding the problem.

The last of the three filters is the Loop Restoration filter. It consists of two configurable and switchable filters, a Wiener filter and a Self-Guided filter. Both are convolving filters that try to build a kernel to restore some lost quality of the original input image and are usually used for denoising and/or edge enhancement. For purposes of AV1, they’re effectively general-purpose denoising filters that remove DCT basis noise via a configurable amount of blurring.

The filter between the two, the Constrained Directional Enhancement Filter (CDEF) is the one we’re interested in here; like the loop restoration filter, it removes ringing and basis noise around sharp edges, but unlike the loop restoration filter, it’s directional. It can follow edges, as opposed to blindly filtering in all directions like most filters. This makes CDEF especially interesting; it’s the first practical and useful directional filter applied in video coding.

The Long and Winding Road

The CDEF story isn’t perfectly linear; it’s long and full of backtracks, asides, and dead ends. CDEF brings multiple research paths together, each providing an idea or an inspiration toward the final Constrained Directional Enhancement Filter in AV1. The ‘Directional’ aspect of CDEF is especially novel in implementation, but draws ideas and inspiration from several different places.

The whole point of transforming blocks of pixel data using the DCT and DCT-like transforms is to represent that block of pixels using fewer numbers. The DCT is pretty good at compacting the energy in most visual images, that is, it tends to collect spread out pixel patterns into just a few important output coefficients.

There are exceptions to the DCT’s compaction efficiency. To name the two most common examples, the DCT does not represent directional edges or patterns very well. If we plot the DCT output of a sharp diagonal edge, we find the output coefficients also form…. a sharp diagonal edge! The edge is different after transformation, but it’s still there and usually more complex than it started. Compaction defeated!

a sharp edge (left) and its DCT transform coefficients (right) illustrating the problem with sharp features

Sharp features are a traditional problem for DCT-based codecs as they do not compact well, if at all. Here we see a sharp edge (left) and its DCT transform coefficients (right). The energy of the original edge is spread through the DCT output in a directional rippling pattern.

Over the past two decades, video codec research has increasingly looked at transforms, filters, and prediction methods that are inherently directional as a way of better representing directional edges and patterns, and correcting this fundamental limitation of the DCT.

Classic Directional Predictors

Directional intra prediction is probably one of the best known directional techniques used in modern video codecs. We’re all familiar with h.264’s and VP9’s directional prediction modes, where the codec predicts a directional pattern into a new block, based on the surrounding pixel pattern of already decoded blocks. The goal is to remove (or greatly reduce) the energy contained in hard, directional edges before transforming the block. By predicting and removing features that can’t be compacted, we improve the overall efficiency of the codec.

AVC/H.264 intra prediction modes, illustrating modes 0-8.

Illustration of directional prediction modes available in AVC/H.264 for 4×4 blocks. The predictor extends values taken from a one-pixel-wide strip of neighboring pixels into the predicted block in one of eight directions, plus an averaging mode for simple DC prediction.

Motion compensation, an even older idea, is also a form of directional prediction, though we seldom think of it that way. It displaces blocks in specific directions, again to predict and remove energy prior to the DCT. This block displacement is directional and filtered, and like directional intra-prediction, uses carefully constructed resampling filters when the displacement isn’t an integer number of pixels.

Directional Filters

As noted earlier, video codecs make heavy use of filtering to remove blocking artifacts and basis noise. Although the filters work on a 2D plane, the filters themselves tend to be separable, that is, they’re usually 1D filters that are run horizontally and vertically in separate steps.

Directional filtering attempts to run filters in directions besides just horizontal and vertical. The technique is already common in image processing, where noise removal and special effects filters are often edge- and direction-aware. However, these directional filters are often based on filtering the output of directional transforms, for example, the [somewhat musty] image denoising filters I wrote based on dual-tree complex wavelets.

The directional filters in which we’re most interested for video coding need to work on pixels directly, following along a direction, rather than filtering the frequency-domain output of a directional transform. Once you try to design such a beast, you quickly hit the first Big Design Question: how do you ‘follow’ directions other than horizontal and vertical, when your filter tap positions no longer land squarely on pixels arranged in a grid?

One possibility is the classic approach used in high-quality image processing: transform the filter kernel and resample the pixel space as needed. One might even argue this is the only ‘correct’ or ‘complete’ answer. It’s used in subpel motion compensation, which cannot get good results without at least decent resampling, and in directional prediction which typically uses a fast approximation.

That said, even a fast approximation is expensive when you don’t need to do it, so avoiding the resampling step is a worthy goal if possible. The speed penalty is part of the reason we’ve not seen directional filtering in video coding practice.

Directional Transforms

Directional transforms attempt to fix the DCT’s edge compaction problems in the transform itself.

Experimental directional transforms fall into two categories. There are the transforms that use inherently directional bases, such as directional wavelets. These transforms tend to be oversampled/overcomplete, that is, they produce more output data than they take input data— usually massively more. That might seem like working backwards; you want to reduce the amount of data, not increase it! But these transforms still compact the energy, and the encoder still chooses some small subset of the output to encode, so it’s really no different from usual lossy DCT coding. That said, overcomplete transforms tend to be expensive in terms of memory and computation, and for this reason, they’ve not taken hold in mainstream video coding.

The second category of directional transform takes a regular, non-directional transform such as the DCT, and modifies it by altering the input or output. The alteration can be in the form of resampling, a matrix multiplication (which can be viewed as a specialized form of resampling), or juggling of the order of the input data.

It’s this last idea that’s the most powerful, because it’s fast. There’s no actual math to do when simply rearranging numbers.

A few practical complications make implementation tricky. Rearranging a square to make a diagonal edge into a [mostly] vertical or horizontal line results in a non-square matrix of numbers as an input. Conceptually, that’s not a problem; the 2D DCT is separable, and since we can run the row and column transforms independently, we can simply use different sized 1D DCTs for each length row and column, as in the figure above. In practice this means we’d need a different DCT factorization for every possible column length, and shortly after realizing that, the hardware team throws you out a window.

There are also other ways of handling the non-squareness of a rearrangement, or coming up with resampling schemes that keep the input square or only operate on the output. Most of the directional transform papers mentioned below are concerned with the various schemes for doing so.

But here’s where the story of directional transforms mostly ends for now. Once you work around the various complications of directional transforms and deploy something practical, they don’t work well in a modern codec for an unexpected reason: They compete with variable blocksize for gains. That is, in a codec with a fixed blocksize, adding directional transforms alone gets impressive efficiency gains. Adding variable blocksize alone gets even better gains. Combining variable blocksize and directional transforms gets no benefit over variable blocksize alone. Variable blocksize has already effectively eliminated the same redundancies exploited by directional transforms, at least the ones we currently have, and done a better job of it.

Nathan Egge and I both experimented extensively with directional transforms during Daala research. I approached the problem from both the input and output side, using sparse matrix multiplications to transform the outputs of diagonal edges into a vertical/horizontal arrangement. Nathan ran tests on mainstream directional approaches with rearranged inputs. We came to the same conclusion: there was no objective or subjective gain to be had for the additional complexity.

Directional transforms may have been a failure in Daala (and other codecs), but the research happened to address a question posed earlier: How to filter quickly along edges without a costly resampling step? The answer: don’t resample. Approximate the angle by moving along the nearest whole pixel. Approximate the transformed kernel by literally or conceptually rearranging pixels. This approach introduces some aliasing, but it works well enough, and it’s fast enough.

Directional Predictors, part 2: The Daala Chronicles

The Daala side of the CDEF story began while trying to do something entirely different: normal, boring, directional intra-prediction. Or at least what passed for normal in the Daala codec.

I wrote about Daala’s frequency-domain intra prediction scheme at the time we were just beginning to work on it. The math behind the scheme works; there was never any concern about that. However, a naive implementation requires an enormous matrix multiplication that was far too expensive for a production codec. We hoped that sparsifying— eliminating matrix elements that didn’t contribute much to the prediction— could reduce the computational cost to a few percent of the full multiply.

Sparsification didn’t work as hoped. At least as we implemented it, sparsification simply lost too much information for the technique to be practical.

Of course, Daala still needed some form of intra-prediction, and Jean-Marc Valin had a new idea: A stand-alone prediction codec, one that worked in the spatial domain, layered onto the frequency-domain Daala codec. As a kind of symbiont that worked in tandem with but had no dependencies on the Daala codec, it was not constrained by Daala’s lapping and frequency domain requirements. This became Intra Paint.

A photo of Sydney Harbor with some interesting painting-like features created by the algorithm

An example of the Intra Paint prediction algorithm as applied to a photograph of Sydney Harbor. The visual output is clearly directional and follows the edges and features in the original image well, producing a pleasing (if somewhat odd) result with crisp edges.

The way intra paint worked was also novel; it coded 1-dimensional vectors along only the edges of blocks, then swept the pattern along the selected direction. It was much like squirting down a line of colored paint dots, then sweeping the paint in different directions across the open areas.

Intra paint was promising and produced some stunningly beautiful results on its own, but again wasn’t efficient enough to work as a standard intra predictor. It simply didn’t gain back enough bits over the bits it had to use to code its own information.

A gray image showing areas of difference between the Sydney Harbor photo and the Intra-Paint result

Difference between the original Sydney Harbor photo and the Intra Paint result. Despite the visually pleasing output of Intra Paint, we see that it is not an objectively super-precise predictor. The difference between the original photo and the intra-paint result is fairly high, even along many edges that it appeared to reproduce well.

The intra paint ‘failure’ again planted the seed of a different idea; although the painting may not be objectively precise enough for a predictor, much of its output looked subjectively quite good. Perhaps the paint technique could be used as a post-processing filter to improve subjective visual quality? Intra paint follows strong edges very well, and so could potentially be used to eliminate basis noise that tends to be strongest along the strongest edges. This is the idea behind the original Daala paint-deringing filter, which eventually leads to CDEF itself.

There’s one more interesting mention on the topic of directional prediction, although it too is currently a dead-end for video coding. David Schleef implemented an interesting edge/direction aware resampling filter called Edge-Directed Interpolation (EDI). Other codecs (such as the VPx series and for a while AV1) have experimented with downsampled reference frames, transmitting the reference in a downsampled state to save coding bits, and then upsampling the reference for use at full resolution. We’d hoped that much-improved upsampling/interpolation provided by EDI would improve the technique to the point it was useful. We also hoped to use EDI as an improved subpel interpolation filter for motion compensation. Sadly, those ideas remain an unfulfilled dream.

Bridging the Gap, Merging the Threads

At this point, I’ve described all the major background needed to approach CDEF, but chronologically the story involves some further wandering in the desert. Intra paint gave rise to the original Daala paint-dering filter, which reimplemented the intra-paint algorithm to perform deringing as a post-filter. Paint-dering proved to be far too slow to use in production.

As a result, we packed up the lessons we learned from intra paint and finally abandoned the line of experimentation. Daala imported Thor’s CLPF for a time, and then Jean-Marc built a second, much faster Daala deringing filter based on the intra-paint edge direction search (which was fast and worked well) and a Conditional Replacement Filter. The CRF is inspired somewhat by a median filter and produces results similar to a bilateral filter, but is inherently highly vectorizable and therefore much faster.

A series of graphs showing the original signal and the effects of various filters

Demonstration of a 7-tap linear filter vs the constrained replacement filter as applied to a noisy 1-dimensional signal, where the noise is intended to simulate the effects of quantization on the original signal.

The final Daala deringing filter used two 1-dimensional CRF filters, a 7-tap filter run in the direction of the edge, and a weaker 5-tap filter run across it. Both filters operate on whole pixels only, performing no resampling. At that point, the Daala deringing filter began to look a lot like what we now know as CDEF.

We’d recently submitted Daala to AOM as an input codec, and this intermediate filter became the AV1 daala_dering experiment. Cisco also submitted their own deringing filter, the Constrained Low-Pass Filter (CLPF) from the Thor codec. For some time the two deringing filters coexisted in the AV1 experimental codebase where they could be individually enabled, and even run together. This led both to noticing useful synergies in their operation, as well as additional similarities in various stages of the filters.

And so, we finally arrive at CDEF: The merging of Cisco’s CLPF filter and the second version of the Daala deringing filter into a single, high-performance, direction-aware deringing filter.

Modern CDEF

The CDEF filter is simple and bears a deep resemblance to our preceding filters. It is built out of three pieces (directional search, the constrained replacement/lowpass filter, and integer-pixel tap placement) that we’ve used before. Given the lengthy background preamble to this point, you might almost look at the finished CDEF and think, “Is that it? Where’s the rest?” CDEF is an example of gains available by getting the details of a filter exactly right as opposed to just making it more and more complex. Simple and effective is a good place to be.

Direction search

CDEF operates in a specific direction, and so it is necessary to determine that direction. The search algorithm used is the same as from intra paint and paint-dering, and there are eight possible directions.

filtering direction with discrete lines of operation for each direction

The eight possible filtering directions of the current CDEF filter. The numbered lines in each directional block correspond to the ‘k’ parameter within the direction search.

We determine the filter direction by making “directional” variants of the input block, one for each direction, where all of the pixels along a line in the chosen direction are forced to have the same value. Then we pick the direction where the result most closely matches the original block. That is, for each direction d, we first find the average value of the pixels in each line k, and then sum, along each line, the squared error between a given pixel value and the average value of that pixel line.

Example illustrating determination of CDEF direction

An example process of selecting the direction d that best matches the input block. First we determine the average pixel value for each line of operation k for each direction. This is illustrated above by setting each pixel of a given line k to that average value. Then, we sum the error for a given direction, pixel by pixel, by subtracting the input value from the average value. The direction with the lowest error/variance is selected as the best direction.

This gives us the total squared error, and the lowest total squared error is the direction we choose. Though the pictured example above does so, there’s no reason to convert the squared error to variance; each direction considers the same number of pixels, so both will choose the same answer. Save the extra division!

This is the intuitive, long-way-around to compute the directional error. We can simplify the mechanical process down to the following equation: equation for determining filter directionIn this equation, E is the error, p is a pixel, xp is the value of a pixel, k is one of the numbered lines in the directional diagram above, and Nd,k is the cardinality of (the number of pixels in) the line k for direction d. This equation can be simplified in practice; for example the first term is the same for each given d. In the end, the AV1 implementation of CDEF currently requires 5.875 additions and 1.9375 multiplications per pixel and can be deeply vectorized, resulting in a total cost less than an 8×8 DCT.

Filter taps

The CDEF filter works pixel-by-pixel across a full block. The direction d selects the specific directional filter to be used, each consisting of a set of filter taps (that is, input pixel locations) and tap weights.

CDEF conceptually builds a directional filter out of two 1-dimensional filters. A primary filter is run along the chosen direction, like in the Daala deringing filter. The secondary filter is run twice in a cross-pattern, at 45° angles to the primary filter, like in Thor’s CLPF.

Illustration of primary and secondary filter directions and taps overlaid on top of CDEF filter direction

Illustration of primary and secondary 1-D filter directionality in relation to selected direction d. The primary filter runs along the selected filter direction, the secondary filters run across the selected direction at a 45° angle. Every pixel in the block is filtered identically.

The filters run at angles that often place the ideal tap locations between pixels. Rather than resampling, we choose the nearest exact pixel location, taking care to build a symmetric filter kernel.

Each tap in a filter also has a fixed weight. The filtering process takes the input value at each tap, applies the constraint function, multiplies the result by the tap’s fixed weight, and then adds this output value to the pixel being filtered.

illustration of primary and secondary taps

Primary and secondary tap locations and fixed weights (w) by filter direction. For primary taps and even Strengths a = 2 and b = 4, whereas for odd Strengths a = 3 and b = 3. The filtered pixel in shown in gray.

In practice, the primary and secondary filters are not run separately, but combined into a single filter kernel that’s run in one step.

Constraint function

CDEF uses a constrained low-pass filter in which the value of each filter tap is first processed through a constraint function parameterized by the difference between the tap value and pixel being filtered d, the filter strength S, and the filter damping parameter D: The constraint function is designed to deemphasize or outright reject consideration of pixels that are too different from the pixel being filtered. Tap value differences within a certain range from the center pixel value (as set by the Strength parameter S) are wholly considered. Value differences that fall between the Strength and Damping parameters are deemphasized. Finally, tap value differences beyond the Damping parameter are ignored.

An illustration of the constraint function

An illustration of the constraint function. In both figures, the difference (d) between the center pixel and the tap pixel being considered is along the x axis. The output value of the constraint function is along y. The figure on the left illustrates the effect of varying the Strength (S) parameter. The figure on the right demonstrates the effect of varying Damping (D).

The output value of the constraint function is then multiplied by the fixed weight associated with each tap position relative to the center pixel. Finally the resulting values (one for each tap) are added to the center filtered pixel, giving us the final, filtered pixel value. It all rolls up into:…where the introduced (p) and (s) mark values for the primary and secondary sets of taps.

There are a few additional implementation details regarding rounding and clipping not needed for understanding; if you’re intending to implement CDEF they can of course be found in the full CDEF paper.

Results

CDEF is intended to remove or reduce basis noise and ringing around hard edges in an image without blurring or damaging the edge. As used in AV1 right now, the effect is subtle but consistent. It may be possible to lean more heavily on CDEF in the future.

An example illustrating application of CDEF to a picture with ringing artifacts

An example of ringing/basis noise reduction in an encode of the image Fruits. The first inset closeup shows the area without processing by CDEF, the second inset shows the same area after CDEF.

The quantitative value of any enhancement filter must be determined via subjective testing. Better objective metrics numbers as well wouldn’t exactly be shocking, but the kind of visual improvements that motivate CDEF are mostly outside the evaluation ability of primitive objective testing tools such as PSNR or SSIM.

As such, we conducted multiple rounds of subjective testing, first during the development of CDEF (when Daala dering and Thor CLPF were still technically competitors) and then more extensive testing of the merged CDEF filter. Because CDEF is a new filter that isn’t present at all in previous generations of codecs, testing primarily consisted of AV1 with CDEF enabled, vs AV1 without CDEF.

A series of graphs showing test results of AV1 with and without CDEF

Subjective A-B comparison results (with ties) for CDEF vs. no CDEF for the high-latency configuration.

Subjective results show a statistically significant (p<.05) improvement for 3 out of 6 clips. This normally corresponds to a 5-10% improvement in coding efficiency, a fairly large gain for a single tool added to an otherwise mature codec.

Objective testing, as expected, shows more modest improvements of approximately 1%, however objective testing is primarily useful only insofar as it agrees with subjective results. Subjective testing is the gold standard, and the subjective results are clear.

Testing also shows that CDEF performs better when encoding with fewer codec ‘tools’; like directional transforms, CDEF is competing for coding gains with other, more-complex techniques within AV1. As CDEF is simple, small, and fast, it may provide future means to reduce the complexity of AV1 encoders. In terms of decoder complexity, CDEF represents between 3% and 10% of the AV1 decoder depending on the configuration.

Additional Resources

  1. Xiph.Org’s standard ‘derf’ test sets, hosted at media.xiph.org
  2. Automated testing harness and metrics used by Daala and AV1 development: Are We Compressed Yet?
  3. The AV1 Constrained Directional Enhancement Filter (CDEF)
    Steinar Midtskogen, Jean-Marc Valin, October 2017
  4. CDEF Presentation Slide Deck for ICASSP 2018, Steinar Midtskogen, Jean-Marc Valin
  5. A Deringing Filter for Daala and Beyond, Jean-Marc Valin
    This is an earlier deringing filter developed during research for the Daala codec that contributed to the CDEF used in AV1.
  6. Daala: Painting Images For Fun and Profit, Jean-Marc Valin
    A yet earlier intra-paint-based enhancement filter that led to the Daala deringing filter, which in turn led to CDEF
  7. Intra Paint Deringing Filter, Jean-Marc Valin 2015
    Notes on the enhancement/deringing filter built out of the Daala Intra Paint prediction experiment
  8. Guided Image Filtering Kaiming He, Jian Sun, Xiaoou Tang, 2013
  9. Direction-Adaptive Discrete Wavelet Transform for Image Compression, Chuo-Ling Chang, Bernd Girod, IEEE Transactions on Image Processing, vol. 16, no. 5, May 2007
  10. Direction-adaptive transforms for image communication, Chuo-Ling Chang, Stanford PhD dissertation 2009
    This dissertation presents a good summary of the state of the art of directional transforms in 2009; sadly it appears there are no online-accessible copies.
  11. Direction-Adaptive Partitioned Block Transform for Color Image Coding, Chuo-Ling Chang, Mina Makar, Sam S. Tsai, Bernd Girod, IEEE Transactions on Image Processing, vol. 19, no. 7, July 2010
  12. Pattern-based Assembled DCT scheme with DC prediction and adaptive mode coding, Zhibo Chen, Xiaozhong Xu
    Note this paper is behind the IEEE paywall
  13. Direction-Adaptive Transforms for Coding Prediction Residuals, Robert A. Cohen, Sven Klomp, Anthony Vetro, Huifang Sun, Proceedings of 2010 IEEE 17th International Conference on Image Processing, September 26-29, 2010, Hong Kong
  14. An Orientation-Selective Orthogonal Lapped Transform, Dietmar Kunz 2008
    Note this paper is behind the IEEE paywall.
  15. Rate-Distortion Analysis of Directional Wavelets, Arian Maleki, Boshra Rajaei, Hamid Reza Pourreza, IEEE Transactions on Image Processing, vol. 21, no. 2, February 2012
  16. Theoretical Analysis of Trend Vanishing Moments for
    Directional Orthogonal Transforms
    Shogo Murumatsu, Dandan Han, Tomoya Kobayashi, Hisakazu Kikuchi
    Note that this paper is behind the IEEE paywall. However a ‘poster’ version of the paper is freely available.
  17. An Overview of Directional Transforms in Image Coding,
    Jizheng Xu, Bing Zeng, Feng Wu
  18. Directional Filtering Transform for Image/Intra-Frame Compression, Xiulian Peng, Jizheng Xu, Feng Wu, IEEE Transaction in Image Processing, Vol. 19, No. 11, November 2010
    Note that this paper is behind the IEEE paywall.
  19. Approximation and Compression with Sparse
    Orthonormal Transforms
    , O. G. Sezer, O. G. Guleryuz, Yucel Altunbasak, 2008
  20. Robust Learning of 2-D Separable Transforms for Next-Generation Video Coding O. G. Sezer, R. Cohen, A. Vetro, March 2011
  21. Joint sparsity-based optimization of a set of orthonormal 2-D separable block transforms, Joel Sole, Peng Yin, Yunfei Zheng, Cristina Gomila, 2009
    Note that this paper is behind the IEEE paywall.
  22. Directional
    Lapped Transforms for Image Coding
    ,Jizheng Xu, Feng Wu, Jie Liang, Wenjun Zhang, IEEE Transactions on Image Processing, April 2008
  23. Directional Discrete Cosine Transforms—A New
    Framework for Image Coding
    ,Bing Zeng, Jingjing Fu, IEEE Transactions on Circuits and Systems for Video Technology, April 2008
  24. The Dual-Tree Complex Wavelet Transform, Ivan W. Selesnick, Richard G. Baraniuk, and Nick G. Kingsbury, IEEE Signal Processing Magazine, November 2005

More articles by Christopher Montgomery…

Original Link

Firefox 61 – Quantum of Solstice

Summertime1, and the browser is speedy! Firefox 61 is now available, and with it come new performance improvements that make the fox faster than ever! Let’s take a short tour of the highlights:

Parallel CSS Parsing

a road sign that says "parallel parsing only"adapted from mjfmjfmjf on flickr

Quantum CSS boosted our style system by calculating computed styles in parallel. Firefox 61 adds even more power to Quantum CSS by also parallelizing the parsing step! The extra horsepower pays real dividends on sites with large stylesheets and complex layouts.

Retained Display Lists

One of the final steps before a page is painted onto the screen is building a list of everything that is going to be drawn, from lowest z-order to highest (“back to front”). In Firefox, this is called the display list, and it’s the final chance to determine what’s on screen before painting begins. The display list contains backgrounds, borders, box shadows, text, everything that’s about to be pixels on the screen.

Historically, the entire display list was computed prior to every new paint. This meant that if a 60fps animation was running, the display list would be computed 60 times a second. For complex pages, this gets to be costly and can lead to reduced script execution budget and, in severe cases, dropped frames. Firefox 61 enables “retained” display lists, which, as the name would suggest, are retained from paint to paint. If only a small amount of the page has changed, the renderer may only have to recompute a small portion of the display list. With retained display lists enabled, the graphics team has observed a near 40% reduction in dropped frames due to list building! We’ve decided to pass those savings on to you, and are enabling the initial work in Firefox 61.

You can dive deeper into display lists in Matt Woodrow’s recent post here on Hacks.

Pretty snazzy stuff, but there’s more than just engine improvements! Here are few other new things to check out in this release:

Accessibility Inspector

A great website is one that works for everyone! The web platform has accessibility features baked right in that let users with assistive technologies make use of web content. Just as the JS engine can see and interact with a tree of elements on a page, there is a separate “accessibility tree” that is available to assistive technologies so they can better describe and understand the structure and UI of a website. Firefox 61 ships with an Accessibility Inspector that lets developers view the computed accessibility tree and better understand which aspects of their site are friendly to assistive technologies and spot areas where accessible markup is needed. Useful for spotting poorly-labeled buttons and for debugging advanced interactions annotated with ARIA.

You can learn more about how to use the Accessibility Inspector in Marco Zehe’s introductory blog post.

Speaking of DevTools…

  • The tabs for each panel are now draggable, so you can put your most-used tools where you want them!
  • No need to open Responsive Design Mode to enable simulated network throttling – it’s now also available from the top menu of the Network pane.
  • See a custom property in the Inspector? Hover to see its value. While typing css, custom properties autocomplete, including color swatches when the value is a color.

Tab Management

One of the most popular uses of browser extensions is to help users better hoard manage their open tabs. Firefox 61 ships with new extension APIs to help power users use tabs more powerfully! Extensions with the tabs permission can now hide and restore tabs on the browser’s tab bar. The hidden tabs are still loaded, they’re simply not shown. Extensions for productivity and organization can now swap out groups of tabs based on task or context. Firefox also includes an always-available menu that lists all open tabs regardless of their hidden state.

Wrapping Up

Curious about everything that’s new or changed in Firefox 61? You can view the Release Notes or see the full platform changelog on MDN Web Docs.

Have A Great Summer!

1. In the Northern hemisphere, that is. I mean, I hope the Southern hemisphere has a great summer too- it’s just a ways off.

Potch is a Web Platform Advocate at Mozilla.

More articles by Potch…

Original Link

Retained Display Lists for improved page performance

Continuing Firefox Quantum’s investment in a high-performance engine, the Firefox 61 release will boost responsiveness of modern interfaces with an optimization that we call Retained Display Lists. Similar to Quantum’s Stylo and WebRender features, developers don’t need to change anything on their sites to reap the benefits of these improvements.

I wrote about this feature on the Mozilla Graphics Team blog back in January, when it was first implemented in Nightly. Now it’s ready to ship with Firefox 61. If you’re already familiar with retained display lists and how this feature optimizes painting performance, you might want to skip ahead to read about the results of our efforts and the future work we’re planning.

Working with the display list

Display list building is the process in which we collect the set of high-level items to display on screen (borders, backgrounds, text and much more), and then sort the list according to CSS painting rules into the correct back-to-front order. It’s at this point that we figure out which parts of the page are already visible on screen.

Currently, whenever we want to update what’s on the screen, we build a full new display list from scratch and then we use it to paint everything on the screen. This is great for simplicity: we don’t have to worry about figuring out which bits changed or went away. Unfortunately, the process can take a really long time. This has always been a performance problem, but as websites have become more complex and more users have access to higher resolution monitors, the problem has been magnified.

The solution is to retain the display list between paints—we build a new display list only for the parts of the page that changed since we last painted and then merge the new list into the old to get an updated list. This adds a lot more complexity, since we need to figure out which items to remove from the old list, and where to insert new items. The upside is that, in many cases, the new list can be significantly smaller than the full list. This creates the opportunity to improve perceived performance and save significant amounts of time.

Motivation

As part of the lead up to the release of Firefox Quantum, we added new telemetry to Firefox to help us measure painting performance, which therefore enabled us to make more informed decisions as to where to direct our efforts. One of these measurements defined a minimum threshold for a ‘slow’ paint (16ms), and recorded percentages of time spent in various paint stages when it occurred. We expected display list building to be significant, but were still surprised with the results: On average, display list building was consuming more than 40% of the total paint time, for work that was often almost identical to the previous frame. We’d long been planning to overhaul how we built and managed display lists, but with this new data we decided to make it a top priority for our Painting team.

Results

Once we had everything working, the next step was to see how much of an effect we’d made on performance! We had the feature enabled for the first half of the Beta 60 cycle, and compared the results with and without it enabled.

The first and most significant change: The median amount of time spent painting (the full pipeline, not just display list building) dropped by more than 33%!

Time spent in the paint pipeline for content

As you can see in the graph, the median time spent painting is around 3ms when retained display lists are enabled. Once the feature was disabled on April 18th, the paint time jumped up to 4.5ms. That frees up lots of extra time for the browser to spend on running JavaScript, doing layout, and responding to input events.

Another important improvement is in the frequency at which slow paints occurred. With retained display lists disabled, we miss the 16ms deadline around 7.8% of the time. With it enabled, this drops to 4.7%, an almost 40% reduction in frequency. We can see that we’re not just making fast paints faster, but we’re also having a significant impact on the slow cases.

Future work

As mentioned above, we aren’t always able to retain the display list. We’ve spent time working out which parts of the page have changed; when our analysis shows that most of the page has changed, then we still have to rebuild the full display list and time spent on the analysis is time wasted. Work is ongoing to try detect this as early as possible, but it’s unlikely that we’ll be able to entirely prevent it. We’re also actively working to minimize how long the preparation work takes, so that we can make the most of opportunities for a partial update.

Retaining the display list also doesn’t help for the first time we paint a webpage when it loads. The first paint always has to build the full list from scratch, so in the future we’ll be looking at ways to make that faster across the board.

Thanks to everyone who has helped make this possible, including: Miko Mynttinen, Jet Villegas, Timothy Nikkel, Markus Stange, David Anderson, Ethan Lin, and Jonathan Watt.

Gecko Graphics and Layout engineer.

More articles by Matt Woodrow…

Original Link

Overscripted! Digging into JavaScript execution at scale

This research was conducted in partnership with the UCOSP (Undergraduate Capstone Open Source Projects) initiative. UCOSP facilitates open source software development by connecting Canadian undergraduate students with industry mentors to practice distributed development and data projects.

The team consisted of the following Mozilla staff: Martin Lopatka, David Zeber, Sarah Bird, Luke Crouch, Jason Thomas

2017 student interns — crawler implementation and data collection: Ruizhi You, Louis Belleville, Calvin Luo, Zejun (Thomas) Yu

2018 student interns — exploratory data analysis projects: Vivian Jin, Tyler Rubenuik, Kyle Kung, Alex McCallum


As champions of a healthy Internet, we at Mozilla have been increasingly concerned about the current advertisement-centric web content ecosystem. Web-based ad technologies continue to evolve increasingly sophisticated programmatic models for targeting individuals based on their demographic characteristics and interests. The financial underpinnings of the current system incentivise optimizing on engagement above all else. This, in turn, has evolved an insatiable appetite for data among advertisers aggressively iterating on models to drive human clicks.

Most of the content, products, and services we use online, whether provided by media organisations or by technology companies, are funded in whole or in part by advertising and various forms of marketing.

–Timothy Libert and Rasmus Kleis Nielsen [link]

We’ve talked about the potentially adverse effects on the Web’s morphology and how content silos can impede a diversity of viewpoints. Now, the Mozilla Systems Research Group is raising a call to action. Help us search for patterns that describe, expose, and illuminate the complex interactions between people and pages!

Inspired by the Web Census recently published by Steven Englehardt and Arvind Narayanan of Princeton University, we adapted the OpenWPM crawler framework to perform a comparable crawl gathering a rich set of information about the JavaScript execution on various websites. This enables us to delve into further analysis of web tracking, as well as a general exploration of client-page interactions and a survey of different APIs employed on the modern Web.

In short, we set out to explore the unseen or otherwise not obvious series of JavaScript execution events that are triggered once a user visits a webpage, and all the first- and third-party events that are set in motion when people retrieve content. To help enable more exploration and analysis, we are providing our full set of data about JavaScript executions open source.

The following sections will introduce the data set, how it was collected and the decisions made along the way. We’ll share examples of insights we’ve discovered and we’ll provide information on how to participate in the associated “Overscripted Web: A Mozilla Data Analysis Challenge”, which we’ve launched today with Mozilla’s Open Innovation Team.

The Dataset

In October 2017, several Mozilla staff and a group of Canadian undergraduate students forked the OpenWPM crawler repository to begin tinkering, in order to collect a plethora of information about the unseen interactions between modern websites and the Firefox web browser.

Preparing the seed list

The master list of pages we crawled in preparing the dataset was itself generated from a preliminary shallow crawl we performed in November 2017. We ran a depth-1 crawl, seeded by Alexa’s top 10,000 site list, using 4 different machines at 4 different IP addresses (all in residential non-Amazon IP addresses served by Canadian internet service providers). The crawl was implemented using the Requests Python library and collected no information except for an indication of successful page loads.

Of the 2,150,251 pages represented in the union of the 4 parallel shallow crawls, we opted to use the intersection of the four lists in order to prune out dynamically generated (e.g. personalized) outbound links that varied between them. This meant a reduction to 981,545 URLs, which formed the seed list for our main OpenWPM crawl.

The Main Collection

The following workflow describes (at a high level) the collection of page information contained in this dataset.

  1. Alexa top 10k (10,000 high traffic pages as of November 1st, 2017)
  2. Precrawl using the python Requests library, visits each one of those pages
    1. Request library requests that page
    2. That page sends a response
    3. All href tags in the response are captured to a depth of 1 (away from Alexa page)
      1. For each of those href tags all valid pages (starts with “http”) are added to the link set.
      2. The link set union (2,150,251) was examined using the request library in parallel, which gives us the intersection list of 981,545.
      3. The set of urls in the list 981,545 is passed to the deeper crawl for JavaScript analysis in a parallelized form.
  3. Each of these pages was sent to our adapted version of OpenWPM to have the execution of JavaScript recorded for 10 seconds.
  4. The window.location was hashed as the unique identifier of the location where the JavaScript was executed (to ensure unique source attribution).
    1. When OpenWPM hits content that is inside an iFrame, the location of the content is reported.
    2. Since we use the window.location to determine the location element of the content, each time an iFrame is encountered, that location can be split into the parent location of the page and the iFrame location.
    3. Data collection and aggregation performed through a websocket associates all the activity linked to a location hash for compilation of the crawl dataset.

Interestingly, for the Alexa top 10,000 sites, our depth-1 crawl yielded properties hosted on 41,166 TLDs across the union of our 4 replicates, whereas only 34,809 unique TLDs remain among the 981,545 pages belonging to their intersection.

A modified version of OpenWPM was used to record JavaScript calls potentially used for browsers tracking data from these pages. The collected JavaScript execution trace was written into an s3 bucket for later aggregation and analysis. Several additional parameters were defined based on cursory ad hoc analyses.

For example, the minimum dwell time per page required to capture the majority of JavaScript activity was set as 10 seconds per page. This was based on a random sampling of the seed list URLs and showed a large variation in time until no new JavaScript was being executed (from no JavaScript, to what appeared to be an infinite loop of self-referential JavaScript calls). This dwell time was chosen to balance between capturing the majority of JavaScript activity on a majority of the pages and minimizing the time required to complete the full crawl.

Several of the probes instrumented in the Data Leak repo were ported over to our hybrid crawler, including instrumentation to monitor JavaScript execution occuring inside an iFrame element (potentially hosted on a third-party domain). This would prove to provide much insight into the relationships between pages in the crawl data.

Exploratory work

In January 2018, we got to work analyzing the dataset we had created. After substantial data cleaning to work through the messiness of real world variation, we were left with a gigantic  Parquet dataset (around 70GB) containing an immense diversity of potential insights. Three example analyses are summarized below. The most important finding is that we have only just scratched the surface of the insights this data may hold.

Examining session replay activity

Session replay is a service that lets websites track users’ interactions with the page—from how they navigate the site, to their searches, to the input they provide. Think of it as a “video replay” of a user’s entire session on a webpage. Since some session replay providers may record personal information such as personal addresses, credit card information and passwords, this can present a significant risk to both privacy and security.

We explored the incidence of session replay usage, and a few associated features, across the pages in our crawl dataset. To identify potential session replay, we obtained the Princeton WebTAP project list, containing 14 Alexa top-10,000 session replay providers, and checked for calls to script URLs belonging to the list.

Out of 6,064,923 distinct script references among page loads in our dataset, we found 95,570 (1.6%) were to session replay providers. This translated to 4,857 distinct domain names (netloc) making such calls, out of a total of 87,325, or 5.6%. Note that even if scripts belonging to session replay providers are being accessed, this does not necessarily mean that session replay functionality is being used on the site.

Given the set of pages making calls to session replay providers, we also looked into the consistency of SSL usage across these calls. Interestingly, the majority of such calls were made over HTTPS (75.7%), and 49.9% of the pages making these calls were accessed over HTTPS. Additionally, we found no pages accessed over HTTPS making calls to session replay scripts over HTTP, which was surprising but encouraging.

Finally, we examined the distribution of TLDs across sites making calls to session replay providers, and compared this to TLDs over the full dataset. We found that, along with .com, .ru accounted for a surprising proportion of sites accessing such scripts (around 33%), whereas .ru domain names made up only 3% of all pages crawled. This implies that 65.6% of .ru sites in our dataset were making calls to potential session replay provider scripts. However, this may be explained by the fact that Yandex is one of the primary session replay providers, and it offers a range of other analytics services of interest to Russian-language websites.

Eval and dynamically created function calls

JavaScript allows a function call to be dynamically created from a string with the eval() function or by creating a new Function() object. For example, this code will print hello twice:

eval("console.log('hello')")
var my_func = new Function("console.log('hello')")
my_func()

While dynamic function creation has its uses, it also opens up users to injection attacks, such as cross-site scripting, and can potentially be used to hide malicious code.

In order to understand how dynamic function creation is being used on the Web, we analyzed its prevalence, location, and distribution in our dataset. The analysis was initially performed on 10,000 randomly selected pages and validated against the entire dataset. In terms of prevalence, we found that 3.72% of overall function calls were created dynamically, and these originated from across 8.76% of the websites crawled in our dataset.

These results suggest that, while dynamic function creation is not used heavily, it is still common enough on the Web to be a potential concern. Looking at call frequency per page showed that, while some Web pages create all their function calls dynamically, the majority tend to have only 1 or 2 dynamically generated calls (which is generally 1-5% of all calls made by a page).

We also examined the extent of this practice among the scripts that are being called. We discovered that they belong to a relatively small subset of script hosts (at an average ratio of about 33 calls per URL), indicating that the same JavaScript files are being used by multiple webpages. Furthermore, around 40% of these are known trackers (identified using the disconnectme entity list), although only 33% are hosted on a different domain from the webpage that uses them. This suggests that web developers may not even know that they are using dynamically generated functions.

Cryptojacking

Cryptojacking refers to the unauthorized use of a user’s computer or mobile device to mine cryptocurrency. More and more websites are using browser-based cryptojacking scripts as cryptocurrencies rise in popularity. It is an easy way to generate revenue and a viable alternative to bloating a website with ads. An excellent contextualization of crypto-mining via client-side JavaScript execution can be found in the unabridged cryptojacking analysis prepared by Vivian Jin.

We investigated the prevalence of cryptojacking among the websites represented in our dataset. A list of potential cryptojacking hosts (212 sites total) was obtained from the adblock-nocoin-list GitHub repo. For each script call initiated on a page visit event, we checked whether the script host belonged to the list. Among 6,069,243 distinct script references on page loads in our dataset, only 945 (0.015%) were identified as cryptojacking hosts. Over half of these belonged to CoinHive, the original script developer. Only one use of AuthedMine was found. Viewed in terms of domains reached in the crawl, we found calls to cryptojacking scripts being made from 49 out of 29,483 distinct domains (0.16%).

However, it is important to note that cryptojacking code can be executed in other ways than by including the host script in a script tag. It can be disguised, stealthily executed in an iframe, or directly used in a function of a first-party script. Users may also face redirect loops that eventually lead to a page with a mining script. The low detection rate could also be due to the popularity of the sites covered by the crawl, which might  dissuade site owners from implementing obvious cryptojacking scripts. It is likely that the actual rate of cryptojacking is higher.

The majority of the domains we found using cryptojacking are streaming sites. This is unsurprising, as users have streaming sites open for longer while they watch video content, and mining scripts can be executed longer. A Chinese variety site called 52pk.com accounted for 207 out of the overall 945 cryptojacking script calls we found in our analysis, by far the largest domain we observed for cryptojacking calls.

Another interesting fact: although our cryptojacking host list contained 212 candidates, we found only 11 of them to be active in our dataset, or about 5%.

Limitations and future directions

While this is a rich dataset allowing for a number of interesting analyses, it is limited in visibility mainly to behaviours that occur via JS API calls.

Another feature we investigated using our dataset is the presence of Evercookies. Evercookies is a tracking tool used by websites to ensure that user data, such as a user ID, remains permanently stored on a computer. Evercookies persist in the browser by leveraging a series of tricks including Web API calls to a variety of available storage mechanisms. An initial attempt was made to search for evercookies in this data by searching for consistent values being passed to suspect Web API calls.

Acar et al., “The Web Never Forgets: Persistent Tracking Mechanisms in the Wild”, (2014) developed techniques for looking at evercookies at scale. First, they proposed a mechanism to detect identifiers. They applied this mechanism to HTTP cookies but noted that it could also be applied to other storage mechanisms, although some modification would be required. For example, they look at cookie expiration, which would not be applicable in the case of localStorage. For this dataset we could try replicating their methodology for set calls to window.document.cookie and window.localStorage.

They also looked at Flash cookies respawning HTTP cookies and HTTP respawning Flash cookies. Our dataset contains no information on the presence of Flash cookies, so additional crawls would be required to obtain this information. In addition, they used multiple crawls to study Flash respawning, so we would have to replicate that procedure.

In addition to our lack of information on Flash cookies, we have no information about HTTP cookies, the first mechanism by which cookies are set. Knowing which HTTP cookies are initially set can serve as an important complement and validation for investigating other storage techniques then used for respawning and evercookies.

Beyond HTTP and Flash, Samy Kamkar’s evercookie library documents over a dozen mechanisms for storing an id to be used as an evercookie. Many of these are not detectable by our current dataset, e.g. HTTP Cookies, HSTS Pinning, Flask Cookies, Silverlight Storage, ETags, Web cache, Internet Explorer userData storage, etc. An evaluation of the prevalence of each technique would be a useful contribution to the literature. We also see the value of an ongoing repeated crawl to identify changes in prevalence and accounting for new techniques as they are discovered.

However, it is possible to continue analyzing the current dataset for some of the techniques described by Samy. For example, window.name caching is listed as a technique. We can look at this property in our dataset, perhaps by applying the same ID technique outlined by Acar et al., or perhaps by looking at sequences of calls.

Conclusions

Throughout our preliminary exploration of this data it became quickly apparent that the amount of superficial JavaScript execution on a Web page only tells part of the story. We have observed several examples of scripts running parallel to the content-serving functionality of webpages, these appear to fulfill a diversity of other functions. The analyses performed so far have led to some exciting discoveries, but so much more information remains hidden in the immense dataset available.

We are calling on any interested individuals to be part of the exploration. You’re invited to participate in the Overscripted Web: A Mozilla Data Analysis Challenge and help us better understand some of the hidden workings of the modern Web!

Acknowledgements

Extra special thanks to Steven Englehardt for his contributions to the OpenWPM tool and advice throughout this project. We also thank Havi Hoffman for valuable editorial contributions to earlier versions of this post. Finally, thanks to Karen Reid of University of Toronto for coordinating the UCOSP program.

More articles by Martin Lopatka…

Original Link

Baby’s First Rust+WebAssembly module: Say hi to JSConf EU!

A secret project has been brewing for JSConf EU, and this weekend is the big reveal…

The Arch is a larger-than-life experience that uses 30,000 colored LEDs to create a canvas for light animations.

And you can take charge of this space. Using modules, you can create a light animation.

But even though this is JSConf, these animations aren’t just powered by JavaScript modules. In fact, we hope you will try something new… Rust + WebAssembly.

Rust logo and JS logo with a heart in between

Why this project?

One of the hardest problems when you’re learning a new programming language is finding a project that can teach you the basics, but that’s still fun enough to keep you learning more. And “Hello World” is only fun your first few times… it has no real world impact.

But what if your Hello World could have an impact on the real world? What if it could control a structure like this one?

So let’s get started on baby’s first Rust to WebAssembly module.

A baby putting together JS and WebAssembly blocks

And in a way, this the perfect project for your first WebAssembly project… but not because this is the kind of project that you’d use WebAssembly for.

People usually use WebAssembly because they want to supercharge their application and make it run faster. Or because they want to use the same code across both the web and different devices, with their different operating systems.

This project doesn’t do either of those.

WebAssembly benefits: fast and portable crossed out in red

The reason this is a good project for getting started with WebAssembly is not because this is what you would use WebAssembly for.

Instead, it’s useful because it gives you a mental model of how JavaScript and WebAssembly work together. So let’s look at what we need to do to take control of this space with WebAssembly. And then I’ll explain why this makes it a good mental model for how WebAssembly and JavaScript work together.

The space/time continuum as bytes

What we have here is a 3D space. Or really, if you think about it, it’s more like a four dimensional space, because we’re going through time as well.

The computer can’t think in these four dimensions, though. So how do we make these four dimensions make sense to the computer? Let’s start with the fourth dimension and collapse down from there.

You’re probably familiar with the way that we make time the fourth dimension make sense to computers. That’s by using these things called frames.

The screen is kind of like a flipbook. And each frame is like a page in that flip book.

Picture of a flipbook with a smooth animation next to it

On the web, we talk about having 60 frames per second. That’s what you need to have smooth animations across the screen. What that really means is that you have 60 different snapshots of the screen… of what the animation should look like at each of those 60 points during the second.

In our case, the snapshot is a snapshot of what the lights on the space should look like.

So that brings us down to a sequence of snapshots of the space. A sequence of 3D representations of the space.

Now we want to go from 3D to 2D. And in this case, it is pretty easy. All we need to do is take the space and flatten it out into basically a big sheet of graph paper.

So now we’re down to 2D. We just need to collapse this one more time.

We can do that by taking all the rows and putting them next to each other.

A grid being turned into a line of rows

Now we’re down to this line of pixels. And this we can put in memory. Because memory is basically just a line of boxes.

This means we’ve gotten it down to a one-dimensional structure. We still have all of the data that we had in a two-, three- or four-dimensional representation. It’s just being represented in a different way. It’s being represented as a line.

Why is this a good model for learning WebAssembly? Linear memory.

The reason that this is a good mental model for how WebAssembly and JavaScript work together is because one of the main ways to communicate between WebAssembly and JavaScript is through something called linear memory. It’s basically a line of memory that you use to represent things.

The WebAssembly module and the JavaScript that’s running it both have access to this object.

It’s a JavaScript object called an ArrayBuffer. An array buffer is just an array of bytes, and bytes are just numbers. So to make this animation happen, JavaScript tells the WebAssembly module, “Okay, fill in the animation now.”

It will do this by calling a method on the WebAssembly module.

WebAssembly will go and fill in all of the colors for each pixel in the linear memory.

Then the JavaScript code can pull those colors out and turn them into a JSON array that will get sent to the space.

Let’s look at how you use this data from JS.

Linear memory, the hard way

If you’re doing everything yourself and not using any libraries, then you’ll be working directly with the linear memory.

This linear memory is just one big line of 1s and 0s. When you want to create meaning from these 1s and 0s, you have to figure out how to split them up. What you do is create a typed array view on the ArrayBuffer.

Basically this just tells JavaScript how to break up the the bits in this ArrayBuffer. It’s basically like drawing boxes around the bits to say which bits belong to which number.

For example, if you were using hexadecimal values, then your numbers would be 24 bits wide. So you’d need a box that can fit 24 bits. And each box would contain a pixel.

The smallest box that would fit is 32 bits. So we would create an Int32 view on the buffer. And that would wrap the bits up into boxes. In this case we’d have to add a little padding to fill it out (I’m not showing that, but there would be extra zeros).

In contrast, if we used RGB values, the boxes would only be 8 bits wide. To get one RGB value, you would take every three boxes and use those as your R — G — and B values. This means you would iterate over the boxes and pull out the numbers.

Since we’re doing things the hard way here, you need to write the code to do this. The code will iterate over the linear memory and move the data around into more sensible data structures.

For a project like this, that’s not too bad. Colors map well to numbers, so they are easy to represent in linear memory. And the data structures we’re using (RGB values) aren’t too complex. But when you start getting more complex data structures, having to deal directly with memory can be a big pain.

It would be a lot easier if you could pass a JS object into WebAssembly and just have the WebAssembly manipulate that. And this will be possible in the future with specification work currently happening in the WebAssembly community group.

But that doesn’t mean that you have to wait until it’s in the spec before you can start working with objects. You can pass objects into your WebAssembly and return them to JS today. All you need to do is add one tiny library.

Linear memory, the easy way

This library is called wasm-bindgen. It wraps the WebAssembly module in a JavaScript wrapper.

This wrapper knows how to take complex JavaScript objects and write them into linear memory. Then, when the WebAssembly function returns a value, the JS wrapper will take the data from linear memory and turn it back into a JS object.

JS passing the string Hello to <code>wasm-bindgen</code>, which does all of the other work

To do this, it looks at the function signatures in your Rust code and figures out exactly what JavaScript is needed. This works for built-in types like strings. It also works for types that you define in your code. wasm-bidgen will take those Rust structs and turn them into JavaScript classes.

Right now, this tool is specific to Rust. But with the way that it’s architected, we can add support for this kind of higher level interaction for other languages — languages like C/C++.

In conclusion…

Hopefully you now see how to take control of this space… How you can say Hello World, and hello to the world of WebAssembly.

Before I wrap this up, I do want to give credit to the people that made this project possible.

The seeds of the idea for this project came from a dance party in a space like this I attended in Pittsburgh. But this project was only possible because of the amazing group of people that gathered to make it a reality.

  • Sandra Persing — I came to her with a vision and she made that vision real
  • Dan Brown and Maciej Pluta, who took that vision and turned it into something even more exciting and engaging than I had imagined
  • Till Schneidereit, who helped me figure out how all the pieces fit together
  • Josh Marinacci, who created the site and made taking control of the space possible
  • Dan Callahan, who jumped in with his development and debugging wizardry to ensure all of the pieces worked together
  • Trevor Flowers, who created the virtual space so that everyone can experience the Arch, even if they aren’t at the event
  • Michael Bebenita and Yury Delendik, whose work on WebAssembly Studio makes it possible to share WebAssembly with a whole new audience
  • Rustaceans: Alex Crichton, Ashley Williams, Sarah Meyers, Jan-Erik Rediger, Florian Gilcher, Steve Klabnik, Fabian, Istvan ‘Flaki’ Szmozsanszky, who worked on WebAssembly Studio’s Rust integration and helped new, aspiring Rust developers level up their skills
  • The JSConf EU team for all of their hard work making sure that we could get the project off the ground
  • Ian Brill, the artist who’s work inspired this project and who’s hard work ensured we could share it with you

Lin is an engineer on the Mozilla Developer Relations team. She tinkers with JavaScript, WebAssembly, Rust, and Servo, and also draws code cartoons.

More articles by Lin Clark…

Original Link

A cartoon intro to DNS over HTTPS

Threats to users’ privacy and security are growing. At Mozilla, we closely track these threats. We believe we have a duty to do everything we can to protect Firefox users and their data.

We’re taking on the companies and organizations that want to secretly collect and sell user data. This is why we added tracking protection and created the Facebook container extension. And you’ll be seeing us do more things to protect our users over the coming months.

Icons for security projects that we’ve introduced

Two more protections we’re adding to that list are:

  • DNS over HTTPS, a new IETF standards effort that we’ve championed
  • Trusted Recursive Resolver, a new secure way to resolve DNS that we’ve partnered with Cloudflare to provide

With these two initiatives, we’re closing data leaks that have been part of the domain name system since it was created 35 years ago. And we’d like your help in testing them. So let’s look at how DNS over HTTPS and Trusted Recursive Resolver protect our users.

But first, let’s look at how web pages move around the Internet.

If you already know how DNS and HTTPS work, you can skip to how DNS over HTTPS helps.

A brief HTTP crash course

When people explain how a browser downloads a web page, they usually explain it this way:

  1. Your browser makes a GET request to a server.
  2. The server sends a response, which is a file containing HTML.

browser GET request + response

This system is called HTTP.

But this diagram is a little oversimplified. Your browser doesn’t talk directly to the server. That’s because they probably aren’t close to each other.

Instead, the server could be thousands of miles away. And there’s likely no direct link between your computer and the server.

image of client and server on opposite ends of the network

So this request needs to get from the browser to that server, and it will go through multiple hands before it gets there. And the same is true for the response coming back from the server.

I think of this like kids passing notes to each other in class. On the outside, the note will say who it’s supposed to go to. The kid who wrote the note will pass it to their neighbor. Then that next kid passes it to one of their neighbors — probably not the eventual recipient, but someone who’s in that direction.

kids passing notes

The problem with this is that anyone along the path can open up the note and read it. And there’s no way to know in advance which path the note is going to take, so there’s no telling what kind of people will have access to it.

It could end up in the hands of people who do harmful things…

Like sharing the contents of the note with everyone.

kid saying “Ooo, hey everybody… Danny loves Sandy!”

Or changing the response.

kid saying “Do you like me? Y/N… Heh, I’m going to prank him and put no here”

To fix these issues, a new, secure version of HTTP was created. This is called HTTPS. With HTTPS, it’s kind of like each message has a lock on it.

open envelope next to locked envelope

Both the browser and the server know the combination to that lock, but no one in between does.

With this, even if the messages go through multiple routers in between, only you and the web site will actually be able to read the contents.

This solves a lot of the security issues. But there are still some messages going between your browser and the server that aren’t encrypted. This means people along the way can still pry into what you’re doing.

One place where data is still exposed is in setting up the connection to the server. When you send your initial message to the server, you send the server name as well (in a field called “Server Name Indication”). This lets server operators run multiple sites on the same machine while still knowing who you are trying to talk to. This initial request is part of setting up encryption, but the initial request itself isn’t encrypted.

The other place where data is exposed is in DNS. But what is DNS?

DNS: the Domain Name System

In the passing notes metaphor above, I said that the name of the recipient had to be on the outside of the note. This is true for HTTP requests too… they need to say who they are going to.

But you can’t use a name for them. None of the routers would know who you were talking about. Instead, you have to use an IP address. That’s how the routers in between know which server you want to send your request to.

network with IP addresses

This causes a problem. You don’t want users to have to remember your site’s IP address. Instead, you want to be able to give your site a catchy name… something that users can remember.

This is why we have the domain name system (DNS). Your browser uses DNS to convert the site name to an IP address. This process — converting the domain name to an IP address — is called domain name resolution.

domain and address equivalence

How does the browser know how to do this?

One option would be to have a big list, like a phone book in the browser. But as new web sites came online, or as sites moved to new servers, it would be hard to keep that list up-to-date.

So instead of having one list which keeps track of all of the domain names, there are lots of smaller lists that are linked to each other. This allows them to be managed independently.

one list, vs lots of smaller lists

In order to get the IP address that corresponds to a domain name, you have to find the list that contains that domain name. Doing this is kind of like a treasure hunt.

What would this treasure hunt look like for a site like the English version of wikipedia, en.wikipedia.org?

We can split this domain into parts.

domain split into top level, second level, and subdomain.

With these parts, we can hunt for the list that contains the IP address for the site. We need some help in our quest, though. The tool that will go on this hunt for us and find the IP address is called a resolver.

First, the resolver talks to a server called the Root DNS. It knows of a few different Root DNS servers, so it sends the request to one of them. The resolver asks the Root DNS where it can find more info about addresses in the .org top-level domain.

The Root DNS will give the resolver an address for a server that knows about about .org addresses.

resolver talking to Root DNS

This next server is called a top-level domain (TLD) name server. The TLD server knows about all of the second-level domains that end with .org.

It doesn’t know anything about the subdomains under wikipedia.org, though, so it doesn’t know the IP address for en.wikipedia.org.

The TLD name server will tell the resolver to ask Wikipedia’s name server.

resolver talking to TLD DNS

The resolver is almost done now. Wikipedia’s name server is what’s called the authoritative server. It knows about all of the domains under wikipedia.org. So this server knows about en.wikipedia.org, and other subdomains like the German version, de.wikipedia.org. The authoritative server tells the resolver which IP address has the HTML files for the site.

resolver talking to authoritative DNS

The resolver will return the IP address for en.wikipedia.org to the operating system.

This process is called recursive resolution, because you have to go back and forth asking different servers what’s basically the same question.

I said we need a resolver to help us in our quest. But how does the browser find this resolver? In general, it asks the computer’s operating system to set it up with a resolver that can help.

browser asking OS for resolver

How does the operating system know which resolver to use? There are two possible ways.

You can configure your computer to use a resolver you trust. But very few people do this.

Instead, most people just use the default. And by default, the OS will just use whatever resolver the network told it to. When the computer connects to the network and gets its IP address, the network recommends a resolver to use.

operating system getting a recommendation from the network

This means that the resolver that you’re using can change multiple times per day. If you head to the coffee shop for an afternoon work session, you’re probably using a different resolver than you were in the morning. And this is true even if you have configured your own resolver, because there’s no security in the DNS protocol.

How can DNS be exploited?

So how can this system make users vulnerable?

Usually a resolver will tell each DNS server what domain you are looking for. This request sometimes includes your full IP address. Or if not your full IP address, increasingly often the request includes most of your IP address, which can easily be combined with other information to figure out your identity.

DNS request

This means that every server that you ask to help with domain name resolution sees what site you’re looking for. But more than that, it also means that anyone on the path to those servers sees your requests, too.

There are a few ways that this system puts users’ data at risk. The two major risks are tracking and spoofing.

Tracking

Like I said above, it’s easy to take the full or partial IP address info and figure out who’s asking for that web site. This means that the DNS server and anyone along the path to that DNS server — called on-path routers — can create a profile of you. They can create a record of all of the web sites that they’ve seen you look up.

And that data is valuable. Many people and companies will pay lots of money to see what you are browsing for.

a router offering to sell data

Even if you didn’t have to worry about the possibly nefarious DNS servers or on-path routers, you still risk having your data harvested and sold. That’s because the resolver itself — the one that the network gives to you — could be untrustworthy.

Even if you trust your network’s recommended resolver, you’re probably only using that resolver when you’re at home. Like I mentioned before, whenever you go to a coffee shop or hotel or use any other network, you’re probably using a different resolver. And who knows what its data collection policies are?

Beyond having your data collected and then sold without your knowledge or consent, there are even more dangerous ways the system can be exploited.

Spoofing

With spoofing, someone on the path between the DNS server and you changes the response. Instead of telling you the real IP address, a spoofer will give you the wrong IP address for a site. This way, they can block you from visiting the real site or send you to a scam one.

spoofer sending user to wrong site

Again, this is a case where the resolver itself might act nefariously.

For example, let’s say you’re shopping for something at Megastore. You want to do a price check to see if you can get it cheaper at a competing online store, big-box.com.

But if you’re on Megastore WiFi, you’re probably using their resolver. That resolver could hijack the request to big-box.com and lie to you, saying that the site is unavailable.

How can we fix this with Trusted Recursive Resolver (TRR) and DNS over HTTPS (DoH)?

At Mozilla, we feel strongly that we have a responsibility to protect our users and their data. We’ve been working on fixing these vulnerabilities.

We are introducing two new features to fix this — Trusted Recursive Resolver (TRR) and DNS over HTTPS (DoH). Because really, there are three threats here:

  1. You could end up using an untrustworthy resolver that tracks your requests, or tampers with responses from DNS servers.
  2. On-path routers can track or tamper in the same way.
  3. DNS servers can track your DNS requests.

the three threats—resolvers, on-path routers, and DNS servers

So how do we fix these?

  1. Avoid untrustworthy resolvers by using Trusted Recursive Resolver.
  2. Protect against on-path eavesdropping and tampering using DNS over HTTPS.
  3. Transmit as little data as possible to protect users from deanonymization.

Avoid untrustworthy resolvers by using Trusted Recursive Resolver

Networks can get away with providing untrustworthy resolvers that steal your data or spoof DNS because very few users know the risks or how to protect themselves.

Even for users who do know the risks, it’s hard for an individual user to negotiate with their ISP or other entity to ensure that their DNS data is handled responsibly.

However, we’ve spent time studying these risks… and we have negotiating power. We worked hard to find a company to work with us to protect users’ DNS data. And we found one: Cloudflare.

Cloudflare is providing a recursive resolution service with a pro-user privacy policy. They have committed to throwing away all personally identifiable data after 24 hours, and to never pass that data along to third-parties. And there will be regular audits to ensure that data is being cleared as expected.

With this, we have a resolver that we can trust to protect users’ privacy. This means Firefox can ignore the resolver that the network provides and just go straight to Cloudflare. With this trusted resolver in place, we don’t have to worry about rogue resolvers selling our users’ data or tricking our users with spoofed DNS.

Why are we picking one resolver? Cloudflare is as excited as we are about building a privacy-first DNS service. They worked with us to build a DoH resolution service that would serve our users well in a transparent way. They’ve been very open to adding user protections to the service, so we’re happy to be able to collaborate with them.

But this doesn’t mean you have to use Cloudflare. Users can configure Firefox to use whichever DoH-supporting recursive resolver they want. As more offerings crop up, we plan to make it easy to discover and switch to them.

Protect against on-path eavesdropping and tampering using DNS over HTTPS

The resolver isn’t the only threat, though. On-path routers can track and spoof DNS because they can see the contents of the DNS requests and responses. But the Internet already has technology for ensuring that on-path routers can’t eavesdrop like this. It’s the encryption that I talked about before.

By using HTTPS to exchange the DNS packets, we ensure that no one can spy on the DNS requests that our users are making.

Transmit as little data as possible to protect users from deanonymization

In addition to providing a trusted resolver which communicates using the DoH protocol, Cloudflare is working with us to make this even more secure.

Normally, a resolver would send the whole domain name to each server—to the Root DNS, the TLD name server, the second-level name server, etc. But Cloudflare will be doing something different. It will only send the part that is relevant to the DNS server it’s talking to at the moment. This is called QNAME minimization.

image showing resolver only asking the relevant question

The resolver will also often include the first 24 bits of your IP address in the request. This helps the DNS server know where you are and pick a CDN closer to you. But this information can be used by DNS servers to link different requests together.

Instead of doing this, Cloudflare will make the request from one of their own IP addresses near the user. This provides geolocation without tying it to a particular user. In addition to this, we’re looking into how we can enable even better, very fine-grained load balancing in a privacy-sensitive way.

Doing this — removing the irrelevant parts of the domain name and not including your IP address — means that DNS servers have much less data that they can collect about you.

DNS request with client subnet and first part of domain cross out

What isn’t fixed by TRR with DoH?

With these fixes, we’ve reduced the number of people who can see what sites you’re visiting. But this doesn’t eliminate data leaks entirely.

After you do the DNS lookup to find the IP address, you still need to connect to the web server at that address. To do this, you send an initial request. This request includes a server name indication, which says which site on the server you want to connect to. And this request is unencrypted.

That means that your ISP can still figure out which sites you’re visiting, because it’s right there in the server name indication. Plus, the routers that pass that initial request from your browser to the web server can see that info too.

However, once you’ve made that connection to the web server, then everything is encrypted. And the neat thing is that this encrypted connection can be used for any site that is hosted on that server, not just the one that you initially asked for.

This is sometimes called HTTP/2 connection coalescing, or simply connection reuse. When you open a connection to a server that supports it, that server will tell you what other sites it hosts. Then you can visit those other sites using that existing encrypted connection.

Why does this help? You don’t need to start up a new connection to visit these other sites. This means you don’t need to send that unencrypted initial request with its server name indication saying which site you’re visiting. Which means you can visit any of the other sites on the same server without revealing what sites you’re looking at to your ISP and on-path routers.

With the rise of CDNs, more and more independent sites are being served by a single server. And since you can have multiple coalesced connections open, you can be connected to multiple shared servers or CDNs at once, visiting all of the sites across the different servers without leaking data. This means this will be more and more effective as a privacy shield.

What is the status?

You can enable DNS over HTTPS in Firefox today, and we encourage you to.

We’d like to turn this on as the default for all of our users. We believe that every one of our users deserves this privacy and security, no matter if they understand DNS leaks or not.

But it’s a big change and we need to test it out first. That’s why we’re conducting a study. We’re asking half of our Firefox Nightly users to help us collect data on performance.

We’ll use the default resolver, as we do now, but we’ll also send the request to Cloudflare’s DoH resolver. Then we’ll compare the two to make sure that everything is working as we expect.

For participants in the study, the Cloudflare DNS response won’t be used yet. We’re simply checking that everything works, and then throwing away the Cloudflare response.

diagram showing a person timing both and then throwing away Cloudflare response

We are thankful to have the support of our Nightly users — the people who help us test Firefox every day — and we hope that you will help us test this, too.

Lin is an engineer on the Mozilla Developer Relations team. She tinkers with JavaScript, WebAssembly, Rust, and Servo, and also draws code cartoons.

More articles by Lin Clark…

Original Link

Progressive Web Games

With the recent release of the Progressive Web Apps core guides on MDN, it’s easier than ever to make your website look and feel as responsive as native on mobile devices. But how about games?

In this article, we’ll explore the concept of Progressive Web Games to see if the concept is practical and viable in a modern web development environment, using PWA features built with Web APIs.

Let’s look at the Enclave Phaser Template (EPT) — a free, open sourced mobile boilerplate for HTML5 games that I created using the Phaser game engine. I’m using it myself to build all my Enclave Games projects.

The template was recently upgraded with some PWA features: Service Workers provide the ability to cache and serve the game when offline, and a manifest file allows it to be installed on the home screen. We also provide access to notifications, and much more. These PWA features are already built-in into the template, so you can focus on developing the game itself.

We will see how those features can solve problems developers have today: adding to home screen and working offline. The third part of this article will introduce the concept of progressive loading.

Add to Home screen

Web games can show their full potential on mobile, especially if we create some features and shortcuts for developers. The Add to Home screen feature makes it easier to build games that can compete with native games for screen placement and act as first class citizens on mobile devices.

Progressive Web Apps can be installed on modern devices with the help of this feature. You enable it by including a manifest file — the icon, modals and install banners are created based on the information from ept.webmanifest:

{ "name": "Enclave Phaser Template", "short_name": "EPT", "description": "Mobile template for HTML5 games created using the Phaser game engine.", "icons": [ { "src": "img/icons/icon-32.png", "sizes": "32x32", "type": "image/png" }, // ... { "src": "img/icons/icon-512.png", "sizes": "512x512", "type": "image/png" } ], "start_url": "/index.html", "display": "fullscreen", "theme_color": "#DECCCC", "background_color": "#CCCCCC"
}

It’s not the only requirement though — be sure to check the Add to Home Screen article for all the details.

Offline capabilities

Developers often have issues getting desktop games (or mobile-friendly games showcased on a PC with a monitor) to work offline. This is especially challenging when demoing a game at a conference with unreliable wifi! Best practice is to plan ahead and have all the files of the game available locally, so that you can launch them offline.

Screenshot of Hungry Fridge web game on the desktop

Offline builds can be tricky, as you’ll have to manage the files yourself, remember your versions, and whether you’ve applied the latest patch or fixed that bug from previous conferences, work out the hardware setup, etc. This takes time and extra preparation.

Web games are easier to handle online when you have reliable connectivity: You point the browser to a URL and you have the latest version of your game running in no time. The network connection is the problem. It would be nice to have an offline solution.

Image of HTML5 Games demo station with no offline gameplay

The good news is that Progressive Web Apps can help — Service Workers cache and serve assets offline, so an unstable network connection is not the problem it used to be.

The Service Worker file in the Enclave Phaser Template contains everything we need. It starts with the list of files to be cached:

var cacheName = 'EPT-v1';
var appShellFiles = [ './', './index.html', // ... './img/overlay.png', './img/particle.png', './img/title.png'
];

Then the install event is handled, which adds all the files to the cache:

self.addEventListener('install', function(e) { e.waitUntil( caches.open(cacheName).then(function(cache) { return cache.addAll(appShellFiles); }) );
});

Next comes the fetch event, which will serve content from cache and add a new one, if needed:

self.addEventListener('fetch', function(e) { e.respondWith( caches.match(e.request).then(function(r) { return r || fetch(e.request).then(function(response) { return caches.open(cacheName).then(function(cache) { cache.put(e.request, response.clone()); return response; }); }); }) );
});

Be sure to check the Service Worker article for a detailed explanation.

Progressive loading

Progressive loading is an interesting concept that can provide many benefits for web game development. Progressive loading is basically “lazy loading” in the background. It’s not dependent on a specific API, but it follows the PWA approach and uses several of the key features we’ve just described, focused on games, and their specific requirements.

Games are heavier than apps in terms of resources — even for small and casual ones, you usually have to download 5-15 MB of assets, from images to sounds. This is supposed to be instantaneous, but you still have to wait through the loading screen when everything is downloaded. Or, if might be problematic if the player has a poor connection: the longer the download time, the bigger the chance that gameplay will be abandoned and the tab will be closed.

But what if instead of downloading everything, you loaded only what’s really needed first, and then downloaded the rest in the background? This way the player would see the main menu of your game way faster than with the traditional approach. They would spend at least a few seconds looking around while the files for the gameplay are retrieved in the background invisibly. And even if they clicked the play button really quickly, we could show a loading animation while everything else is loaded.

Instant Games are gaining in popularity, and a game developer building casual mobile HTML5 games should probably consider putting them on the Facebook or Google platforms. There are some requirements to meet, especially concerning the initial file size and download time, as the games are supposed to be instantly available for play.

Using the lazy loading technique, the game will feel faster than it would otherwise, given the amount of data required for it to be playable. You can achieve these results using the Phaser framework, which is pre-built to load most of the assets after the main menu resources arrive. You can also implement this yourself in pure JavaScript, using the link prefetch/defer mechanism. There’s more than one way to achieve progressive loading – it’s a powerful idea that’s independent of the specific approach or tools you choose, following the principles of Progressive Web Apps.

Conclusion

Do you have any more ideas on how to enhance the gaming experience? Feel free to play and experiment, and shape the future of web games. Drop a comment here if you’ve got ideas to share.

HTML5 Game Developer, Enclave Games indie studio founder, Gamedev.js Weekly newsletter publisher and js13kGames competition creator. Tech Speaker passionate about new, open web technologies, excited about WebVR.

More articles by Andrzej Mazur…

Original Link

Progressive Web Games

With the recent release of the Progressive Web Apps core guides on MDN, it’s easier than ever to make your website look and feel as responsive as native on mobile devices. But how about games?

In this article, we’ll explore the concept of Progressive Web Games to see if the concept is practical and viable in a modern web development environment, using PWA features built with Web APIs.

Let’s look at the Enclave Phaser Template (EPT) — a free, open sourced mobile boilerplate for HTML5 games that I created using the Phaser game engine. I’m using it myself to build all my Enclave Games projects.

The template was recently upgraded with some PWA features: Service Workers provide the ability to cache and serve the game when offline, and a manifest file allows it to be installed on the home screen. We also provide access to notifications, and much more. These PWA features are already built-in into the template, so you can focus on developing the game itself.

We will see how those features can solve problems developers have today: adding to home screen and working offline. The third part of this article will introduce the concept of progressive loading.

Add to Home screen

Web games can show their full potential on mobile, especially if we create some features and shortcuts for developers. The Add to Home screen feature makes it easier to build games that can compete with native games for screen placement and act as first class citizens on mobile devices.

Progressive Web Apps can be installed on modern devices with the help of this feature. You enable it by including a manifest file — the icon, modals and install banners are created based on the information from ept.webmanifest:

{ "name": "Enclave Phaser Template", "short_name": "EPT", "description": "Mobile template for HTML5 games created using the Phaser game engine.", "icons": [ { "src": "img/icons/icon-32.png", "sizes": "32x32", "type": "image/png" }, // ... { "src": "img/icons/icon-512.png", "sizes": "512x512", "type": "image/png" } ], "start_url": "/index.html", "display": "fullscreen", "theme_color": "#DECCCC", "background_color": "#CCCCCC"
}

It’s not the only requirement though — be sure to check the Add to Home Screen article for all the details.

Offline capabilities

Developers often have issues getting desktop games (or mobile-friendly games showcased on a PC with a monitor) to work offline. This is especially challenging when demoing a game at a conference with unreliable wifi! Best practice is to plan ahead and have all the files of the game available locally, so that you can launch them offline.

Screenshot of Hungry Fridge web game on the desktop

Offline builds can be tricky, as you’ll have to manage the files yourself, remember your versions, and whether you’ve applied the latest patch or fixed that bug from previous conferences, work out the hardware setup, etc. This takes time and extra preparation.

Web games are easier to handle online when you have reliable connectivity: You point the browser to a URL and you have the latest version of your game running in no time. The network connection is the problem. It would be nice to have an offline solution.

Image of HTML5 Games demo station with no offline gameplay

The good news is that Progressive Web Apps can help — Service Workers cache and serve assets offline, so an unstable network connection is not the problem it used to be.

The Service Worker file in the Enclave Phaser Template contains everything we need. It starts with the list of files to be cached:

var cacheName = 'EPT-v1';
var appShellFiles = [ './', './index.html', // ... './img/overlay.png', './img/particle.png', './img/title.png'
];

Then the install event is handled, which adds all the files to the cache:

self.addEventListener('install', function(e) { e.waitUntil( caches.open(cacheName).then(function(cache) { return cache.addAll(appShellFiles); }) );
});

Next comes the fetch event, which will serve content from cache and add a new one, if needed:

self.addEventListener('fetch', function(e) { e.respondWith( caches.match(e.request).then(function(r) { return r || fetch(e.request).then(function(response) { return caches.open(cacheName).then(function(cache) { cache.put(e.request, response.clone()); return response; }); }); }) );
});

Be sure to check the Service Worker article for a detailed explanation.

Progressive loading

Progressive loading is an interesting concept that can provide many benefits for web game development. Progressive loading is basically “lazy loading” in the background. It’s not dependent on a specific API, but it follows the PWA approach and uses several of the key features we’ve just described, focused on games, and their specific requirements.

Games are heavier than apps in terms of resources — even for small and casual ones, you usually have to download 5-15 MB of assets, from images to sounds. This is supposed to be instantaneous, but you still have to wait through the loading screen when everything is downloaded. Or, if might be problematic if the player has a poor connection: the longer the download time, the bigger the chance that gameplay will be abandoned and the tab will be closed.

But what if instead of downloading everything, you loaded only what’s really needed first, and then downloaded the rest in the background? This way the player would see the main menu of your game way faster than with the traditional approach. They would spend at least a few seconds looking around while the files for the gameplay are retrieved in the background invisibly. And even if they clicked the play button really quickly, we could show a loading animation while everything else is loaded.

Instant Games are gaining in popularity, and a game developer building casual mobile HTML5 games should probably consider putting them on the Facebook or Google platforms. There are some requirements to meet, especially concerning the initial file size and download time, as the games are supposed to be instantly available for play.

Using the lazy loading technique, the game will feel faster than it would otherwise, given the amount of data required for it to be playable. You can achieve these results using the Phaser framework, which is pre-built to load most of the assets after the main menu resources arrive. You can also implement this yourself in pure JavaScript, using the link prefetch/defer mechanism. There’s more than one way to achieve progressive loading – it’s a powerful idea that’s independent of the specific approach or tools you choose, following the principles of Progressive Web Apps.

Conclusion

Do you have any more ideas on how to enhance the gaming experience? Feel free to play and experiment, and shape the future of web games. Drop a comment here if you’ve got ideas to share.

HTML5 Game Developer, Enclave Games indie studio founder, Gamedev.js Weekly newsletter publisher and js13kGames competition creator. Tech Speaker passionate about new, open web technologies, excited about WebVR.

More articles by Andrzej Mazur…

Original Link

New in Firefox 61: Developer Edition

Firefox 61: Developer Edition is available now, and contains a ton of great new features and under-the-hood improvements.

A Darker Dark Theme

Taking inspiration from Spinal Tap, Developer Edition’s dark theme now darkens more parts of the browser, including the new tab page.

Screenshot comparing the new darker Dark Theme with the older version

Searchable websites can now be added to Firefox via the “Add Search Engine” item inside the Page Action menu. The sites must describe their search APIs using OpenSearch metadata.

Screenshot of the OpenSearch "Add Search Engine" item in the page menu

And yes, the Page Action menu is also dark, if you’re using a dark theme.

More Powerful Developer Tools

More than just source maps, Firefox 61 understands how tools like Babel and Webpack work, making it possible to seamlessly inspect and interact with your original code right within the Debugger, as if it had never been bundled and minified in the first place. We’re also working to add native support for inspecting components and scopes in modern frameworks like React.

To learn more, see our separate, in-depth blog post: Debugging Modern Web Applications.

Nicer Developer Tools

The Developer Tools have seen numerous quality-of-life improvements.

You can now rearrange tools to suit your individual workflow, and any tabs that don’t fit in the current window remain readily accessible in an overflow menu.

Animation of DevTools tabs rearranging

The Network panel also gained prominent drop-down menus for controlling network throttling and importing/exporting HTTP Archive (“HAR”) files.

Screenshot of HAR import and export menu in DevTools

We’ve also sped up the DevTools across the board, and are measuring and tracking performance as an explicit goal for the team. Even more improvements are on the way.

In Firefox Quantum, we re-implemented many of the DevTools using basic web technologies: HTML, JavaScript, and CSS. We’re even using React inside the DevTools themselves! This means that if you know how to build for the web, you know how to hack on the DevTools. If you’d like to get involved, we have a great getting started guide with pointers to good first bugs to tackle.

The Accessibility Inspector

There’s also an entirely new tool available, the Accessibility Inspector, which reveals the logical structure of your page, as it might appear to a screen reader or other assistive software.

A screenshot of the new Accessibility Inspector that helps you assess what the browser and screen reader can 'see'

This is a low-level tool meant to help you understand how Firefox and screen readers “see” your content. To learn more, including how to enable and use this new tool, check out Marco Zehe’s article Introducing the Accessibility Inspector. If you’re looking for more opinionated tools to help audit and improve your site’s accessibility, consider add-ons like the aXe Developer Tools or the WAVE Accessibility Extension.

Behind the Scenes

Lastly, we landed a number of improvements and refactorings that should make Firefox a better all-around browser.

  • Firefox now parses CSS stylesheets in multiple parallel threads. This can significantly improve the time to first paint for websites, especially when there are many stylesheets on a single page.
  • The multi-tier WebAssembly compiler has been implemented for the AArch64 CPU architecture common in smartphones and tablets. You can read more about the benefits of this compiler design in Lin Clark’s article, Making WebAssembly Even Faster.
  • On macOS, like on Windows, browser add-ons now run in a separate, dedicated process. A continuation of our work with multi-process Firefox, this helps Firefox itself stay responsive, even when an add-on is busy doing work.

Firefox 61 is currently available in Beta and Developer Edition, and it will become the stable version of Firefox on June 26th. If you’d like to keep up with Firefox development as it happens, we recommend reading the Firefox Nightly Blog, or following @FirefoxNightly on Twitter.

Engineer with Mozilla Developer Relations, former Mozilla Persona developer.

More articles by Dan Callahan…

Original Link

Debugging Modern Web Applications

Building and debugging modern JavaScript applications in Firefox DevTools just took a quantum leap forward. In collaboration with Logan Smyth, Tech Lead for Babel, we leveled up the debugger’s source map support to let you inspect the code that you actually wrote. Combined with the ongoing initiative to offer first-class JS framework support across all our devtools, this will boost productivity for modern web app developers.

Modern JS frameworks and build tools play a critical role today. Frameworks like React, Angular, and Ember let developers build declarative user interfaces with JSX, directives, and templates. Tools like Webpack, Babel, and PostCSS let developers use new JS and CSS features before they are supported by browser vendors. These tools help developers write simpler code, but generate more complicated code to debug.

In the example below, we use Webpack and Babel to compile ES Modules and async functions into vanilla JS, which can run in any browser. The original code on the left is pretty simple. The generated, browser-compatible code on the right is much more complicated.

In the example below, we use Webpack and Babel to compile ES Modules and async functions into vanilla JS. The original code on the left is pretty simple. The generated, browser-compatible code on the right is much more complicated.
Figure 1. Original file on the left, generated file on the right.

When the debugger pauses, it uses source maps to navigate from line 13 in the generated code to line 4 in the original code. Unfortunately, because pausing actually happens on line 13, it can be difficult for the user to figure out what the value of dancer is at that time. Moving the mouse over the variable dancer returns undefined and the only way to find the scope of dancer is to open all six of the available scopes in the Scopes pane followed by expanding the _emojis object! This complicated and frustrating process is why many people opt to disable source maps.

A view of the disconnect between the original code file and the generated code, which opens multiple scopes
Figure 2. Value of dancer is undefined, six separate scopes in the Scopes pane.

To address this problem we teamed up with Logan Smyth to see if it was possible to make the interaction feel more natural, as if you were debugging your original code. The result is a new engine that maps source maps data with Babel’s syntax tree to show the variables you expect to see the way you want to see them.

Now the panel displays the correct value of dancer, and the Scopes pane shows one scope
Figure 3. Correct value of dancer is displayed, Scopes pane shows one scope.

These improvements were first implemented for Babel and Webpack, and we’re currently adding support for TypeScript, Angular, Vue, Ember, and many others. If your project already generates source maps there is a good chance this feature will work for you out of the box.

To try it out, just head over and download Firefox Developer Edition. You can help us by testing this against your own project and reporting any issues. If you want to follow along, say hello, or contribute, you can also find us on the devtools channel Github or Mozilla Discourse or in the devtools Slack!

Our 2018 goal is to improve the lives of web developers who are building modern apps using the latest frameworks, build tools and best practices. Fixing variables is just the beginning. The future is bright!

Staff Engineer on Firefox Developer Tools devving the DevTools and debugging the debugger.

More articles by jlaster…

Guiding Firefox’s Devtools and Performance work. Come hangout with the DevTools team!

More articles by Harald Kirschner…

Original Link

Firefox 60 – Modules and More

Firefox 60 is here, and the Quantum lineage continues apace. The parallel processing prowess of Quantum CSS is now available on Firefox for Android, and work continues on WebRender, which modernizes the whole idea of what it means to draw a web page. But we’re not just spreading the love on internals. Firefox 60 boasts a number of web platform and developer-facing improvements as well. Here are a few highlights:

ES Modules are Here!

A Code Cartoon of a module tree

Modular code isn’t just a good idea, it’s the law it’s a great idea! Being able to separate functional units of software allows for cleaner re-use of individual modules and easier inclusion of third-party code. Many languages have support for modules, and if you’re familiar with Node.JS, they’ve been available in some form with the CommonJS require API, but a standardized syntax was created as part of ES2015 (ES6).

Although the syntax for ES modules was standardized, it was left as an exercise for browsers to understand and retrieve the modules. This took a bit of extra time, but now that the browser loading behavior is standardized, support has started rolling out, and this release brings that support to Spidermonkey, Firefox’s JavaScript engine. You can check out the docs on MDN, and of course don’t miss Lin Clark’s breakdown of ES modules either!

Keep Your Cookies to Yourself

Firefox 60 supports the Same-Site attribute when setting cookies. When set, the browser will not send cookies along with a cross-origin request to the issuing server, e.g. during fetch or loading an image. This helps mitigate against common silent forms of Cross-Origin Request Forgery. There is a “lax” mode that does the above, as well as a strict mode that, in addition to the lax behavior, will also not send cookies with an in-bound navigation. This helps prevent a malicious site deep-linking to a page where unintentional behavior could occur when cookies are included.

Read more on the Mozilla Security Blog.

Web Authentication API

It’s been known for a while now that in many contexts, a well-known username (like an email address) and a user-generated password are not sufficiently secure for authentication. This has led to the rise of Multi-Factor Authentication, usually 2-factor authentication, in which in addition to a password, users must also provide information from an additional source. Many sites will send an SMS message with a code to a mobile device, and some also accept tokens generated by a code-generator app or purpose-built hardware “key”. This whole exchange has required the user to copy numbers from a screen into a text field, or at minimum the hardware key has had to simulate key presses.

The Web Authentication API (WebAuthn for short) seeks to eliminate the clunkier aspects of this process by letting a multi-factor authentication device or app communicate directly with a requesting site. The particulars of making this work securely are a bit too complex to cover in this post, but you can learn more about WebAuthn on MDN or here on the Hacks Blog.

A Stroke of Style

The (as-of-yet non-standard) text-stroke property defines a solid fixed-width stroke centered along the path of the characters of text. It allows for effects that aren’t achievable with text-shadow alone. A wide stroke will occlude portions of the characters because by default, the stroke is drawn over top of the font glyph. This can be a bit ugly. To fix this, browsers are borrowing paint-order property from the SVG standard. When properly set, browsers will draw the stroke underneath the text glyphs. For example:

It’s super nifty- but don’t forget that it’s not yet a standard, and you should always check that text is legible without stroke effects applied! You can read more on MDN and check out the compatibility matrix there.

ESR / Group Policy

Firefox 60 is the next version of Firefox to be designated an “Extended Support Release”, or ESR. ESR releases are intended for system administrators who deploy and maintain desktop environments in large organizations. They receive security and stability updates in sync with the latest Release versions of Firefox, and each ESR release’s support overlaps with the next one. This overlap period allows a large organization to certify and deploy new ESR versions before leaving the support window for the prior release.

Firefox 60 ships along with the first incarnation of a new Policy Engine that allows organizational adminstrators to configure Firefox for all their users en masse. On Windows, this is accomplished using Windows Group Policy, and via a configuration file on other platforms. It’s not a feature that most Firefox users will ever need, but if your job is managing thousands of installations of Firefox, we hope you’ll find this a welcome addition.

Would You Like to Know More?

As always, the full list of developer-facing changes is on MDN, and you can find the release notes here.

Keep on rocking the free web!

Potch is a Web Platform Advocate at Mozilla.

More articles by Potch…

Original Link

Creating Web Things with Python, Node.js, and Java

The Mozilla IoT team recently released the Things Framework, which allows you to build IoT devices that speak the Web Thing API. Last week, James built an ESP8266 web thing. This time, I’m going to show you how to build web things with Python, Node.js, or Java. These languages are definitely not optimal for small embedded devices; this tutorial is intended for higher-end devices that can run these languages with ease, or even your own desktop computer.

To demonstrate, we’ll be turning the Music Player Daemon (MPD) into a web thing. The libraries we’ll be using here are webthing-python, webthing-node, and webthing-java.

Intro to the Music Player Daemon (MPD)

The Music Player Daemon is an open-source music player that has been around since 2003. MPD operates in a client/server model, and is controllable over TCP with a simple text-based protocol. I won’t cover installation here, but MPD can be installed via your package manager on Linux, Homebrew on Mac OS X, or with binaries on Windows.

Some Quick Notes

Although this example is written for MPD, it would be easily portable to other music players with an API, or you could build your own player using this example as a base. More importantly, this example is primarily meant to demonstrate the Things Framework, which can be used to create an endless variety of web things.

The Python, Node.js, and Java web thing libraries all have essentially the same API. While they may not be entirely idiomatic for their respective languages, doing this allows us to maintain all the libraries more easily, which is incredibly valuable while the Web Thing API is still a living draft.

Getting Started

To start, initialize an empty project for your language of choice. For example, you might create a new project in IntelliJ IDEA for Java, or run npm init to start a new Node.js project. You’ll then want to add the webthing library as a dependency. Instructions for doing so can be found on the respective Github project page, or you can look here.

Now we’re ready to write some code. Essentially we’ll do the following:

  1. Create a Thing.
    1. Define its properties.
    2. Define its actions.
    3. Define its events.
  2. Attach the Thing to a WebThingServer.

Create a Thing

First off, let’s import our library of choice.

Python

from webthing import Action, Event, Property, Thing, Value, WebThingServer

JavaScript

const { Action, Event, Property, Thing, Value, WebThingServer,
} = require('webthing');

Java

import org.mozilla.iot.webthing.Action;
import org.mozilla.iot.webthing.Event;
import org.mozilla.iot.webthing.Property;
import org.mozilla.iot.webthing.Thing;
import org.mozilla.iot.webthing.Value;
import org.mozilla.iot.webthing.WebThingServer;

Now, we’ll create a basic subclass of Thing.

Python

class MPDThing(Thing): def __init__(self): Thing.__init__(self, 'MPD', 'musicPlayer', 'Music Player Daemon')

JavaScript

class MPDThing extends Thing { constructor() { super('MPD', 'musicPlayer', 'Music Player Daemon'); }
}

Java

public class MPDThing extends Thing { public MPDThing() { super("MPD", "musicPlayer", "Music Player Daemon"); }
}

Add Properties

Now that we have our MPDThing, we’ll need to add some properties to it. Obviously, a music player will have quite a few properties. To demonstrate, I will show you how to add one, and I’ll link to the rest of the code at the end of this post.

To add a property, you can do the following inside the MPDThing constructor.

Python

status = self.get_status()
self.add_property( Property(self, 'volume', Value(self.get_volume(status), self.set_volume), metadata={ 'type': 'number', 'description': 'Playback volume', 'minimum': 0, 'maximum': 100, }))

JavaScript

this.getStatus().then((status) => { this.getVolume(status).then((v) => { this.addProperty( new Property(this, 'volume', new Value(v, this.setVolume.bind(this)), { type: 'number', description: 'Playback volume', minimum: 0, maximum: 100, })); });
});

Java

ServerStatus status = this.client.getServerStatus();
Map<String, Object> volumeMetadata = new HashMap<>();
volumeMetadata.put("type", "number");
volumeMetadata.put("description", "Playback volume");
volumeMetadata.put("minimum", 0);
volumeMetadata.put("maximum", 100);
this.volumeValue = new Value(status.getVolume(), v -> this.setVolume((int)v));
this.addProperty(new Property(this, "volume", this.volumeValue, volumeMetadata));

We’ve now created a Property that allows us to GET and PUT the playback volume. The Value piece is an object that essentially stores a cached value and a “value forwarder” callback. When the volume is set via a PUT request, the value forwarder is used to forward the new value to the actual device, which in this case is the MPD server. We’ve also set up some metadata for the property, including a description, value type, and a minimum and maximum value.

Add Actions

A music player can also have a lot of actions. While the MPD web thing has several basic actions like play, pause, and skip, I’ve added one that takes some additional input, which will queue a series of random songs. Action inputs are verified with a JSON Schema validator.

First, let’s create an Action subclass.

Python

class QueueRandomAction(Action): def __init__(self, thing, input_): Action.__init__( self, uuid.uuid4().hex, thing, 'queueRandom', input_=input_) def perform_action(self): songs = self.thing.list() if songs: for _ in range(0, int(self.input['count'])): self.thing.add(random.choice(songs)) playlist = self.thing.get_playlist() if playlist is not None: self.thing.add_event( PlaylistUpdatedEvent(self.thing, playlist))

JavaScript

class QueueRandomAction extends Action { constructor(thing, input) { super(uuidv4(), thing, 'queueRandom', input); } performAction() { return this.thing.list().then((songs) => { const promises = []; if (songs) { for (let i = 0; i < this.input.count; ++i) { const uri = songs[Math.floor(Math.random() * songs.length)].file; promises.push(this.thing.add(uri)); } promises.push(this.thing.getPlaylist().then((playlist) => { if (playlist) { this.thing.addEvent(new PlaylistUpdatedEvent(this.thing, playlist)); } })); } return Promise.all(promises); }); }
}

Java

public static class QueueRandomAction extends Action { public QueueRandomAction(Thing thing, JSONObject input) { super(UUID.randomUUID().toString(), thing, "queueRandom", input); } @Override public void performAction() { MPDThing thing = (MPDThing)this.getThing(); Random random = new Random(); List<MPDFile> songs = thing.list(); for (int i = 0; i < this.getInput().getInt("count"); ++i) { MPDFile file = songs.get(random.nextInt(songs.size())); thing.add(file); } String playlist = thing.getPlaylist(); thing.addEvent(new PlaylistUpdatedEvent(thing, playlist)); }
}

QueueRandomAction takes an input, count, queues that number of random songs to the current playlist, and then emits a PlaylistUpdatedEvent (to be defined shortly). To add this new action to our MPDThing, do the following inside the MPDThing constructor:

Python

self.add_available_action( 'queueRandom', {'description': 'Queue a series of random songs', 'input': { 'type': 'object', 'required': [ 'count', ], 'properties': { 'count': { 'type': 'number', 'minimum': 1, }, }, }}, QueueRandomAction)

JavaScript

this.addAvailableAction( 'queueRandom', { description: 'Queue a series of random songs', input: { type: 'object', required: [ 'count', ], properties: { count: { type: 'number', minimum: 1, }, }, }, }, QueueRandomAction);

Java

Map<String, Object> queueRandomMetadata = new HashMap<>();
queueRandomMetadata.put("description", "Queue a series of random songs");
Map<String, Object> queueRandomInputMetadata = new HashMap<<>();
queueRandomInputMetadata.put("type", "object");
queueRandomInputMetadata.put("required", new String[]{"count"});
Map<String, Object> queueRandomInputPropertiesMetadata = new HashMap<>();
Map<String, Object> queueRandomInputPropertiesCountMetadata = new HashedMap();
queueRandomInputPropertiesCountMetadata.put("type", "number");
queueRandomInputPropertiesCountMetadata.put("minimum", 1);
queueRandomInputPropertiesMetadata.put("count", queueRandomInputPropertiesCountMetadata);
queueRandomInputMetadata.put("properties", queueRandomInputPropertiesMetadata);
queueRandomMetadata.put("input", queueRandomInputMetadata);
this.addAvailableAction("queueRandom", queueRandomMetadata,
QueueRandomAction.class);

Add Events

The final piece of our Thing is its events. Since MPD is a client/server model, it can be updated externally by any number of other clients. As such, I created an event that will fire when the current playlist is updated.

As with Thing and Action, we’ll create an Event subclass.

Python

class PlaylistUpdatedEvent(Event): def __init__(self, thing, data): Event.__init__(self, thing, 'playlistUpdated', data=data)

JavaScript

class PlaylistUpdatedEvent extends Event { constructor(thing, data) { super(thing, 'playlistUpdated', data); }
}

Java

public static class PlaylistUpdatedEvent extends Event { public PlaylistUpdatedEvent(Thing thing, String data) { super(thing, "playlistUpdated", data); }
}

This is a basic Event. The data member will be filled in with a string representation of the current playlist.

To add this Event to our thing, we’ll do the following in the MPDThing constructor:

Python

self.add_available_event( 'playlistUpdated', {'description': 'The current playlist has been updated', 'type': 'string'})

JavaScript

this.addAvailableEvent( 'playlistUpdated', { description: 'The current playlist has been updated', type: 'string', });

Java

Map<String, Object> playlistUpdatedMetadata = new HashMap<>();
playlistUpdatedMetadata.put("description", "The current playlist has been updated");
playlistUpdatedMetadata.put("type", "string");
this.addAvailableEvent("playlistUpdated", playlistUpdatedMetadata);

Create a WebThingServer

Now that we have a thing with properties, actions, and events, we’ll create a WebThingServer and attach the MPDThing to it.

Python

thing = MPDThing() server = WebThingServer([thing], port=8888) try: server.start()
except KeyboardInterrupt: server.stop()

JavaScript

const thing = new MPDThing(); const server = new WebThingServer([thing], null, 8888); process.on('SIGINT', () => { server.stop(); process.exit();
}); server.start();

Java

MPDThing thing = new MPDThing(); List<Thing> things = new ArrayList<>();
things.add(thing); try { WebThingServer server = new WebThingServer(things, null, 8888); Runtime.getRuntime() .addShutdownHook(new Thread(() -> server.stop())); server.start(false);
} catch (IOException e) { System.out.println(e); System.exit(1);
}

Controlling the Web Thing

Our web thing is complete and it’s now controllable via the Web Thing API. Here’s how to add it to the Things Gateway:

The Things Gateway doesn’t currently provide a way to use actions or display events, but those are in the works.

Alternatively, you can control the web thing via cURL or any other HTTP library you choose:

Shell

$ curl \ -X POST http://127.0.0.1:8888/actions \ -H 'Content-Type: application/json' \ -d '{"play":{}}'

Wrapping Up

Your imagination is really your only limit as to what you can turn into a web thing. If you’d like to see the rest of this example and how the actual MPD controls are implemented, all of the code is available on Github.

If you have any questions or suggestions, you can head over to Discourse or find us in #iot on irc.mozilla.org. Additionally, feel free to create issues, or even better, submit pull requests, to the webthing library repositories!

Michael is a software engineer at Mozilla working on Project Things.

More articles by Michael Stegeman…

Original Link

Progressive Web Apps core guides on MDN Web Docs

Progressive web apps (PWAs) are a new way of building websites, but are they really all that new? The basic principles of PWAs came out of older strategies for app design such as progressive enhancement, responsive design, mobile-first, etc. Progressive web apps bring together proven techniques such as these with a new set of APIs and other features under one umbrella term; 2018 could be the year of PWA.

On MDN Web Docs you’ll find a set of Core PWA Guides published for everyone who’s interested in checking them out. As a game developer I couldn’t stop myself from including a gamedev-related example in the series. js13kPWA is my website listing all the entries from the A-Frame category in the js13kGames 2017 competition.

When the user visits the PWA with a supporting mobile browser, it should display a banner indicating that it’s possible to install the app as a PWA:

When the user visits the PWA with a supporting mobile browser, it should display a banner indicating that it's possible to install the app as a PWA.

It is still just a website like any other, so you can try it yourself with no extra steps needed. The source code is available on GitHub, and you can also view it live.

There are currently five Core PWA Guide articles:

Introduction to progressive web apps

The first article of the series introduces progressive web apps: defining what is a PWA, what makes an app progressive, whether it’s worth building, and the advantages it brings over regular web apps.

You can see the browser support, and check the js13k PWA example — the implementation of its contents is explained in subsequent articles.

Progressive web app structure

Now that you’ve learned the theory behind PWAs, you can look at the recommended structure of an actual app. This article goes through the differences between server-side rendering and client-side rendering, shows how you can mix them both, and explains how PWAs can be built using any approach you like.

The app shell concept is the most popular approach for building the structure of an app. It’s important to follow the rules of being linkable, progressive, and responsive by design. We also discuss briefly how the Streams API will be able to help with faster progressive loading in the near future.

Making PWAs work offline with service workers

This article goes into details of implementing offline capabilities with service workers, explains the offline first approach, and looks at why the “progressive” in PWA is important.

We then inspect the js13kPWA source code to learn how to register a service worker, and what its lifecycle looks like, with installation, activation and updates explained in detail, and how to clear the cache if needed.

How to make PWAs installable

The fourth article describes how to install PWAs on your mobile device with supporting mobile browsers, so the app can be used as if it were native.

If the user clicks the button, there will be an extra step showing what the app will look like. When confirmed, the app will be installed on the home screen:

If the user clicks the button, there will be an extra step showing what the app will look like. When confirmed, the app will be installed on the home screen.

This section reviews basic requirements, the contents of the manifest file, the add-to-home-screen feature, and splash screen configuration.

How to make PWAs reengageable using notifications and push

The last article in the Core Guide offers a way to stay engaged with users. Both the Push API and the Notifications API help achieve this goal – the first one can deliver new content whenever it is available while the other is used to show its details to the user.

Up until now, we’ve use js13kPWA, which is served from backend-less GitHub Pages, to illustrate key PWA concepts, but the Push API requires the server side to work. That’s why I’ve taken the Push Payload Demo example from the Service Workers Cookbook and explained it in detail: subscribing, getting VAPID keys, and posting and receiving content.

Conclusion

I hope these excerpts have sparked your interest. Dive into the content right away to learn about implementing PWA features in your apps. The benefits greatly exceed the effort it will take to make your apps more engaging.

It’s possible that in the next few years we may forget the term PWA, as it becomes the primary technique used to build interactive websites. Be sure to learn how to do it effectively now. Just as responsive web design approach is now the standard for building mobile optimized websites, PWA techniques are on track to become the standard of the future.

HTML5 Game Developer, Enclave Games indie studio founder, Gamedev.js Weekly newsletter publisher and js13kGames competition creator. Tech Speaker passionate about new, open web technologies, excited about WebVR.

More articles by Andrzej Mazur…

Original Link

Making a Web Thing on the ESP8266

Today I’m going to walk you through creating a simple Web Thing using an inexpensive off-the-shelf ESP8266 board.

The power of web things comes from their ability to connect the digital world of web pages with the physical world of things. We recently released the Things Framework, a collection of software intended to make it easy to create new web things. The relevant library for this example is the webthing-esp8266 library, which makes easy it to connect Arduino-programmed ESP8266 boards with the Web of Things. We hope that this lowers the barrier to creating compelling experiences with our gateway and the Web Thing API.

Lamp example running on the ESP8266

The first step in our journey is to install the Arduino IDE and its ESP8266 board support. Visit Adafruit’s ESP8266 documentation for a very detailed walkthrough of setting up. At the end of setup, you should have an Arduino IDE installed that has the ESP8266WiFi, ESP8266mDNS, and ESP8266WebServer libraries available.

System diagram

Now that we have all the prerequisites out of the way, let’s get to the fun part. The webthing-esp8266 library works by assembling a collection of components that come together to expose the Web Thing API. The main coordinator is the WebThingAdapter which keeps track of a ThingDevice that in turn has an assortment of ThingProperties.

The WebThingAdapter knows how to speak the Web of Things API with our Gateway and handles all the translation necessary for the Gateway to discover and interact with the ThingDevice. The ThingDevice represents the physical object we want to put on the web. In a simple case, this may be a few LEDs. Once we get more complex, this could be a quadcopter, an OLED display, or even a Tesla coil. Before we get ahead of ourselves, let’s step through a basic example which exposes the ESP8266’s built-in LED.

To start, create a new sketch in the Arduino IDE. Now that we have a place to write code, we need to include all the libraries this sketch uses. These are ESP8266WiFi and WiFiClient for connecting to our WiFi network, Thing for creating Web-of-Things-compatible objects, and WebThingAdapter for translating these objects into a web server.

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <Thing.h>
#include <WebThingAdapter.h>

The next step is to configure the constants we’re using. These are ssid for our WiFi network’s name, password for its password, and lampPin for the pin of the LED we want to control.

const char* ssid = "......";
const char* password = ".........."; const int lampPin = LED_BUILTIN;

Now we get to specify what kind of web thing we’re creating. First, we create the adapter, which sets the name of the board. If you want to have multiple ESP8266 boards on the same network, you’ll need to make sure their names are unique.

WebThingAdapter adapter("esp8266");

Then we need to specify the ThingDevice we want to have on our gateway. In this case, we want to expose the LED as a dimmableLight called “My Lamp” which will allow us to turn it on and control its brightness from the gateway.

ThingDevice lamp("lamp", "My Lamp", "dimmableLight");

Next we define the properties we want the ThingDevice to have. A dimmableLight needs two properties: “on” and “level”.

ThingProperty lampOn("on", "Whether the lamp is turned on", BOOLEAN);
ThingProperty lampLevel("level", "The level of light from 0-100", NUMBER);

In the start of our setup function we initialize the LED, connect to our WiFi network, and turn on the Serial port for debugging.

void setup() { pinMode(lampPin, OUTPUT); digitalWrite(lampPin, HIGH); analogWriteRange(255); Serial.begin(115200); WiFi.mode(WIFI_STA); WiFi.begin(ssid, password); Serial.println(""); // Wait for connection while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); Serial.print("Connected to "); Serial.println(ssid); Serial.print("IP address: "); Serial.println(WiFi.localIP());

With that boilerplate out of the way, we can tie together our ThingProperties, ThingDevice, and WebThingAdapter. The lamp needs to know that it owns the lampOn and lampLevel properties, while the adapter needs to know that the lamp exists.

 lamp.addProperty(&lampOn); lamp.addProperty(&lampLevel); adapter.addDevice(&lamp); adapter.begin(); Serial.println("HTTP server started");
}

In our continuously running loop function we first update the adapter so it can handle talking to the gateway or any other Web of Things clients. Next, we update the light based on the property values. Note that we take the 0-100 brightness level and map it from 255-0 because the brightness of the ESP8266’s built-in LED is inverted.

void loop() { adapter.update(); if (lampOn.getValue().boolean) { int level = map(lampLevel.getValue().number, 0, 100, 255, 0); analogWrite(lampPin, level); } else { analogWrite(lampPin, 255); }
}

Our sketch is done! If you upload this to your ESP8266 you should be able to see the “My Lamp” thing in your gateway’s Add Things list. You can then click to turn the LED on and off, or click on the “splat” icon to control its brightness level.

And, if you need more inspiration, check out the demo of my LED lamp:

You can also control your web thing directly by issuing HTTP requests to http://esp8266.local/. For example, you can use this curl command to turn the LED on from the terminal.

curl -X PUT http://esp8266.local/things/lamp/properties/on -H 'Content-Type: application/json' -d '{"on":true}'

This is just the beginning of what you can do with the Web Thing API. If you’d prefer working in a higher-level language, check out our other web thing examples. If you think on/off and level aren’t cool enough on their own, we also support string properties!

Level 24 Computer Wizard on a quest to keep the Internet of Things free and open.

More articles by James Hobin…

Original Link

Testing Strategies for React and Redux

When the Firefox Add-ons team ported addons.mozilla.org to a single page app backed by an API, we chose React and Redux for powerful state management, delightful developer tools, and testability. Achieving the testability part isn’t completely obvious since there are competing tools and techniques.

Below are some testing strategies that are working really well for us.

Testing must be fast and effective

We want our tests to be lightning fast so that we can ship high-quality features quickly and without discouragement. Waiting for tests can be discouraging, yet tests are crucial for preventing regressions, especially while restructuring an application to support new features.

Our strategy is to only test what’s necessary and only test it once. To achieve this we test each unit in isolation, faking out its dependencies. This is a technique known as unit testing and in our case, the unit is typically a single React component.

Unfortunately, it’s very difficult to do this safely in a dynamic language such as JavaScript since there is no fast way to make sure the fake objects are in sync with real ones. To solve this, we rely on the safety of static typing (via Flow) to alert us if one component is using another incorrectly — something a unit test might not catch.

A suite of unit tests combined with static type analysis is very fast and effective. We use Jest because it too is fast, and because it lets us focus on a subset of tests when needed.

Testing Redux connected components

The dangers of testing in isolation within a dynamic language are not entirely alleviated by static types, especially since third-party libraries often do not ship with type definitions (creating them from scratch is cumbersome). Also, Redux-connected components are hard to isolate because they depend on Redux functionality to keep their properties in sync with state. We settled on a strategy where we trigger all state changes with a real Redux store. Redux is crucial to how our application runs in the real world so this makes our tests very effective.

As it turns out, testing with a real Redux store is fast. The design of Redux lends itself very well to testing due to how actions, reducers, and state are decoupled from one another. The tests give the right feedback as we make changes to application state. This also makes it feel like a good fit for testing. Aside from testing, the Redux architecture is great for debugging, scaling, and especially development.

Consider this connected component as an example: (For brevity, the examples in this article do not define Flow types but you can learn about how to do that here.)

import { connect } from 'react-redux';
import { compose } from 'redux'; // Define a functional React component.
export function UserProfileBase(props) { return ( <span>{props.user.name}</span> );
} // Define a function to map Redux state to properties.
function mapStateToProps(state, ownProps) { return { user: state.users[ownProps.userId] };
} // Export the final UserProfile component composed of
// a state mapper function.
export default compose( connect(mapStateToProps),
)(UserProfileBase);

You may be tempted to test this by passing in a synthesized user property but that would bypass Redux and all of your state mapping logic. Instead, we test by dispatching a real action to load the user into state and make assertions about what the connected component rendered.

import { mount } from 'enzyme';
import UserProfile from 'src/UserProfile'; describe('<UserProfile>', () => { it('renders a name', () => { const store = createNormalReduxStore(); // Simulate fetching a user from an API and loading it into state. store.dispatch(actions.loadUser({ userId: 1, name: 'Kumar' })); // Render with a user ID so it can retrieve the user from state. const root = mount(<UserProfile userId={1} store={store} />); expect(root.find('span')).toEqual('Kumar'); });
});

Rendering the full component with Enzyme’s mount() makes sure mapStateToProps() is working and that the reducer did what this specific component expected. It simulates what would happen if the real application requested a user from the API and dispatched the result. However, since mount() renders all components including nested components, it doesn’t allow us to test UserProfile in isolation. For that we need a different approach using shallow rendering, explained below.

Shallow rendering for dependency injection

Let’s say the UserProfile component depends on a UserAvatar component to display the user’s photo. It might look like this:

export function UserProfileBase(props) { const { user } = props; return ( <div> <UserAvatar url={user.avatarURL} /> <span>{user.name}</span> </div> );
}

Since UserAvatar will have unit tests of its own, the UserProfile test just has to make sure it calls the interface of UserAvatar correctly. What is its interface? The interface to any React component is simply its properties. Flow helps to validate property data types but we also need tests to check the data values.

With Enzyme, we don’t have to replace dependencies with fakes in a traditional dependency injection sense. We can simply infer their existence through shallow rendering. A test would look something like this:

import UserProfile, { UserProfileBase } from 'src/UserProfile';
import UserAvatar from 'src/UserAvatar';
import { shallowUntilTarget } from './helpers'; describe('<UserProfile>', () => { it('renders a UserAvatar', () => { const user = { userId: 1, avatarURL: 'https://cdn/image.png', }; store.dispatch(actions.loadUser(user)); const root = shallowUntilTarget( <UserProfile userId={1} store={store} />, UserProfileBase ); expect(root.find(UserAvatar).prop('url')) .toEqual(user.avatarURL); });
});

Instead of calling mount(), this test renders the component using a custom helper called shallowUntilTarget(). You may already be familiar with Enzyme’s shallow() but that only renders the first component in a tree. We needed to create a helper called shallowUntilTarget() that will render all “wrapper” (or higher order) components until reaching our target, UserProfileBase.

Hopefully Enzyme will ship a feature similar to shallowUntilTarget() soon, but the implementation is simple. It calls root.dive() in a loop until root.is(TargetComponent) returns true.

With this shallow rendering approach, it is now possible to test UserProfile in isolation yet still dispatch Redux actions like a real application.

The test looks for the UserAvatar component in the tree and simply makes sure UserAvatar will receive the correct properties (the render() function of UserAvatar is never executed). If the properties of UserAvatar change and we forget to update the test, the test might still pass, but Flow will alert us about the violation.

The elegance of both React and shallow rendering just gave us dependency injection for free, without having to inject any dependencies! The key to this testing strategy is that the implementation of UserAvatar is free to evolve on its own in a way that won’t break the UserProfile tests. If changing the implementation of a unit forces you to fix a bunch of unrelated tests, it’s a sign that your testing strategy may need rethinking.

Composing with children, not properties

The power of React and shallow rendering really come into focus when you compose components using children instead of passing JSX via properties. For example, let’s say you wanted to wrap UserAvatar in a common InfoCard for layout purposes. Here’s how to compose them together as children:

export function UserProfileBase(props) { const { user } = props; return ( <div> <InfoCard> <UserAvatar url={user.avatarURL} /> </InfoCard> <span>{user.name}</span> </div> );
}

After making this change, the same assertion from above will still work! Here it is again:

expect(root.find(UserAvatar).prop('url')) .toEqual(user.avatarURL);

In some cases, you may be tempted to pass JSX through properties instead of through children. However, common Enzyme selectors like root.find(UserAvatar) would no longer work. Let’s look at an example of passing UserAvatar to InfoCard through a content property:

export function UserProfileBase(props) { const { user } = props; const avatar = <UserAvatar url={user.avatarURL} />; return ( <div> <InfoCard content={avatar} /> <span>{user.name}</span> </div> );
}

This is still a valid implementation but it’s not as easy to test.

Testing JSX passed through properties

Sometimes you really can’t avoid passing JSX through properties. Let’s imagine that InfoCard needs full control over rendering some header content.

export function UserProfileBase(props) { const { user } = props; return ( <div> <InfoCard header={<Localized>Avatar</Localized>}> <UserAvatar url={user.avatarURL} /> </InfoCard> <span>{user.name}</span> </div> );
}

How would you test this? You might be tempted to do a full Enzyme mount() as opposed to a shallow() render. You might think it will provide you with better test coverage but that additional coverage is not necessary — the InfoCard component will already have tests of its own. The UserProfile test just needs to make sure InfoCard gets the right properties. Here’s how to test that.

import { shallow } from 'enzyme';
import InfoCard from 'src/InfoCard';
import Localized from 'src/Localized';
import { shallowUntilTarget } from './helpers'; describe('<UserProfile>', () => { it('renders an InfoCard with a custom header', () => { const user = { userId: 1, avatarURL: 'https://cdn/image.png', }; store.dispatch(actions.loadUser(user)); const root = shallowUntilTarget( <UserProfile userId={1} store={store} />, UserProfileBase ); const infoCard = root.find(InfoCard); // Simulate how InfoCard will render the // header property we passed to it. const header = shallow( <div>{infoCard.prop('header')}</div> ); // Now you can make assertions about the content: expect(header.find(Localized).text()).toEqual('Avatar'); });
});

This is better than a full mount() because it allows the InfoCard implementation to evolve freely so long as its properties don’t change.

Testing component callbacks

Aside from passing JSX through properties, it’s also common to pass callbacks to React components. Callback properties make it very easy to build abstractions around common functionality. Let’s imagine we are using a FormOverlay component to render an edit form in a UserProfileManager component.

import FormOverlay from 'src/FormOverlay'; export class UserProfileManagerBase extends React.Component { onSubmit = () => { // Pretend that the inputs are controlled form elements and // their values have already been connected to this.state. this.props.dispatch(actions.updateUser(this.state)); } render() { return ( <FormOverlay onSubmit={this.onSubmit}> <input id="nameInput" name="name" /> </FormOverlay> ); }
} // Export the final UserProfileManager component.
export default compose( // Use connect() from react-redux to get props.dispatch() connect(),
)(UserProfileManagerBase);

How do you test the integration of UserProfileManager with FormOverlay? You might be tempted once again to do a full mount(), especially if you’re testing integration with a third-party component, something like Autosuggest. However, a full mount() is not necessary.

Just like in previous examples, the UserProfileManager test can simply check the properties passed to FormOverlay. This is safe because FormOverlay will have tests of its own and Flow will validate the properties. Here is an example of testing the onSubmit property.

import FormOverlay from 'src/FormOverlay';
import { shallowUntilTarget } from './helpers'; describe('<UserProfileManager>', () => { it('updates user information', () => { const store = createNormalReduxStore(); // Create a spy of the dispatch() method for test assertions. const dispatchSpy = sinon.spy(store, 'dispatch'); const root = shallowUntilTarget( <UserProfileManager store={store} />, UserProfileManagerBase ); // Simulate typing text into the name input. const name = 'Faye'; const changeEvent = { target: { name: 'name', value: name }, }; root.find('#nameInput').simulate('change', changeEvent); const formOverlay = root.find(FormOverlay); // Simulate how FormOverlay will invoke the onSubmit property. const onSubmit = formOverlay.prop('onSubmit'); onSubmit(); // Make sure onSubmit dispatched the correct ation. const expectedAction = actions.updateUser({ name }); sinon.assertCalledWith(dispatchSpy, expectedAction); });
});

This tests the integration of UserProfileManager and FormOverlay without relying on the implementation of FormOverlay. It uses sinon to spy on the store.dispatch() method to make sure the correct action is dispatched when the user invokes onSubmit().

Every change starts with a Redux action

The Redux architecture is simple: when you want to change application state, dispatch an action. In the last example of testing the onSubmit() callback, the test simply asserted a dispatch of actions.updateUser(...). That’s it. This test assumes that once the updateUser() action is dispatched, everything will fall into place.

So how would an application like ours actually update the user? We would connect a saga to the action type. The updateUser() saga would be responsible for making a request to the API and dispatching further actions when receiving a response. The saga itself will have unit tests of its own. Since the UserProfileManager test runs without any sagas, we don’t have to worry about mocking out the saga functionality. This architecture makes testing very easy; something like redux-thunk may offer similar benefits.

Summary

These examples illustrate patterns that work really well at addons.mozilla.org for solving common testing problems. Here is a recap of the concepts:

  • We dispatch real Redux actions to test application state changes.
  • We test each component only once using shallow rendering.
  • We resist full DOM rendering (with mount()) as much as possible.
  • We test component integration by checking properties.
  • Static typing helps validate our component properties.
  • We simulate user events and make assertions about what action was dispatched.

Want to get more involved in Firefox Add-ons community? There are a host of ways to contribute to the add-ons ecosystem – and plenty to learn, whatever your skills and level of experience.

Kumar hacks on Mozilla web services and tools for various projects, such as those supporting Firefox Add-ons. He hacks on lots of random open source projects too.

More articles by kumar303…

Original Link

Testing Strategies for React and Redux

When the Firefox Add-ons team ported addons.mozilla.org to a single page app backed by an API, we chose React and Redux for powerful state management, delightful developer tools, and testability. Achieving the testability part isn’t completely obvious since there are competing tools and techniques.

Below are some testing strategies that are working really well for us.

Testing must be fast and effective

We want our tests to be lightning fast so that we can ship high-quality features quickly and without discouragement. Waiting for tests can be discouraging, yet tests are crucial for preventing regressions, especially while restructuring an application to support new features.

Our strategy is to only test what’s necessary and only test it once. To achieve this we test each unit in isolation, faking out its dependencies. This is a technique known as unit testing and in our case, the unit is typically a single React component.

Unfortunately, it’s very difficult to do this safely in a dynamic language such as JavaScript since there is no fast way to make sure the fake objects are in sync with real ones. To solve this, we rely on the safety of static typing (via Flow) to alert us if one component is using another incorrectly — something a unit test might not catch.

A suite of unit tests combined with static type analysis is very fast and effective. We use Jest because it too is fast, and because it lets us focus on a subset of tests when needed.

Testing Redux connected components

The dangers of testing in isolation within a dynamic language are not entirely alleviated by static types, especially since third-party libraries often do not ship with type definitions (creating them from scratch is cumbersome). Also, Redux-connected components are hard to isolate because they depend on Redux functionality to keep their properties in sync with state. We settled on a strategy where we trigger all state changes with a real Redux store. Redux is crucial to how our application runs in the real world so this makes our tests very effective.

As it turns out, testing with a real Redux store is fast. The design of Redux lends itself very well to testing due to how actions, reducers, and state are decoupled from one another. The tests give the right feedback as we make changes to application state. This also makes it feel like a good fit for testing. Aside from testing, the Redux architecture is great for debugging, scaling, and especially development.

Consider this connected component as an example: (For brevity, the examples in this article do not define Flow types but you can learn about how to do that here.)

import { connect } from 'react-redux';
import { compose } from 'redux'; // Define a functional React component.
export function UserProfileBase(props) { return ( <span>{props.user.name}</span> );
} // Define a function to map Redux state to properties.
function mapStateToProps(state, ownProps) { return { user: state.users[ownProps.userId] };
} // Export the final UserProfile component composed of
// a state mapper function.
export default compose( connect(mapStateToProps),
)(UserProfileBase);

You may be tempted to test this by passing in a synthesized user property but that would bypass Redux and all of your state mapping logic. Instead, we test by dispatching a real action to load the user into state and make assertions about what the connected component rendered.

import { mount } from 'enzyme';
import UserProfile from 'src/UserProfile'; describe('<UserProfile>', () => { it('renders a name', () => { const store = createNormalReduxStore(); // Simulate fetching a user from an API and loading it into state. store.dispatch(actions.loadUser({ userId: 1, name: 'Kumar' })); // Render with a user ID so it can retrieve the user from state. const root = mount(<UserProfile userId={1} store={store} />); expect(root.find('span')).toEqual('Kumar'); });
});

Rendering the full component with Enzyme’s mount() makes sure mapStateToProps() is working and that the reducer did what this specific component expected. It simulates what would happen if the real application requested a user from the API and dispatched the result. However, since mount() renders all components including nested components, it doesn’t allow us to test UserProfile in isolation. For that we need a different approach using shallow rendering, explained below.

Shallow rendering for dependency injection

Let’s say the UserProfile component depends on a UserAvatar component to display the user’s photo. It might look like this:

export function UserProfileBase(props) { const { user } = props; return ( <div> <UserAvatar url={user.avatarURL} /> <span>{user.name}</span> </div> );
}

Since UserAvatar will have unit tests of its own, the UserProfile test just has to make sure it calls the interface of UserAvatar correctly. What is its interface? The interface to any React component is simply its properties. Flow helps to validate property data types but we also need tests to check the data values.

With Enzyme, we don’t have to replace dependencies with fakes in a traditional dependency injection sense. We can simply infer their existence through shallow rendering. A test would look something like this:

import UserProfile, { UserProfileBase } from 'src/UserProfile';
import UserAvatar from 'src/UserAvatar';
import { shallowUntilTarget } from './helpers'; describe('<UserProfile>', () => { it('renders a UserAvatar', () => { const user = { userId: 1, avatarURL: 'https://cdn/image.png', }; store.dispatch(actions.loadUser(user)); const root = shallowUntilTarget( <UserProfile userId={1} store={store} />, UserProfileBase ); expect(root.find(UserAvatar).prop('url')) .toEqual(user.avatarURL); });
});

Instead of calling mount(), this test renders the component using a custom helper called shallowUntilTarget(). You may already be familiar with Enzyme’s shallow() but that only renders the first component in a tree. We needed to create a helper called shallowUntilTarget() that will render all “wrapper” (or higher order) components until reaching our target, UserProfileBase.

Hopefully Enzyme will ship a feature similar to shallowUntilTarget() soon, but the implementation is simple. It calls root.dive() in a loop until root.is(TargetComponent) returns true.

With this shallow rendering approach, it is now possible to test UserProfile in isolation yet still dispatch Redux actions like a real application.

The test looks for the UserAvatar component in the tree and simply makes sure UserAvatar will receive the correct properties (the render() function of UserAvatar is never executed). If the properties of UserAvatar change and we forget to update the test, the test might still pass, but Flow will alert us about the violation.

The elegance of both React and shallow rendering just gave us dependency injection for free, without having to inject any dependencies! The key to this testing strategy is that the implementation of UserAvatar is free to evolve on its own in a way that won’t break the UserProfile tests. If changing the implementation of a unit forces you to fix a bunch of unrelated tests, it’s a sign that your testing strategy may need rethinking.

Composing with children, not properties

The power of React and shallow rendering really come into focus when you compose components using children instead of passing JSX via properties. For example, let’s say you wanted to wrap UserAvatar in a common InfoCard for layout purposes. Here’s how to compose them together as children:

export function UserProfileBase(props) { const { user } = props; return ( <div> <InfoCard> <UserAvatar url={user.avatarURL} /> </InfoCard> <span>{user.name}</span> </div> );
}

After making this change, the same assertion from above will still work! Here it is again:

expect(root.find(UserAvatar).prop('url')) .toEqual(user.avatarURL);

In some cases, you may be tempted to pass JSX through properties instead of through children. However, common Enzyme selectors like root.find(UserAvatar) would no longer work. Let’s look at an example of passing UserAvatar to InfoCard through a content property:

export function UserProfileBase(props) { const { user } = props; const avatar = <UserAvatar url={user.avatarURL} />; return ( <div> <InfoCard content={avatar} /> <span>{user.name}</span> </div> );
}

This is still a valid implementation but it’s not as easy to test.

Testing JSX passed through properties

Sometimes you really can’t avoid passing JSX through properties. Let’s imagine that InfoCard needs full control over rendering some header content.

export function UserProfileBase(props) { const { user } = props; return ( <div> <InfoCard header={<Localized>Avatar</Localized>}> <UserAvatar url={user.avatarURL} /> </InfoCard> <span>{user.name}</span> </div> );
}

How would you test this? You might be tempted to do a full Enzyme mount() as opposed to a shallow() render. You might think it will provide you with better test coverage but that additional coverage is not necessary — the InfoCard component will already have tests of its own. The UserProfile test just needs to make sure InfoCard gets the right properties. Here’s how to test that.

import { shallow } from 'enzyme';
import InfoCard from 'src/InfoCard';
import Localized from 'src/Localized';
import { shallowUntilTarget } from './helpers'; describe('<UserProfile>', () => { it('renders an InfoCard with a custom header', () => { const user = { userId: 1, avatarURL: 'https://cdn/image.png', }; store.dispatch(actions.loadUser(user)); const root = shallowUntilTarget( <UserProfile userId={1} store={store} />, UserProfileBase ); const infoCard = root.find(InfoCard); // Simulate how InfoCard will render the // header property we passed to it. const header = shallow( <div>{infoCard.prop('header')}</div> ); // Now you can make assertions about the content: expect(header.find(Localized).text()).toEqual('Avatar'); });
});

This is better than a full mount() because it allows the InfoCard implementation to evolve freely so long as its properties don’t change.

Testing component callbacks

Aside from passing JSX through properties, it’s also common to pass callbacks to React components. Callback properties make it very easy to build abstractions around common functionality. Let’s imagine we are using a FormOverlay component to render an edit form in a UserProfileManager component.

import FormOverlay from 'src/FormOverlay'; export class UserProfileManagerBase extends React.Component { onSubmit = () => { // Pretend that the inputs are controlled form elements and // their values have already been connected to this.state. this.props.dispatch(actions.updateUser(this.state)); } render() { return ( <FormOverlay onSubmit={this.onSubmit}> <input id="nameInput" name="name" /> </FormOverlay> ); }
} // Export the final UserProfileManager component.
export default compose( // Use connect() from react-redux to get props.dispatch() connect(),
)(UserProfileManagerBase);

How do you test the integration of UserProfileManager with FormOverlay? You might be tempted once again to do a full mount(), especially if you’re testing integration with a third-party component, something like Autosuggest. However, a full mount() is not necessary.

Just like in previous examples, the UserProfileManager test can simply check the properties passed to FormOverlay. This is safe because FormOverlay will have tests of its own and Flow will validate the properties. Here is an example of testing the onSubmit property.

import FormOverlay from 'src/FormOverlay';
import { shallowUntilTarget } from './helpers'; describe('<UserProfileManager>', () => { it('updates user information', () => { const store = createNormalReduxStore(); // Create a spy of the dispatch() method for test assertions. const dispatchSpy = sinon.spy(store, 'dispatch'); const root = shallowUntilTarget( <UserProfileManager store={store} />, UserProfileManagerBase ); // Simulate typing text into the name input. const name = 'Faye'; const changeEvent = { target: { name: 'name', value: name }, }; root.find('#nameInput').simulate('change', changeEvent); const formOverlay = root.find(FormOverlay); // Simulate how FormOverlay will invoke the onSubmit property. const onSubmit = formOverlay.prop('onSubmit'); onSubmit(); // Make sure onSubmit dispatched the correct ation. const expectedAction = actions.updateUser({ name }); sinon.assertCalledWith(dispatchSpy, expectedAction); });
});

This tests the integration of UserProfileManager and FormOverlay without relying on the implementation of FormOverlay. It uses sinon to spy on the store.dispatch() method to make sure the correct action is dispatched when the user invokes onSubmit().

Every change starts with a Redux action

The Redux architecture is simple: when you want to change application state, dispatch an action. In the last example of testing the onSubmit() callback, the test simply asserted a dispatch of actions.updateUser(...). That’s it. This test assumes that once the updateUser() action is dispatched, everything will fall into place.

So how would an application like ours actually update the user? We would connect a saga to the action type. The updateUser() saga would be responsible for making a request to the API and dispatching further actions when receiving a response. The saga itself will have unit tests of its own. Since the UserProfileManager test runs without any sagas, we don’t have to worry about mocking out the saga functionality. This architecture makes testing very easy; something like redux-thunk may offer similar benefits.

Summary

These examples illustrate patterns that work really well at addons.mozilla.org for solving common testing problems. Here is a recap of the concepts:

  • We dispatch real Redux actions to test application state changes.
  • We test each component only once using shallow rendering.
  • We resist full DOM rendering (with mount()) as much as possible.
  • We test component integration by checking properties.
  • Static typing helps validate our component properties.
  • We simulate user events and make assertions about what action was dispatched.

Want to get more involved in Firefox Add-ons community? There are a host of ways to contribute to the add-ons ecosystem – and plenty to learn, whatever your skills and level of experience.

Kumar hacks on Mozilla web services and tools for various projects, such as those supporting Firefox Add-ons. He hacks on lots of random open source projects too.

More articles by kumar303…

Original Link

Testing Strategies for React and Redux

When the Firefox Add-ons team ported addons.mozilla.org to a single page app backed by an API, we chose React and Redux for powerful state management, delightful developer tools, and testability. Achieving the testability part isn’t completely obvious since there are competing tools and techniques.

Below are some testing strategies that are working really well for us.

Testing must be fast and effective

We want our tests to be lightning fast so that we can ship high-quality features quickly and without discouragement. Waiting for tests can be discouraging, yet tests are crucial for preventing regressions, especially while restructuring an application to support new features.

Our strategy is to only test what’s necessary and only test it once. To achieve this we test each unit in isolation, faking out its dependencies. This is a technique known as unit testing and in our case, the unit is typically a single React component.

Unfortunately, it’s very difficult to do this safely in a dynamic language such as JavaScript since there is no fast way to make sure the fake objects are in sync with real ones. To solve this, we rely on the safety of static typing (via Flow) to alert us if one component is using another incorrectly — something a unit test might not catch.

A suite of unit tests combined with static type analysis is very fast and effective. We use Jest because it too is fast, and because it lets us focus on a subset of tests when needed.

Testing Redux connected components

The dangers of testing in isolation within a dynamic language are not entirely alleviated by static types, especially since third-party libraries often do not ship with type definitions (creating them from scratch is cumbersome). Also, Redux-connected components are hard to isolate because they depend on Redux functionality to keep their properties in sync with state. We settled on a strategy where we trigger all state changes with a real Redux store. Redux is crucial to how our application runs in the real world so this makes our tests very effective.

As it turns out, testing with a real Redux store is fast. The design of Redux lends itself very well to testing due to how actions, reducers, and state are decoupled from one another. The tests give the right feedback as we make changes to application state. This also makes it feel like a good fit for testing. Aside from testing, the Redux architecture is great for debugging, scaling, and especially development.

Consider this connected component as an example: (For brevity, the examples in this article do not define Flow types but you can learn about how to do that here.)

import { connect } from 'react-redux';
import { compose } from 'redux'; // Define a functional React component.
export function UserProfileBase(props) { return ( <span>{props.user.name}</span> );
} // Define a function to map Redux state to properties.
function mapStateToProps(state, ownProps) { return { user: state.users[ownProps.userId] };
} // Export the final UserProfile component composed of
// a state mapper function.
export default compose( connect(mapStateToProps),
)(UserProfileBase);

You may be tempted to test this by passing in a synthesized user property but that would bypass Redux and all of your state mapping logic. Instead, we test by dispatching a real action to load the user into state and make assertions about what the connected component rendered.

import { mount } from 'enzyme';
import UserProfile from 'src/UserProfile'; describe('<UserProfile>', () => { it('renders a name', () => { const store = createNormalReduxStore(); // Simulate fetching a user from an API and loading it into state. store.dispatch(actions.loadUser({ userId: 1, name: 'Kumar' })); // Render with a user ID so it can retrieve the user from state. const root = mount(<UserProfile userId={1} store={store} />); expect(root.find('span')).toEqual('Kumar'); });
});

Rendering the full component with Enzyme’s mount() makes sure mapStateToProps() is working and that the reducer did what this specific component expected. It simulates what would happen if the real application requested a user from the API and dispatched the result. However, since mount() renders all components including nested components, it doesn’t allow us to test UserProfile in isolation. For that we need a different approach using shallow rendering, explained below.

Shallow rendering for dependency injection

Let’s say the UserProfile component depends on a UserAvatar component to display the user’s photo. It might look like this:

export function UserProfileBase(props) { const { user } = props; return ( <div> <UserAvatar url={user.avatarURL} /> <span>{user.name}</span> </div> );
}

Since UserAvatar will have unit tests of its own, the UserProfile test just has to make sure it calls the interface of UserAvatar correctly. What is its interface? The interface to any React component is simply its properties. Flow helps to validate property data types but we also need tests to check the data values.

With Enzyme, we don’t have to replace dependencies with fakes in a traditional dependency injection sense. We can simply infer their existence through shallow rendering. A test would look something like this:

import UserProfile, { UserProfileBase } from 'src/UserProfile';
import UserAvatar from 'src/UserAvatar';
import { shallowUntilTarget } from './helpers'; describe('<UserProfile>', () => { it('renders a UserAvatar', () => { const user = { userId: 1, avatarURL: 'https://cdn/image.png', }; store.dispatch(actions.loadUser(user)); const root = shallowUntilTarget( <UserProfile userId={1} store={store} />, UserProfileBase ); expect(root.find(UserAvatar).prop('url')) .toEqual(user.avatarURL); });
});

Instead of calling mount(), this test renders the component using a custom helper called shallowUntilTarget(). You may already be familiar with Enzyme’s shallow() but that only renders the first component in a tree. We needed to create a helper called shallowUntilTarget() that will render all “wrapper” (or higher order) components until reaching our target, UserProfileBase.

Hopefully Enzyme will ship a feature similar to shallowUntilTarget() soon, but the implementation is simple. It calls root.dive() in a loop until root.is(TargetComponent) returns true.

With this shallow rendering approach, it is now possible to test UserProfile in isolation yet still dispatch Redux actions like a real application.

The test looks for the UserAvatar component in the tree and simply makes sure UserAvatar will receive the correct properties (the render() function of UserAvatar is never executed). If the properties of UserAvatar change and we forget to update the test, the test might still pass, but Flow will alert us about the violation.

The elegance of both React and shallow rendering just gave us dependency injection for free, without having to inject any dependencies! The key to this testing strategy is that the implementation of UserAvatar is free to evolve on its own in a way that won’t break the UserProfile tests. If changing the implementation of a unit forces you to fix a bunch of unrelated tests, it’s a sign that your testing strategy may need rethinking.

Composing with children, not properties

The power of React and shallow rendering really come into focus when you compose components using children instead of passing JSX via properties. For example, let’s say you wanted to wrap UserAvatar in a common InfoCard for layout purposes. Here’s how to compose them together as children:

export function UserProfileBase(props) { const { user } = props; return ( <div> <InfoCard> <UserAvatar url={user.avatarURL} /> </InfoCard> <span>{user.name}</span> </div> );
}

After making this change, the same assertion from above will still work! Here it is again:

expect(root.find(UserAvatar).prop('url')) .toEqual(user.avatarURL);

In some cases, you may be tempted to pass JSX through properties instead of through children. However, common Enzyme selectors like root.find(UserAvatar) would no longer work. Let’s look at an example of passing UserAvatar to InfoCard through a content property:

export function UserProfileBase(props) { const { user } = props; const avatar = <UserAvatar url={user.avatarURL} />; return ( <div> <InfoCard content={avatar} /> <span>{user.name}</span> </div> );
}

This is still a valid implementation but it’s not as easy to test.

Testing JSX passed through properties

Sometimes you really can’t avoid passing JSX through properties. Let’s imagine that InfoCard needs full control over rendering some header content.

export function UserProfileBase(props) { const { user } = props; return ( <div> <InfoCard header={<Localized>Avatar</Localized>}> <UserAvatar url={user.avatarURL} /> </InfoCard> <span>{user.name}</span> </div> );
}

How would you test this? You might be tempted to do a full Enzyme mount() as opposed to a shallow() render. You might think it will provide you with better test coverage but that additional coverage is not necessary — the InfoCard component will already have tests of its own. The UserProfile test just needs to make sure InfoCard gets the right properties. Here’s how to test that.

import { shallow } from 'enzyme';
import InfoCard from 'src/InfoCard';
import Localized from 'src/Localized';
import { shallowUntilTarget } from './helpers'; describe('<UserProfile>', () => { it('renders an InfoCard with a custom header', () => { const user = { userId: 1, avatarURL: 'https://cdn/image.png', }; store.dispatch(actions.loadUser(user)); const root = shallowUntilTarget( <UserProfile userId={1} store={store} />, UserProfileBase ); const infoCard = root.find(InfoCard); // Simulate how InfoCard will render the // header property we passed to it. const header = shallow( <div>{infoCard.prop('header')}</div> ); // Now you can make assertions about the content: expect(header.find(Localized).text()).toEqual('Avatar'); });
});

This is better than a full mount() because it allows the InfoCard implementation to evolve freely so long as its properties don’t change.

Testing component callbacks

Aside from passing JSX through properties, it’s also common to pass callbacks to React components. Callback properties make it very easy to build abstractions around common functionality. Let’s imagine we are using a FormOverlay component to render an edit form in a UserProfileManager component.

import FormOverlay from 'src/FormOverlay'; export class UserProfileManagerBase extends React.Component { onSubmit = () => { // Pretend that the inputs are controlled form elements and // their values have already been connected to this.state. this.props.dispatch(actions.updateUser(this.state)); } render() { return ( <FormOverlay onSubmit={this.onSubmit}> <input id="nameInput" name="name" /> </FormOverlay> ); }
} // Export the final UserProfileManager component.
export default compose( // Use connect() from react-redux to get props.dispatch() connect(),
)(UserProfileManagerBase);

How do you test the integration of UserProfileManager with FormOverlay? You might be tempted once again to do a full mount(), especially if you’re testing integration with a third-party component, something like Autosuggest. However, a full mount() is not necessary.

Just like in previous examples, the UserProfileManager test can simply check the properties passed to FormOverlay. This is safe because FormOverlay will have tests of its own and Flow will validate the properties. Here is an example of testing the onSubmit property.

import FormOverlay from 'src/FormOverlay';
import { shallowUntilTarget } from './helpers'; describe('<UserProfileManager>', () => { it('updates user information', () => { const store = createNormalReduxStore(); // Create a spy of the dispatch() method for test assertions. const dispatchSpy = sinon.spy(store, 'dispatch'); const root = shallowUntilTarget( <UserProfileManager store={store} />, UserProfileManagerBase ); // Simulate typing text into the name input. const name = 'Faye'; const changeEvent = { target: { name: 'name', value: name }, }; root.find('#nameInput').simulate('change', changeEvent); const formOverlay = root.find(FormOverlay); // Simulate how FormOverlay will invoke the onSubmit property. const onSubmit = formOverlay.prop('onSubmit'); onSubmit(); // Make sure onSubmit dispatched the correct ation. const expectedAction = actions.updateUser({ name }); sinon.assertCalledWith(dispatchSpy, expectedAction); });
});

This tests the integration of UserProfileManager and FormOverlay without relying on the implementation of FormOverlay. It uses sinon to spy on the store.dispatch() method to make sure the correct action is dispatched when the user invokes onSubmit().

Every change starts with a Redux action

The Redux architecture is simple: when you want to change application state, dispatch an action. In the last example of testing the onSubmit() callback, the test simply asserted a dispatch of actions.updateUser(...). That’s it. This test assumes that once the updateUser() action is dispatched, everything will fall into place.

So how would an application like ours actually update the user? We would connect a saga to the action type. The updateUser() saga would be responsible for making a request to the API and dispatching further actions when receiving a response. The saga itself will have unit tests of its own. Since the UserProfileManager test runs without any sagas, we don’t have to worry about mocking out the saga functionality. This architecture makes testing very easy; something like redux-thunk may offer similar benefits.

Summary

These examples illustrate patterns that work really well at addons.mozilla.org for solving common testing problems. Here is a recap of the concepts:

  • We dispatch real Redux actions to test application state changes.
  • We test each component only once using shallow rendering.
  • We resist full DOM rendering (with mount()) as much as possible.
  • We test component integration by checking properties.
  • Static typing helps validate our component properties.
  • We simulate user events and make assertions about what action was dispatched.

Want to get more involved in Firefox Add-ons community? There are a host of ways to contribute to the add-ons ecosystem – and plenty to learn, whatever your skills and level of experience.

Kumar hacks on Mozilla web services and tools for various projects, such as those supporting Firefox Add-ons. He hacks on lots of random open source projects too.

More articles by kumar303…

Original Link

Hello wasm-pack!

2 panels, one showing ferris the crab with assorted rust and wasm packages and one with the npm wombat with assorted js wasm and css/html packages. the crab is throwing a package over towards the wombat

As Lin Clark emphasizes in her article about Rust and WebAssembly: the goal of WebAssembly is not to replace JavaScript, but to be an awesome tool to use with JavaScript. Lots of amazing work has been done to simplify crossing the language boundary between JavaScript and WebAssembly, and you can read all about that in Alex Crichton’s post on wasm-bindgen. This post focuses on a different type of JavaScript/Rust integration: package ecosystem and developer workflows.

Both Rust and JavaScript have vibrant package ecosystems. Rust has cargo and crates.io. JavaScript has several CLI tools, including the npm CLI, that interface with the npm registry. In order for WebAssembly to be successful, we need these two systems to work well together, specifically:

  • Rust developers should be able to produce WebAssembly packages for use in JavaScript without requiring a Node.js development environment
  • JavaScript developers should be able to use WebAssembly without requiring a Rust development environment

✨📦 Enter: wasm-pack.

wasm-pack is a tool for assembling and packaging Rust crates that target WebAssembly. These packages can be published to the npm Registry and used alongside other packages. This means you can use them side-by-side with JS and other packages, and in many kind of applications, be it a Node.js server side app, a client-side application bundled by Webpack, or any other sort of application that uses npm dependencies. You can find wasm-pack on crates.io and GitHub.

Development of this tooling has just begun and we’re excited to get more developers from both the Rust and JavaScript worlds involved. Both the JavaScript and Rust ecosystems are focused on developer experience. We know first hand that the key to a productive and happy ecosystem is good tools that automate the boring tasks and get out of the developer’s way. In this article, I’ll talk about where we are, where we’re headed, how to get started using the tooling now, and how to get involved in shaping its future.

💁 What it does today

ferris stands between 2 open packages, one labelled rust, one labelled npm. there is a flow from the Rust package to the npm package with 4 stages. first stage: a lib.rs and cargo.toml file, then a .wasm file, then a .wasm and a .js file, then a .wasm, .js, package.json and a readme

Today, wasm-pack walks you through four basic steps to prepare your Rust code to be published as a WebAssembly package to the npm registry:

1. Compile to WebAssembly

wasm-pack will add the appropriate WebAssembly compilation target using rustup and will compile your Rust to Web Assembly in release mode.

To do this, wasm-pack will:

  • Add the wasm32-unknown-unknown compilation target, if needed
  • Compile your Rust project for release using the wasm target

2. Run wasm-bindgen

wasm-pack wraps the CLI portion of the wasm-bindgen tool and runs it for you! This does things like wrapping your WebAssembly module in JS wrappers which make it easier for people to interact with your module. wasm-bindgen supports both ES6 modules and CommonJS and you can use wasm-pack to produce either type of package!

To do this, wasm-pack will:

  • If needed, install and/or update wasm-bindgen
  • Run wasm-bindgen, generating a new .wasm file and a .js file
  • Moves the generated files to a new pkg directory

3. Generate package.json

wasm-pack reads your Cargo.toml and generates the package.json file necessary to publish your package to the npm registry. It will:

To do this, wasm-pack will:

  • Copy over your project name and description
  • Link to your Rust project’s repository
  • List the generated JavaScript files in the files key. This ensures that those files, and only those files, are included in your npm package. This is particularly important for ensuring good performance if you intend to use this package, or a bundle including this package, in the browser!

4. Documentation

wasm-pack will copy your Rust project’s README.md to the npm package it produces. We’ve got a lot of great ideas about extending this further to support the Rust ecosystem’s documentation feature, rustdoc– more on the in the next section!

🔮 Future Plans

Integrate with rustdoc

The crates.io team surveyed developers, and learned that good documentation was the number one feature that developers looked for when evaluating the use of crate. Contributor Yoshua Wuyts introduced the brilliant idea of generating further README.md content by integrating wasm-pack with the Rust API documentation tool, rustdoc. The Rust-wasm team is committed to making Rust a first class way to write WebAssembly. Offering documentation for Rust-generated WebAssembly packages that’s both easy to write and easy to discover aligns neatly with our goals. Read more about the team’s thoughts in this issue and join in the discussion!

Manage and Optimize your Rust and JS dependency graphs

The next large piece of development work on wasm-pack will focus on using custom segments in compiled WebAssembly to declare dependencies on local Javascript files or other npm packages.

The preliminary work for this feature has already landed in wasm-bindgen, so the next step will be integrating it into wasm-pack. The naive integration won’t be too tricky- but we’re excited to explore the opportunities we have to streamline and optimize Rust dependency trees that contain npm dependencies on several levels! This work will be similar to the optimizations that bundlers like webpack deliver, but on the level of Rust dependencies.

There’s a lot of questions we still have to answer and there’s going be a lot of neat engineering work to do. In a few weeks there will be a full post on this topic, so keep an eye out!

ferris is sitting in a package on a scale, in the distance several interconnected and dependent packages are linked with lines flowing into the package. the scale says "heavy"

Grow Node.js toolchain in Rust

The largest and most ambitious goal of this project is to rewrite the required npm login, npm pack and npm publish steps in Rust so that the required dependency on a Node.js development environment becomes optional for those who don’t currently use Node.js in their workflow. As we’ve said before, we want to ensure that both WebAssembly package producers and users can remain in their familiar workflows. Currently, that is true for JavaScript developers- they do not need to have a Rust development environment or any knowledge of Rust to get started using a Rust-generated WebAssembly module that’s been published with wasm-pack. However, Rust developers still need to install Node.js and npm to publish with wasm-pack, we’re excited to change that by writing a npm package publisher in Rust- and who knows, perhaps we can eventually integrate some Rust elements (perhaps compiled to WebAssembly!) into the npm client!

Further collaboration with npm and bundlers

We’re always communicating with the npm CLI team members Kat Marchan and Rebecca Turner, as well as the folks who work on webpack and Parcel– we’re excited to keep working with them to make it easy for developers to release and use WebAssembly code!

🛠 Start using it today!

wasm-pack is currently a command line tool distributed via Cargo. To install it, setup a Rust development environment, and then run:

cargo install wasm-pack

If you aren’t sure where to start, we have a tutorial for you! This tutorial, by Michael Gattozzi and the Rust-wasm working group, walks you through:

  • writing a small Rust library
  • compiling it to WebAssembly, packaging, and publishing with wasm-pack
  • bundling with webpack to produce a small website

a gif of the wasm pack CLI tool. first we ls a directory with a rust crate, then we run wasm pack. it completes in one minute, then we ls the target directory to see that a wasm binary was compiled, then we ls the pkg directory to see that an npm package was created

👯‍♀️Contribute

The key to all excellent developer tooling is a short feedback cycle between developers of the tool and developers using the tool in their day to day workflows. In order to be successful with wasm-pack, and all of our WebAssembly developer tooling, we need developers of all skill levels and backgrounds to get involved!

Take a look at our Contributor Guidelines and our Issue Tracker (we regularly label things as “good first issue” and provide mentors and mentoring instructions!)- we’re excited to work with you!

Ashley Williams is an engineer at Integer32, contracting for Mozilla on the Rust Programming Language. She is a member of the Rust Core team, leads the Rust Community Team, and helps maintain Rust’s package registry, crates.io. Previously, she worked as an engineer at npm, the package manager for Javascript, and currently is the Individual Membership Director on the Node.js Foundation Board of Directors. A long time teacher, Ashley has focused much of her energies on education programs for open source projects, founding NodeTogether in 2016 and currently leading the RustBridge initiative. She has represented teachers’ perspectives at TC39 meetings to influence the growth of JavaScript and continues to be passionate about growing the web through her work on Web Assembly.

More articles by Ashley Williams…

Original Link