Whitespace and Line Terminators

Status: implemented; JavaScript's other whitespace is refused, see the page

Whitespace is a space (U+0020) or a tab (U+0009). A newline is a line feed (LF, U+000A), a carriage return (CR, U+000D), or CR followed by LF, which is one line break. Those are JSON's four whitespace characters, the same four DataJS admits.

Trivia may stand between any two tokens, before the first and after the last, and it is insignificant there, newlines included, but for one: a statement written without its ; ends at the newline before the next statement (module structure). Inside an expression a line break reads as a space does, exactly as JavaScript reads it: a followed by .length, [0] or + 2 on the next line is a.length, a[0] or a + 2, and f followed by (1) is the call f(1). Trivia is needed only where two tokens would otherwise read as one: const a, not consta (module structure); 1 .x, not 1.x (property access).

The exceptions are the boundaries JavaScript restricts, before => and after return and throw, where a line break is refused (functions). There the trivia is a line break if a newline of any of the three forms stands in it, inside a block comment or not (comments).

Nothing else is trivia. JavaScript's other whitespace — VT (U+000B), FF (U+000C), U+FEFF, and the Unicode space separators such as NBSP (U+00A0) and U+3000 — is refused, although JavaScript accepts it, a byte order mark at the start of the file included (source text). JavaScript's other two line terminators, U+2028 and U+2029, may stand raw only inside a string, as in JSON; anywhere else, a comment included, they are refused, since an invisible line break is no line break here. A hashbang comment, #! at the start of the file, is refused too, although JavaScript admits one (hash comments). Any other character outside a string or a comment is part of a token or an error, as it is in JavaScript.

Examples

The compiler demo offers the programs whitespace in module.f.js, each with its real output: one accepted, one refused with the compiler's own message.

The demo on this page runs sample.f.js, this page's example as a function of one number — tabs, spaces and line breaks between the tokens of one expression — with every output fjs compile makes of it: the expression graph (EDAG), the Rust and the WebAssembly that Rust builds to. Its Run button calls the default export with 1 and 2, as JavaScript and as WebAssembly, and shows what each call returned or threw.

export default (...a) =>
	a[0]
    +
	1;

Reference

Demo

In FunctionalScript

fjs compile accepts this module, and it runs as JavaScript and as WebAssembly alike.

A FunctionalScript module and what fjs compile makes of it: the expression graph it lowers to, the Rust it compiles to on the NaNVM runtime, and the WebAssembly that Rust builds to. Run calls its default export with 1, then 2, first as the JavaScript it already is and then as WebAssembly, and shows what each call returned or threw and how long one call takes, timed over 1000 calls.

FunctionalScript source:

/**
 * The example of [`README.md`](./README.md) (Whitespace and Line
 * Terminators) as a function of one number, for the page's WASM demo: tabs,
 * spaces and line breaks between the tokens of one expression.
 *
 * @module
 */

/** @type {(...a: readonly number[]) => unknown} */
export default (...a) =>
	a[0]
    +
	1;

Expression graph (EDAG) that fjs compile lowers it to:

=>lengthbody0++leftright.1.objidxrest0

The same graph, as fjs compile writes it to a .edag.data.js file:

export default ["{}",[[":","default",["=>",0,[],["+",[".",["rest"],0],1]]]]];

Rust that fjs compile writes from it, on the NaNVM runtime:

// @generated by `fjs compile`. Do not edit: recompile the source module instead.

use nanvm_lib::vm::unstable::{f64_any, string_key};
use nanvm_lib::vm::{Any, Array, IStaticFunction, ToAny, ToObject};

#[rustfmt::skip]
#[allow(clippy::if_same_then_else)]
pub fn module<A: IStaticFunction>(_runtime: Any<A>) -> Result<Any<A>, Any<A>> {
    Ok([(string_key("default"), A::static_function(|_self, _args| {
        let _b132: Any<A> = _args.rest(0).to_any();
        (Any::dot(_b132.clone(), f64_any(0x0000000000000000)).end())? + f64_any(0x3ff0000000000000)
    }, 0, Array::default(), Some("(...$0)=>$0[0]+1")).to_any())].to_object().to_any())
}

WebAssembly text (WAT) of the functions compiled from this source, without the NaNVM runtime:

  (func $nanvm_demos::samples::whitespace::module::<nanvm_lib::naive::Naive> (;291;) (type 1) (param i32 i32)
    (local i32 i32 i32 i32 i32 i32 i32)
    global.get $__stack_pointer
    i32.const 112
    i32.sub
    local.tee 2
    global.set $__stack_pointer
    local.get 2
    i32.const 255
    i32.store8 offset=40
    local.get 2
    i32.const 0
    i32.store16 offset=72
    local.get 2
    i32.const 1057927
    i32.store offset=68
    local.get 2
    i32.const 1057920
    i32.store offset=64
    local.get 2
    local.get 2
    i32.const 40
    i32.add
    i32.store offset=76
    local.get 2
    i32.const 24
    i32.add
    local.get 2
    i32.const 64
    i32.add
    call $"<core::iter::adapters::GenericShunt<core::iter::adapters::map::Map<core::str::iter::EncodeUtf16, core::result::Result<u16, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>::Ok>, core::result::Result<core::convert::Infallible, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>> as alloc::rc::ToRcSlice<u16>>::to_rc_slice"
    local.get 2
    local.get 2
    i32.load offset=28
    local.tee 3
    i32.store offset=60
    local.get 2
    local.get 2
    i32.load offset=24
    local.tee 4
    i32.store offset=56
    block ;; label = @1
      local.get 2
      i32.load8_u offset=40
      local.tee 5
      i32.const 255
      i32.eq
      br_if 0 (;@1;)
      local.get 2
      i32.load offset=44
      local.set 0
      local.get 2
      i32.load offset=48
      local.set 3
      local.get 2
      i32.load offset=52
      local.set 1
      local.get 2
      i32.load16_u offset=41 align=1
      local.set 6
      local.get 2
      i32.load8_u offset=43
      local.set 7
      local.get 4
      local.get 4
      i32.load
      i32.const -1
      i32.add
      local.tee 8
      i32.store
      local.get 2
      local.get 7
      i32.store8 offset=38
      local.get 2
      local.get 6
      i32.store16 offset=36
      block ;; label = @2
        local.get 8
        br_if 0 (;@2;)
        local.get 2
        i32.const 56
        i32.add
        call $"<alloc::rc::Rc<[u16]>>::drop_slow"
      end
      local.get 2
      local.get 5
      i32.store8 offset=64
      local.get 2
      local.get 2
      i32.load16_u offset=36
      i32.store16 offset=65 align=1
      local.get 2
      local.get 2
      i32.load8_u offset=38
      i32.store8 offset=67
      local.get 2
      local.get 1
      i32.store offset=76
      local.get 2
      local.get 3
      i32.store offset=72
      local.get 2
      local.get 0
      i32.store offset=68
      i32.const 1057960
      i32.const 43
      local.get 2
      i32.const 64
      i32.add
      i32.const 1057944
      i32.const 1058004
      call $core::result::unwrap_failed
      unreachable
    end
    local.get 2
    i32.const 0
    i32.store8 offset=40
    local.get 2
    i32.const 16
    i32.add
    local.get 2
    i32.const 40
    i32.add
    call $"<core::iter::adapters::GenericShunt<core::iter::adapters::map::Map<core::iter::sources::empty::Empty<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>, core::result::Result<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>, ()>::Ok>, core::result::Result<core::convert::Infallible, ()>> as alloc::rc::ToRcSlice<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>>::to_rc_slice"
    local.get 2
    local.get 2
    i32.load offset=20
    local.tee 6
    i32.store offset=68
    local.get 2
    local.get 2
    i32.load offset=16
    local.tee 5
    i32.store offset=64
    block ;; label = @1
      block ;; label = @2
        local.get 2
        i32.load8_u offset=40
        i32.const 1
        i32.ne
        br_if 0 (;@2;)
        local.get 5
        local.get 5
        i32.load
        i32.const -1
        i32.add
        local.tee 0
        i32.store
        local.get 0
        br_if 1 (;@1;)
        local.get 2
        i32.const 64
        i32.add
        call $"<alloc::rc::Rc<[nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>]>>::drop_slow"
        br 1 (;@1;)
      end
      local.get 5
      i32.eqz
      br_if 0 (;@1;)
      i32.const 57
      i32.const 0
      local.get 5
      local.get 6
      i32.const 1057927
      i32.const 16
      call $"<nanvm_lib::naive::Naive as nanvm_lib::vm::internal::IVm>::static_function"
      local.set 5
      local.get 2
      i32.const 0
      i32.store8 offset=56
      local.get 2
      local.get 5
      i32.store offset=84
      local.get 2
      i32.const 7
      i32.store8 offset=80
      local.get 2
      local.get 3
      i32.store offset=76
      local.get 2
      local.get 4
      i32.store offset=72
      local.get 2
      i64.const 4294967296
      i64.store offset=64
      local.get 2
      local.get 2
      i32.const 56
      i32.add
      i32.store offset=96
      local.get 2
      i32.const 8
      i32.add
      local.get 2
      i32.const 64
      i32.add
      call $"<core::iter::adapters::GenericShunt<core::iter::adapters::map::Map<core::array::iter::IntoIter<(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>), 1>, core::result::Result<(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>), ()>::Ok>, core::result::Result<core::convert::Infallible, ()>> as alloc::rc::ToRcSlice<(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>)>>::to_rc_slice"
      local.get 2
      local.get 2
      i32.load offset=12
      local.tee 4
      i32.store offset=44
      local.get 2
      local.get 2
      i32.load offset=8
      local.tee 5
      i32.store offset=40
      block ;; label = @2
        block ;; label = @3
          local.get 2
          i32.load8_u offset=56
          i32.const 1
          i32.ne
          br_if 0 (;@3;)
          local.get 5
          local.get 5
          i32.load
          i32.const -1
          i32.add
          local.tee 0
          i32.store
          local.get 0
          br_if 1 (;@2;)
          local.get 2
          i32.const 40
          i32.add
          call $"<alloc::rc::Rc<[(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>)]>>::drop_slow"
          br 1 (;@2;)
        end
        local.get 5
        i32.eqz
        br_if 0 (;@2;)
        local.get 0
        local.get 4
        i32.store offset=16
        local.get 0
        local.get 5
        i32.store offset=12
        local.get 0
        i32.const 5
        i32.store8 offset=8
        local.get 0
        i64.const 0
        i64.store
        local.get 1
        call $"#func272 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
        local.get 2
        i32.const 112
        i32.add
        global.set $__stack_pointer
        return
      end
      i32.const 1064316
      i32.const 43
      local.get 2
      i32.const 111
      i32.add
      i32.const 1064300
      i32.const 1064360
      call $core::result::unwrap_failed
      unreachable
    end
    i32.const 1064316
    i32.const 43
    local.get 2
    i32.const 111
    i32.add
    i32.const 1064300
    i32.const 1064360
    call $core::result::unwrap_failed
    unreachable
  )
  (func $"<nanvm_demos::samples::whitespace::module<nanvm_lib::naive::Naive>::{closure#0} as core::ops::function::FnOnce<(&nanvm_lib::naive::function::Function, nanvm_lib::vm::array::Array<nanvm_lib::naive::Naive>)>>::call_once" (;292;) (type 4) (param i32 i32 i32 i32)
    (local i32 i32)
    global.get $__stack_pointer
    i32.const 128
    i32.sub
    local.tee 4
    global.set $__stack_pointer
    local.get 4
    local.get 3
    i32.store offset=12
    local.get 4
    local.get 2
    i32.store offset=8
    block ;; label = @1
      local.get 2
      i32.load
      local.tee 5
      i32.const -1
      i32.eq
      br_if 0 (;@1;)
      local.get 4
      local.get 3
      i32.store offset=24
      local.get 4
      local.get 2
      i32.store offset=20
      local.get 4
      i32.const 6
      i32.store8 offset=16
      local.get 2
      local.get 5
      i32.const 2
      i32.add
      local.tee 5
      i32.store
      local.get 5
      i32.eqz
      br_if 0 (;@1;)
      local.get 4
      local.get 2
      i32.store offset=116
      local.get 4
      i32.const 6
      i32.store8 offset=112
      local.get 4
      local.get 3
      i64.extend_i32_u
      i64.store offset=120
      local.get 4
      i64.const 0
      i64.store offset=88
      local.get 4
      i32.const 2
      i32.store8 offset=80
      local.get 4
      i32.const 48
      i32.add
      local.get 4
      i32.const 112
      i32.add
      local.get 4
      i32.const 80
      i32.add
      call $<nanvm_lib::vm::lambda::member::Member<nanvm_lib::naive::Naive>>::new
      block ;; label = @2
        block ;; label = @3
          block ;; label = @4
            local.get 4
            i32.load8_u offset=48
            i32.const 255
            i32.ne
            br_if 0 (;@4;)
            local.get 0
            local.get 4
            i64.load offset=64
            i64.store offset=16
            local.get 0
            local.get 4
            i64.load offset=56
            i64.store offset=8
            local.get 0
            i64.const 1
            i64.store
            local.get 4
            i32.const 16
            i32.add
            call $"#func272 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
            local.get 2
            local.get 2
            i32.load
            i32.const -1
            i32.add
            local.tee 3
            i32.store
            local.get 3
            i32.eqz
            br_if 1 (;@3;)
            br 2 (;@2;)
          end
          local.get 4
          local.get 4
          i64.load offset=72
          i64.store offset=104
          local.get 4
          local.get 4
          i64.load offset=64
          i64.store offset=96
          local.get 4
          local.get 4
          i64.load offset=56
          i64.store offset=88
          local.get 4
          local.get 4
          i64.load offset=48
          i64.store offset=80
          local.get 4
          i32.const 112
          i32.add
          local.get 4
          i32.const 80
          i32.add
          call $<nanvm_lib::vm::lambda::member::Member<nanvm_lib::naive::Naive>>::own
          block ;; label = @4
            block ;; label = @5
              local.get 4
              i32.load8_u offset=112
              i32.const 255
              i32.eq
              br_if 0 (;@5;)
              local.get 4
              local.get 4
              i64.load offset=120
              i64.store offset=40
              local.get 4
              local.get 4
              i64.load offset=112
              i64.store offset=32
              br 1 (;@4;)
            end
            local.get 4
            i32.const 256
            i32.store16 offset=32
          end
          local.get 4
          i32.const 80
          i32.add
          call $"#func693 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 4
          i32.const 96
          i32.add
          call $"#func693 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 4
          i64.const 4607182418800017408
          i64.store offset=88
          local.get 4
          i32.const 2
          i32.store8 offset=80
          local.get 0
          local.get 4
          i32.const 32
          i32.add
          local.get 4
          i32.const 80
          i32.add
          call $"<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive> as core::ops::arith::Add>::add"
          local.get 4
          i32.const 16
          i32.add
          call $"#func272 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 2
          local.get 2
          i32.load
          i32.const -1
          i32.add
          local.tee 3
          i32.store
          local.get 3
          br_if 1 (;@2;)
        end
        local.get 4
        i32.const 8
        i32.add
        call $"<alloc::rc::Rc<[nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>]>>::drop_slow"
      end
      local.get 4
      i32.const 128
      i32.add
      global.set $__stack_pointer
      return
    end
    unreachable
  )

Emergent Testing

Idle. Press Run to start the suite.

  • ./proof.f.mjs