Raw String Literals¶

Abstract¶

This JEP proposes the following modifications to JMESPath in order to improve the usability of the language and ease the implementation of parsers:

• Addition of a raw string literal to JMESPath that will allow expressions to contain raw strings that are not mutated by JSON escape sequences (e.g., “\n”, “\r”, “\u005C”).
• Deprecation of the current literal parsing behavior that allows for unquoted JSON strings to be parsed as JSON strings, removing an ambiguity in the JMESPath grammar and helping to ensure consistency among implementations.

This proposal seeks to add the following syntax to JMESPath:

'foobar'
'foo\'bar'
`bar` -> Parse error/warning (implementation specific)

Motivation¶

Raw string literals are provided in various programming languages in order to prevent language specific interpretation (i.e., JSON parsing) and remove the need for escaping, avoiding a common problem called leaning toothpick syndrome (LTS). Leaning toothpick syndrome is an issue in which strings become unreadable due to excessive use of escape characters in order to avoid delimiter collision (e.g., \\\\\\).

When evaluating a JMESPath expression, it is often necessary to utilize string literals that are not extracted from the data being evaluated, but rather statically part of the compiled JMESPath expression. String literals are useful in many areas, but most notably when invoking functions or building up multi-select lists and hashes.

The following expression returns the number of characters found in the string foo. When parsing this expression, `"foo"` is parsed as a JSON value which produces the string literal value of foo:

`"foo"`

The following expression is functionally equivalent. Notice that the quotes are elided from the JSON literal:

`foo`

These string literals are parsed using a JSON parser according to RFC 4627, which will expand unicode escape sequences, newline characters, and several other escape sequences documented in RFC 4627 section 2.5.

For example, the use of an escaped unicode value \u002B is expanded into + in the following JMESPath expression:

`"foo\u002B"` -> "foo+"

You can escape escape sequences in JSON literals to prevent an escape sequence from being expanded:

`"foo\\u002B"` -> "foo\u002B"
`foo\\u002B` -> "foo\u002B"

While this allows you to provide literal strings, it presents the following problems:

1. Incurs an additional JSON parsing penalty.

2. Requires the cognitive overhead of escaping escape characters if you actually want the data to be represented as it was literally provided (which can lead to LTS). If the data being escaped was meant to be used along with another language that uses \ as an escape character, then the number of backslash characters doubles.

3. Introduces an ambiguous rule to the JMESPath grammar that requires a prose based specification to resolve the ambiguity in parser implementations.

   The relevant literal grammar rules are currently defined as follows:

   literal = "`" json-value "`"
   literal =/ "`" 1*(unescaped-literal / escaped-literal) "`"
   unescaped-literal = %x20-21 /       ; space !
                           %x23-5B /   ; # - [
                           %x5D-5F /   ; ] ^ _
                           %x61-7A     ; a-z
                           %x7C-10FFFF ; |}~ ...
   escaped-literal   = escaped-char / (escape %x60)
   json-value = false / null / true / json-object / json-array /
                json-number / json-quoted-string
   false = %x66.61.6c.73.65   ; false
   null  = %x6e.75.6c.6c      ; null
   true  = %x74.72.75.65      ; true
   json-quoted-string = %x22 1*(unescaped-literal / escaped-literal) %x22
   begin-array     = ws %x5B ws  ; [ left square bracket
   begin-object    = ws %x7B ws  ; { left curly bracket
   end-array       = ws %x5D ws  ; ] right square bracket
   end-object      = ws %x7D ws  ; } right curly bracket
   name-separator  = ws %x3A ws  ; : colon
   value-separator = ws %x2C ws  ; , comma
   ws              = *(%x20 /              ; Space
                       %x09 /              ; Horizontal tab
                       %x0A /              ; Line feed or New line
                       %x0D                ; Carriage return
                      )
   json-object = begin-object [ member *( value-separator member ) ] end-object
   member = quoted-string name-separator json-value
   json-array = begin-array [ json-value *( value-separator json-value ) ] end-array
   json-number = [ minus ] int [ frac ] [ exp ]
   decimal-point = %x2E       ; .
   digit1-9 = %x31-39         ; 1-9
   e = %x65 / %x45            ; e E
   exp = e [ minus / plus ] 1*DIGIT
   frac = decimal-point 1*DIGIT
   int = zero / ( digit1-9 *DIGIT )
   minus = %x2D               ; -
   plus = %x2B                ; +
   zero = %x30                ; 0
   

   The literal rule is ambiguous because unescaped-literal includes all of the same characters that json-value match, allowing any value that is valid JSON to be matched on either unescaped-literal or json-value.

Specification¶

A raw string literal is value that begins and ends with a single quote, does not interpret escape characters, and may contain escaped single quotes to avoid delimiter collision.

raw-string        = "'" *raw-string-char "'"
; The first grouping matches any character other than "\"
raw-string-char   = (%x20-26 / %x28-5B / %x5D-10FFFF) / raw-string-escape
raw-string-escape = escape ["'"]

literal = "`" json-value "`"

Impact¶

The impact to existing users of JMESPath is that the use of a JSON literal in which the quotes are elided SHOULD be converted to use the string-literal rule of the grammar. Whether or not this conversion is absolutely necessary will depend on the specific JMESPath implementation.

Implementations MAY choose to support the old syntax of allowing elided quotes in JSON literal expressions. If an implementation chooses this approach, the implementation SHOULD raise some kind of warning to the user to let them know of the deprecation and possible incompatibility with other JMESPath implementations.

In order to support this type of variance in JMESPath implementations, all of the JSON literal compliance test cases that involve elided quotes MUST be removed, and test cases regarding failing on invalid unquoted JSON values MUST not be allowed in the compliance test unless placed in a JEP 12 specific test suite, allowing implementations that support elided quotes in JSON literals to filter out the JEP 12 specific test cases.

Alternative approaches¶

There are several alternative approaches that could be taken.

`[1`

[==[foo=bar]==] -- parsed as "foo=bar"