Utf8 text

Purpose: Convert UTF8 string to text.

 utf8-text <utf8> \
     [ to <text> ] \
     [ length <length> ] \
     [ status <status> ] \
     [ error-text <error text> ]

utf8-text will convert <utf8> text to <text> (specified with "to" clause). If <text> is omitted, then the result of conversion is output.

<utf8> is a string that may contain UTF8 characters (as 2, 3 or 4 bytes representing a unicode character). Encoding creates a string that can be used as a value where text representation of UTF8 is required. utf8-text is performed according to RFC7159 and RFC3629 (UTF8 standard).

Note that hexadecimal characters used for Unicode (such as \u21d7) are always lowercase. Solidus character ("/") is not escaped, although text-utf8 will correctly process it if the input has it escaped.

The number of bytes in <utf8> to be converted can be specified with <length> in "length" clause. If <length> is not specified, it is the length of string <utf8>. Note that a single UTF-8 character can be anywhere between 1 to 4 bytes. For example "љ" is 2 bytes in length.

The status of encoding can be obtained in number <status>. <status> is the string length of the result in <text> (or the number of bytes output if <text> is omitted), or -1 if error occurred (meaning <utf8> is an invalid UTF8) in which case <text> (if specified) is an empty string and the error text can be obtained in <error text> in "error-text" clause.

Examples

Convert UTF8 string to text and verify the expected result:

 // UTF8 string 
 set-string utf8_str = "\"Doc\"\n\t\b\f\r\t⇗⇘\t▷◮𝄞ᏫⲠш\n/\"()\t"

 // Convert UTF8 string to text
 utf8-text utf8_str status encstatus to text_text

 // This is the text expected
 (( expected_result
 @\"Doc\"\n\t\b\f\r\t\u21d7\u21d8\t\u25b7\u25ee\ud834\udd1e\u13eb\u2ca0\u0448\n/\"()\t
 ))

 // Make sure conversion was okay, decs is the length of the result (encj string)
 if-true text_text equal expected_result and encstatus not-equal -1
     @decode-text worked okay
 end-if