Can we process std::vector<string> with the strigi header?

[koheiw]

This might be too much to ask, but is there a way to process std::vector<string> using functions from stringi? I want to do this because we cannot pass CharacterVector to child processes of RcppParallel (Intel TBB).

More specifically, I want to apply stri_sub() to a CharacterVectors with millions of elements with different from and to values, and combined the results.

[gagolews]

Unfortunately, (currently) the only alternative is to use the SEXP interface

For example, to access stri_sub, use:

SEXP stri_sub(SEXP e1, SEXP e2, SEXP e3, SEXP e4, SEXP e5, SEXP e6) {
      static SEXP(*fun)(SEXP,SEXP,SEXP,SEXP,SEXP,SEXP) = NULL;
      if (!fun)
         fun = (SEXP(*)(SEXP,SEXP,SEXP,SEXP,SEXP,SEXP)) R_GetCCallable("stringi", "C_stri_sub");
      return fun(e1, e2, e3, e4, e5, e6);
   }

[koheiw]

Thanks for reply. I don't think TBB allows SEXP either 😞 If we could call from ICU functions from TBB, our packages become massively faster, especially tokenization. We will not give up on this yet....

[gagolews]

But for substring creation you don't need ICU at all...

• see the U8_FWD_1 macro in https://github.com/unicode-org/icu/blob/main/icu4c/source/common/unicode/utf8.h

You can implement that in C yourself, quite easily, I think.

[koheiw]

Very interesting. Does this mean that we can split character boundaries of UTF-8 just by finding these byte marks?

https://github.com/unicode-org/icu/blob/981c182a7f3583fe262552d5bdea2c9f83f0d5b2/icu4c/source/common/unicode/utf8.h#L185

[koheiw]

I suppose it will be like this: https://stackoverflow.com/a/40054802/7050091

[gagolews]

Indeed! See the Code point ↔ UTF-8 conversion Table in https://en.wikipedia.org/wiki/UTF-8#Encoding

The Stack Overflow code seems okay, otherwise, you can inspire yourself by how ICU computes the number of bytes per a code point:

#define U8_LENGTH(c) \
    ((uint32_t)(c)<=0x7f ? 1 : \
        ((uint32_t)(c)<=0x7ff ? 2 : \
            ((uint32_t)(c)<=0xd7ff ? 3 : \
                ((uint32_t)(c)<=0xdfff || (uint32_t)(c)>0x10ffff ? 0 : \
                    ((uint32_t)(c)<=0xffff ? 3 : 4)\
                ) \
            ) \
        ) \
    )

However, these simple solutions only work under the assumption that the UTF-8 streams are not ill-formed....

[gagolews]

Try this minimal subset (icu_utf8_subset.h):

// This is a subset of the original file:
// https://github.com/unicode-org/icu/blob/a08ac00c6757c117f33c319f50388c869b730bfe/icu4c/source/common/unicode/utf8.h




// © 2016 and later: Unicode, Inc. and others.
// License & terms of use: http://www.unicode.org/copyright.html
/*
*******************************************************************************
*
*   Copyright (C) 1999-2015, International Business Machines
*   Corporation and others.  All Rights Reserved.
*
*******************************************************************************
*   file name:  utf8.h
*   encoding:   UTF-8
*   tab size:   8 (not used)
*   indentation:4
*
*   created on: 1999sep13
*   created by: Markus W. Scherer
*/


#ifndef __ICU_UTF8_SUBSET_H__
#define __ICU_UTF8_SUBSET_H__

#include <stdbool.h>



#define U8_NEXT_OR_FFFD(s, i, length, c) U8_INTERNAL_NEXT_OR_SUB(s, i, length, c, 0xfffd)

#define U8_IS_SINGLE(c) (((c)&0x80)==0)

#define U8_LEAD3_T1_BITS "\x20\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x10\x30\x30"

#define U8_LEAD4_T1_BITS "\x00\x00\x00\x00\x00\x00\x00\x00\x1E\x0F\x0F\x0F\x00\x00\x00\x00"

#define U8_INTERNAL_NEXT_OR_SUB(s, i, length, c, sub) { \
    (c)=(uint8_t)(s)[(i)++]; \
    if(!U8_IS_SINGLE(c)) { \
        uint8_t __t = 0; \
        if((i)!=(length) && \
            /* fetch/validate/assemble all but last trail byte */ \
            ((c)>=0xe0 ? \
                ((c)<0xf0 ?  /* U+0800..U+FFFF except surrogates */ \
                    U8_LEAD3_T1_BITS[(c)&=0xf]&(1<<((__t=(s)[i])>>5)) && \
                    (__t&=0x3f, 1) \
                :  /* U+10000..U+10FFFF */ \
                    ((c)-=0xf0)<=4 && \
                    U8_LEAD4_T1_BITS[(__t=(s)[i])>>4]&(1<<(c)) && \
                    ((c)=((c)<<6)|(__t&0x3f), ++(i)!=(length)) && \
                    (__t=(s)[i]-0x80)<=0x3f) && \
                /* valid second-to-last trail byte */ \
                ((c)=((c)<<6)|__t, ++(i)!=(length)) \
            :  /* U+0080..U+07FF */ \
                (c)>=0xc2 && ((c)&=0x1f, 1)) && \
            /* last trail byte */ \
            (__t=(s)[i]-0x80)<=0x3f && \
            ((c)=((c)<<6)|__t, ++(i), 1)) { \
        } else { \
            (c)=(sub);  /* ill-formed*/ \
        } \
    } \
}


#endif

test:

#include "icu_utf8_subset.h"
#include <stdio.h>
#include <string.h>
#include <stdint.h>

int main()
{
    char* s = "abc 123 →πó 🙀🙏";
    size_t n = strlen(s);
    size_t i = 0;
    size_t m = 0;
    uint32_t c;

    while (i < n) {
        U8_NEXT_OR_FFFD(s, i, n, c);
        // c is the current char (utf-32)
        // i is the current index (utf-8)
        m++;
    }

    printf("number of code points: %d\n", (int)m);
    return 0;
}

[koheiw]

Thank you very much. My version is https://github.com/quanteda/quanteda/blob/dev-cpp_sub/src/utf8.h but icu_utf8_subset.h seems more robust against ill formatted strings.