Consider upgrading to simdjson 0.4.0

[lemire]

Highlights

• Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.

New features:

• We added a fast (8GB/s) minifier that works directly on JSON strings.
• We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
• The array and object elements have a constant-time size() method.

Performance:

• Performance improvements to the API (type(), get<>()).
• The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
• We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).

System support:

• C++11 support for older compilers and systems.
• FreeBSD support (and tests).
• We support the clang front-end compiler (clangcl) under Visual Studio.
• It is now possible to target ARM platforms under Visual Studio.
• The simdjson library will never abort or print to standard output/error.

[eddelbuettel]

I should be able to get to that tomorrow. Really appreciate the heads-up and the attention to the concerns imposed upon us:

The simdjson library will never abort or print to standard output/error.

[eddelbuettel]

Does the C++11 support mean we can relax the C++17 requirement ?

[lemire]

Does the C++11 support mean we can relax the C++17 requirement ?

Absolutely. There are some constructs that you cannot use if you lack C++17, but I actually recommend not bothering with C++17 unless you happen to want to use C++ for other reasons.

[eddelbuettel]

I just gave that one really quick check and it failed on me missing some includes once I hardwired C++11. I can follow-up with more detail but likely not now...

[lemire]

@eddelbuettel

We have rather extensive C++11 tests in CI and elsewhere... I have been regularly compiling with C++11...

$ c++ -std=c++11 -o amalgamate_demo amalgamate_demo.cpp -Wall -Wextra
lemire@f3bcc52c783e:/Users/lemire/CVS/github/simdjson/singleheader$ ./amalgamate_demo
Please specify at least one file name.

If you get some failure, we'd be very interested in knowing about it...

We (well, mostly @jkeiser) have worked really hard to bring C++11 support. This was no joke. Lots of seriously hard work.

I expect that I'll have to issue 0.4.1, I always do... but, fundamentally, C++11 should work.

[eddelbuettel]

If you want to play along:

1. check our rcppsimdjson
2. overwrite / repace from your singleheader/simdjson.{h,cpp} into our inst/include/simdjson.{h,cpp}
3. remove or rename our (shell script) configure (to not test C++ std)
4. copy src/Makevars.in to src/Makevars and edit line 3 to be CXX_STD = CXX11

Then I got

edd@rob:~/git/rcppsimdjson(feature/simdjson_0.4.0)$ R CMD INSTALL .                                                                                                                                                
* installing to library ‘/usr/local/lib/R/site-library’                                                                                                                                                            
* installing *source* package ‘RcppSimdJson’ ...                                                                                                                                                                   
** using staged installation                                                                                                                                                                                       
** libs                                                                                                                                                                                                            
ccache g++  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG  -I'/usr/local/lib/R/site-library/Rcpp/include'   -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp -fpic  -g -O3 -Wall -pipe -pedantic  -c RcppExp
orts.cpp -o RcppExports.o                                                                                                                                                                                          
ccache g++  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG  -I'/usr/local/lib/R/site-library/Rcpp/include'   -DSIMDJSON_NO_COMPUTED_GOTO -I../inst/include -fopenmp -fpic  -g -O3 -Wall -pipe -pedantic  -c deseria
lize.cpp -o deserialize.o                                                                                                                                                                                          
In file included from ../inst/include/RcppSimdJson/deserialize/simplify.hpp:5,                                                                                                                                     
                 from ../inst/include/RcppSimdJson/deserialize.hpp:5,                                                                                                                                              
                 from ../inst/include/RcppSimdJson.hpp:4,                                                                                                                                                          
                 from deserialize.cpp:1:                                                                                                                                                                           
../inst/include/RcppSimdJson/deserialize/../common.hpp: In function ‘constexpr R_xlen_t rcppsimdjson::r_length(const _Container&)’:                                                                                
../inst/include/RcppSimdJson/deserialize/../common.hpp:16:37: error: ‘size’ is not a member of ‘std’                                                                                                               
   16 |   return static_cast<R_xlen_t>(std::size(__cont));                                                                                                                                                         
      |                                     ^~~~                                                                                                                                                                   
../inst/include/RcppSimdJson/deserialize/../common.hpp: At global scope:                                                                                                                                           
../inst/include/RcppSimdJson/deserialize/../common.hpp:23:8: warning: inline variables are only available with ‘-std=c++17’ or ‘-std=gnu++17’                                                                      
   23 | static inline constexpr int64_t NA_INTEGER64 = LLONG_MIN;                                                                                                                                                  
      |        ^~~~~~        

... plus pages more of course ...

I am not pointing fingers! I fully believe that the ducks are all aligned at your end. We're simply ... swimming in an adjacent pond here and I don't have time now to dig further.

[knapply]

The rcppsimdjson code has multiple instances of c++ 17 only stuff. Structured bindings can be replaced easily and std::optional wouldn’t be much harder, but I don’t think that if constexpr would be trivial to rip out.

[eddelbuettel]

And in a way it's fun to have structured bindings and all the newfangled stuff....

[eddelbuettel]

The thing that may matter more, as always, is getting RcppSimdJson to the masses of Windows users. Which in the past failed reliably but I also cannot remember why ...

[eddelbuettel]

Also, of course, entirely fair point that the C++17 dependency may as this point be self-imposed (based on older example code).

[knapply]

@eddelbuettel At the end of the day it’s your call, but I’m not a fan of moving backward here.

The nicer code aside, I don’t think it would actually expand the number of users.

A bigger priority should be getting this to pass on CRAN for Windows, which would expand the potential user base WAY more. (Okay, second biggest... after getting the exported functions worked out and on CRAN 😬).

I also suspect that Windows would still require Rtools4.0. GCC 4.9.3 isn’t really c++11, but rcppsimdjson works on GCC 7+ and clang 6+ right now.

All of that said... the code is sooo much nicer.

[eddelbuettel]

Same :)

But note that in Windows, since R 4.0.0, we now have a new Rtools with g++-8. I need to double check what blows up--likely some other posix-ness that may be hard(er) to fill in.

[knapply]

I’ll boot up the windows side of the computer and modify a one-off DESCRIPTION tomorrow to at least confirm we’re good building the 64bit version.

@lemire, assuming 32 bit is still on the “hidden menu” as you mentioned before #14 (comment) , I may be able to sort out what needs to be tweaked.

Worth a shot 🤞

[eddelbuettel]

Well generally we have win-builder for that and a trusted three-line shell script that uploads to win-builder and so I just did (by default to r-release and r-devel) and r-devel just came back with an aborted run, which is ran.

Link per email, valid "roughly 72 hours" as they say: https://win-builder.r-project.org/0GzKWVYr3IFG/ (but boring, seems to have stopped mid-run; could be them too)

Edit Same for r-release. Hm. https://win-builder.r-project.org/wh26UqAAG40R/