38 lines
34 KiB
Plaintext
38 lines
34 KiB
Plaintext
|
rust
|
|||
|
|
’„ #rustc 1.90.0 (1159e78c4 2025-09-14)Á˜ø½E—y88àìOg[ -a0f3a77406fcd134Á ÒµÅL ®Ùp,ØnWš÷*�Ó -eed239b623db52f0Á‡ ˜ is_xid_startÁ is_xid_continueÁ tablesÁ ” FÁ Align8Á
|
|||
|
|
” ò Align64Á
|
|||
|
|
” ò ASCII_STARTÁ ASCII_CONTINUEÁ CHUNKÁ
|
|||
|
|
TRIE_STARTÁ
|
|||
|
|
TRIE_CONTINUEÁ LEAFÁ $ò
|
|||
|
|
‰
|
|||
|
|
$ë " trueÁ ,ˆ
|
|||
|
|
´ø $� " falseÁ Á
|
|||
|
|
€ % ß 0 7 128Á ™
|
|||
|
|
€ ß % ß 0 7 128Á à
|
|||
|
|
|
|||
|
|
@ ßüÀ
|
|||
|
|
# ß ,Ø
|
|||
|
|
$ 64Á �
|
|||
|
|
› ß % ß 0 7 411Á $Ü
|
|||
|
|
ß % ß 0 7 1793Á $âw
|
|||
|
|
ß ! % ß 0 7 7968Á Ü¥ ” ' û ò ä-#3 Ž| _ f m { ‘ � Í ¨ ¯ ¶ äæ $ š ò ·� çña�@ \ c j x Ž š Ž ¥ ¬ ³
|
|||
|
|
˜ Ðt¸\ º\È\ é,Õ\ ¢$Ü\ Šlâ\ õTñ\ ü ±c ü ¦ £ [![github]](https://github.com/dtolnay/unicode-ident) [![crates-io]](https://crates.io/crates/unicode-ident) [![docs-rs]](https://docs.rs/unicode-ident)Á §© ü«j g [github]: https://img.shields.io/badge/github-8da0cb?style=for-the-badge&labelColor=555555&logo=githubÁ ü–n k [crates-io]: https://img.shields.io/badge/crates.io-fc8d62?style=for-the-badge&labelColor=555555&logo=rustÁ ü…m j [docs-rs]: https://img.shields.io/badge/docs.rs-66c2a5?style=for-the-badge&labelColor=555555&logo=docs.rsÁ ó© D÷ <br>Á €© ü„N K Implementation of [Unicode Standard Annex #31][tr31] for determining whichÁ üÓ@ = `char` values are valid in programming language identifiers.Á ”© ü˜1 . [tr31]: https://www.unicode.org/reports/tr31/Á Ê© üÎN K This crate is a better optimized implementation of the older `unicode-xid`Á ü�L I crate. This crate uses less static storage, and is able to classify bothÁ üêL I ASCII and non-ASCII codepoints with better performance, 2–6×Á ô· faster than `unicode-xid`.Á Ö© DÚã ã© üç ## Comparison of performanceÁ ˆ© üŒJ G The following table shows a comparison between five Unicode identifierÁ ¤× implementations.Á ì© üð$ ! - `unicode-ident` is this crate;Á ü• I F - [`unicode-xid`] is a widely used crate run by the "unicode-rs" org;Á üß C @ - `ucd-trie` and `fst` are two data structures supported by theÁ ä£
|
|||
|
|
[`ucd-generate`] tool;Á üÀ
|
|||
|
|
= : - [`roaring`] is a Rust implementation of Roaring bitmap.Á þ
|
|||
|
|
© ü‚ P M The *static storage* column shows the total size of `static` tables that theÁ üÓ = : crate bakes into your binary, measured in 1000s of bytes.Á ‘ © ü• J G The remaining columns show the **cost per call** to evaluate whether aÁ üà G D single `char` has the XID\_Start or XID\_Continue Unicode property,Á ü¨
|
|||
|
|
M J comparing across different ratios of ASCII to non-ASCII codepoints in theÁ |ö
|
|||
|
|
input data.Á †© üŠ> ; [`unicode-xid`]: https://github.com/unicode-rs/unicode-xidÁ üÉ@ = [`ucd-generate`]: https://github.com/BurntSushi/ucd-generateÁ üŠ< 9 [`roaring`]: https://github.com/RoaringBitmap/roaring-rsÁ Ç© üËA > | | static storage | 0% nonascii | 1% | 10% | 100% nonascii |Á ì� |---|---|---|---|---|---|Á ü«L I | **`unicode-ident`** | 10.5 K | 1.03 ns | 1.02 ns | 1.11 ns | 1.66 ns |Á üøJ G | **`unicode-xid`** | 12.0 K | 2.57 ns | 2.74 ns | 3.20 ns | 9.35 ns |Á üÃG D | **`ucd-trie`** | 10.4 K | 1.27 ns | 1.27 ns | 1.41 ns | 2.53 ns |Á ü‹A > | **`fst`** | 144 K | 49.3 ns | 49.1 ns | 47.1 ns | 27.9 ns |Á üÍF C | **`roaring`** | 66.1 K | 4.10 ns | 4.05 ns | 4.02 ns | 5.12 ns |Á ”© ü˜N K Source code for the benchmark is provided in the *bench* directory of thisÁ üç: 7 repo and may be repeated by running `cargo criterion`.Á ¢© D¦ã ¯© ü³$ ! ## Comparison of data structuresÁ Ø© ¤Ü #### unicode-xidÁ ñ© üõO L They use a sorted array of character ranges, and do a binary search to lookÁ üÅB ? up whether a given character lands inside one of those ranges.Á ˆ© \Œ ```rustÁ ü˜" # const _: &str = stringify! {Á ü»6 3 static XID_Continue_table: [(char, char); 763] = [Á üò% " ('\u{30}', '\u{39}'), // 0-9Á ü˜% " ('\u{41}', '\u{5a}'), // A-ZÁ <¾ # "Á \Æ â€¦Á <Òõ' üÚ# ('\u{e0100}', '\u{e01ef}'),Á 4þ ];Á D… # };Á <Ž ```Á –© üšL I The static storage used by this data structure scales with the number ofÁ üçL I contiguous ranges of identifier codepoints in Unicode. Every table entryÁ ü´L I consumes 8 bytes, because it consists of a pair of 32-bit `char` values.Á �© ü…I F In some ranges of the Unicode codepoint space, this is quite a sparseÁ üÏK H representation – there are some ranges where tens of thousands ofÁ ü›M J adjacent codepoints are all valid identifier characters. In other places,Á üéL I the representation is quite inefficient. A characater like `µ` (U+00B5)Á ü¶ L I which is surrounded by non-identifier codepoints consumes 64 bits in theÁ üƒ : 7 table, while it would be just 1 bit in a dense bitmap.Á ¾ © ü N K On a system with 64-byte cache lines, binary searching the table touches 7Á ü‘ F C cache lines on average. Each cache line fits only 8 table entries.Á üØ N K Additionally, the branching performed during the binary search is probablyÁ ü§ 1 . mostly unpredictable to the branch predictor.Á Ù © üÝ M J Overall, the crate ends up being about 6× slower on non-ASCII inputÁ ü« " compared to the fastest crate.Á Î © üÒ N K A potential improvement would be to pack the table entries more compactly.Á ü¡!O L Rust's `char` type is a 21-bit integer padded to 32 bits, which means everyÁ üñ!L I table entry is holding 22 bits of wasted space, adding up to 3.9 K. TheyÁ ü¾"M J could instead fit every table entry into 6 bytes, leaving out some of theÁ üŒ#P M padding, for a 25% improvement in space used. With some cleverness it may beÁ üÝ#K H possible to fit in 5 bytes or even 4 bytes by storing a low char and anÁ ü©$N K extent, instead of low char and high char. I don't expect that performanceÁ üø$P M would improve much but this could be the most efficient for space across allÁ üÉ%3 0 the libraries, needing only about 7 K to store.Á ý%© Œ�& #### ucd-trieÁ “&© ü—&K H Their data structure is a compr
|
|||
|
|
˜§ з ºÌ éá ¢ó Š õ §§§ü¨\T··ÌÌááóó ü¾]% üÿ\> ; Whether the character has the Unicode property XID\_Start.ÁdÅ]
|
|||
|
|
|
|||
|
|
chÁÒ]üÕ`( ü“`A > Whether the character has the Unicode property XID\_Continue.Á|Ü`
|
|||
|
|
Ùxì`T—\–<ˆ\�$‘\lˆ\ „†\ 4›\
|
|||
|
|
lâ è lø þ 4· ” ’Þ ¾ Í ä-#3 Ž| Í R Í
|
|||
|
|
^` dÁ â <ø ” Û{ € Ž ·� çña�@ Ž âI Ž
|
|||
|
|
^` dƒ ü“3 \¥ š ò ·� çña�@ ß € € €J ßüè6 tú °~€ €— ßìÀ
|
|||
|
|
,Ñ
|
|||
|
|
ßüå
|
|||
|
|
/ T÷
|
|||
|
|
û ò ä-#3 Ž| ß › #'-148<@E I M QTX\ `dfjnr(vx|€„ˆŒ�”˜œ +¤ ¦¨2 ® ²¶º¾ ÀC ŠÑÓ ÉÙÝáåé í ïñóRõ ¯ ©÷ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ 5 ßü¯ 3 lÁ û ò ä-#3 Ž| ß ˆ
|
|||
|
|
!%*/16:>BG K O QVZ^ bdhlpt(vz~‚†ŠŽ’–šž¢+¤ ¦«2 ° ´¸¼¾ ÀCÂ È ËÃÆÎÑÕ × ÉÛßãçë í Ì ïñóRõ ¯ ©÷ Ï ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿˆ‚ ßü½w+ $Ïw š ò ·� çña�@ß À> ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ??ÿÿÿÿ??ÿªÿÿÿ?ÿÿÿÿÿÿß_Ü Ïÿ Ü ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿøÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÃÿ P ߸@×ÿÿûÿÿÿÿÿÿÿÿÿ¿ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÃÿ P ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ߸À×ÿÿûÿÿÿÿÿÿÿÿÿ¿ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿüÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿþÿÿÿÿÿÿÿÿ ÿÿÿ‡ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿûüÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿþÿÿÿÿÿÿÿÿþÿÿÿÿ¿¶ ÿÿÿ‡ ÿÿÿÿÿ Àþÿÿÿÿÿÿÿÿÿÿÿ/ `À œ ýÿÿÿ àÿÿÿÿÿÿÿÿÿÿ? üÿÿÿ0 ÿÿÿÿÿÿÿÿÿÿÃÿÿÿÿÿÿÿÿÿÿÿÿïŸÿýÿŸ ÿÿÿÿÿÿÿçÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ?$ÿÿ? ÿÿÿÿÿÿÿþ ÿÿÿÿÿ ðÿÿÿÿÿÿ# ÿ þÿáŸùÿÿýÅ# @ ° ÿÿÿÿÿ? ÿÿÿÿÿÿÿþ€ÿÿÿÿÿÿÿÿÿûÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÏÿþÿïŸùÿÿýÅóŸy€°ÏÿPà‡ùÿÿým ^ à¿ûÿÿýí# àŸùÿÿýí# ° èÇ=ÖÇÿ î‡ùÿÿýmÓ‡9^Àÿ? î¿ûÿÿýíó¿; Ïÿ þîŸùÿÿýíóŸ9à°Ïÿ ìÇ=ÖÇÿÃÇ=� Àÿ àßýÿÿýÿ# 7 áßýÿÿýï# p ðßýÿÿÿÿ' @p€ üàÿüÿÿû/ ÿßýÿÿýÿóß=`7Ïÿ ïßýÿÿýïóß=`pÏÿ ÿßýÿÿÿÿÿß}ð€Ïÿ üîÿüÿÿû/„_ÿÀÿ þÿÿÿÿÿ Ö÷ÿÿ¯ÿ _ ð ÿþÿÿÿ þÿÿÿÿÿÿÿÿ Ö÷ÿÿ¯ÿÿ?_ÿó ÿ Âÿþÿÿÿ þÿßÿÿþÿÿÿ @ ÿÿÿÿÿ € ?<bÀáÿ@ ÿÿÿÿ¿
|
|||
|
|
ÿÿûÿÿÿÿÀ ½ÿ¿ÿÿÿÿÿÿÿÿÿïŸùÿÿýíûŸ9�àÏ ÿKÿÿÿÿ¿ÿ¥÷ ÿÿÿÿÿÿ € € ÿÿÿÿÿÿ ° ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿà ÿÿÿÿÿÿÿÿ¿ ÿ ÿÿÿÿÿÿ?ÿ ? ÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿ ÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿçÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿ €òoÿÿÿ € ÿüÿÿÿÿ
|
|||
|
|
ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ €òoÿÿÿ¿ù ÿ ÿüÿÿÿÿÿü øÿÿÿÿ ðÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿ€ ÿÿÿÿÿÿÿÿÿ# ÿÿÿÿÿÿÿÿÿ ÿ ÿÿÿÿ ÿÿýÿÿÿ üÿÿÿ ûÿÿÿÿ @ ¿ýÿÿÿ ÿÿÿÿÿ ÿýÿÿÿÿÿ ÿ üÿÿÿüÿÿþ ûÿÿÿÿ´ÿ ÿ¿ýÿÿÿûÿÿÿÿÿÿÿ ÿÿ ôÿýÿÿÿ ÿÿ ÿÿýÿÿÿÿÇ ÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ~ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ? ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ? ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿ? ÿÿÿÿÿÿ øÿÿàÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ? ÿÿÿÿÿÿ ÿøÿÿàÿÿ ÿÿÿÿÿ ÿÿÿÿÿ ÿ ÿÿÿÿÿÿÿÿ ÿÿÿùÿÿ ÿÿÿÿÿÿÿÿÿ øÿ | ÿÿÿÿÿÿÿÿ ÿÿÿùÿÿ ÿÿÿÿÿÿÿÿÿ‡ÿÿÿÿÿÿÿ€ÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ? €ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ïoÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ?ÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ àãøç < ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿc ÿÿÿà ÿ ÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿßÿÿÿÿÿÿÿÿßdÞÿëïÿÿÿÿÿÿÿ¿çßßÿÿÿ{_üýÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ?ÿÿÿýÿÿ÷ÿÿÿ÷ÿÿßÿÿÿßÿÿÿÿÿÿÿÿýÿÿÿýÿÿ÷ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ?ÿÿÿýÿÿ÷ÿÿÿ÷ÿÿßÿÿÿßÿÿÿÿÿÿÿÿýÿÿÿýÿÿ÷Ïÿÿÿÿÿÿÿÿÿÿÿÿøÿÿÿÿÿ øþÿ ÿÿÿÿÿÿÿ? ÿÿÿÿÿ €? @ ÿÿùÛÿÿÿÿÿÿÿ? € ÿÿÿÿÿ ÿ?ÿC ÿÿÿ? ÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿ? ÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿ·? À oÿ ÿÿÿÿÿ?À oÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ïÿÿÿ–þ÷
|