RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

Release CircleCI Docker Cloud Build Status Forum Discord Codecov Total alerts

RediSearch

Querying, secondary indexing, and full-text search for Redis

logo

Overview

RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch query language to query that data.

RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint.

RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features.

Getting started

If you're just getting started with RediSearch, check out the official RediSearch tutorial. Also, consider viewing our RediSearch video explainer.

The fastest way to get up and running with RediSearch is by using the RediSearch Docker image.

Trying RediSearch

To try RediSearch, either use the RediSearch Docker image, or create a free Redis Cloud Essentials account to get a RediSearch instance in the cloud.

Docker image

The RediSearch Docker image makes it easy to try RediSearch.

To create a local RediSearch container, run:

$ docker run -p 6379:6379 redislabs/redisearch:latest

To connect to this instance, run:

$ redis-cli

Documentation

The RediSearch documentation provides a complete overview of RediSearch. Helpful sections include:

Mailing list and forum

Got questions? Join us in #redisearch on the Redis Discord server.

If you have a more detailed question, drop us a line on the RediSearch Discussion Forum.

Client libraries

You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API.

Language Library Author License Stars
Python redisearch-py Redis Labs BSD redisearch-py-stars
Java (Jedis client library) JRediSearch Redis Labs BSD JRediSearch-stars
Java (Lettuce client library) lettusearch Redis Labs Apache-2.0 lettusearch-stars
Java spring-redisearch Redis Labs Apache-2.0 spring-redisearch-stars
Java redis-modules-java dengliming Apache-2.0 redis-modules-java-stars
Go redisearch-go Redis Labs BSD redisearch-go-stars
JavaScript RedRediSearch Kyle J. Davis MIT RedRediSearch-stars
JavaScript redis-redisearch Kyle J. Davis MIT redis-redisearch-stars
TypeScript redis-modules-sdk Dani Tseitlin BSD-3-Clause redis-modules-sdk-stars
C# NRediSearch Marc Gravell MIT NRediSearch-stars
PHP redisearch-php Ethan Hann MIT redisearch-php-stars
PHP php-redisearch MacFJA MIT php-redisearch-stars
Rust redisearch-api-rs Redis Labs BSD redisearch-api-rs-stars
Ruby on Rails redi_search_rails Dmitry Polyakovsky MIT redi_search_rails-stars
Ruby redisearch-rb Victor Ruiz MIT redisearch-rb-stars
Ruby redi_search Nick Pezza MIT redi_search-stars

Other available Libraries

Language Library Author License Stars Comments
Rust redisearch-api-rs Redis Labs BSD redisearch-api-rs-stars API for Redis Modules written in Rust

RediSearch features

  • Full-Text indexing of multiple fields in Redis hashes
  • Incremental indexing without performance loss
  • Document ranking (using tf-idf, with optional user-provided weights)
  • Field weighting
  • Complex boolean queries with AND, OR, and NOT operators
  • Prefix matching, fuzzy matching, and exact-phrase queries
  • Support for double-metaphone phonetic matching
  • Auto-complete suggestions (with fuzzy prefix suggestions)
  • Stemming-based query expansion in many languages (using Snowball)
  • Support for Chinese-language tokenization and querying (using Friso)
  • Numeric filters and ranges
  • Geospatial searches using Redis geospatial indexing
  • A powerful aggregations engine
  • Supports for all utf-8 encoded text
  • Retrieve full documents, selected fields, or only the document IDs
  • Sorting results (for example, by creation date)

Cluster support

RediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of Redis Enterprise Cloud and Redis Enterprise Software.

See RediSearch on Redis Enterprise for more information.

License

RediSearch is licensed under the Redis Source Available License Agreement.

Owner
A query and indexing engine for Redis, providing secondary indexing, full-text search, and aggregations.
null
Comments
  • Faulty replication; some changes slaved, some not; mem usage increasing but

    Faulty replication; some changes slaved, some not; mem usage increasing but "on hold"

    We had an outage on our live slave systems last week which seems to have been caused by RediSearch (we use a pre-release of RediSearch against Redis 4.0.2, and have protected ourselves enough to minimize enduser impact, and we were able to contain the situation).

    Our live system uses RediSearch branch aof_fixes on master as well as slaves.

    So not the latest version of RediSearch, and we fully understand we have to upgrade, but we will have to wait a few days until we can update the master (planning, free timeslot, dependencies).

    For now, all we are interested in is: "Is it probable that this is fixed in RC 0.99.0?"

    What I did just now, is create a new redis slave on a test system, with RediSearch 0.99.0, with slaveof to a live master Redis instance, which still uses version aof_fixes.

    The result is similar to what we saw in live slaves a few days back. Here are the symptoms:

    • A simple set hello world does replicate (Note: we did not test this at crisis time on live)
    • A complex update from a Lua script (but not hitting any RediSearch commands) does not replicate
    • The master is 100% okay (redis contents are daily diffed to origin), also RediSearch data seems accurate (but not daily diffed yet)
    • We disabled snapshotting, and rdb_changes_since_last_save only rises when doing set hello world examples
    • The used_memory_dataset does reflect all changes coming in (so data is recieved from the master, and is put somewhere in memory, but the Lua initiated changes never make it to the keyspace as far as we can tell)
    • We consistently get master_link_status:up, master_last_io_seconds_ago:0 and master_sync_in_progress:0
    • The logs do not show anything out of the ordinary
    • On the day of the live issue, there was a partial sync at around 02:00AM, which went fine. We scanned the AOF of the master (not rewritten), and the first command of that day (!!!!!) that was able to trigger a RediSearch index write (from a Lua script, SAFEADD) was at 11:43AM. Our two slaves stopped replicating at 11:43AM, both at the same time. After that, memory usage kept climbing on both of the slaves, but the data was stale, and rdb_changes_since_last_save stopped rising (and snapshotting is disabled in the config, so no rdb saves are ever done, except for service stop or client request).

    Here are some Graphs of the live issue. From: 2017-11-16 11:00:00 To: 2017-11-16 12:00:00

    Master (shows derivative of rdb_changes_since_last_save): image

    Slave 1 (shows derivative of rdb_changes_since_last_save, notice the flatliner after 11:43): image

    Slave 2 (shows derivative of rdb_changes_since_last_save, notice the flatliner after 11:43): image

    Graphs below are of Slave 1, notice the mem usage after 11:43: image image

    At this point, could you provide hints on more data to gather, which could be useful at a later stage (after upgrading RediSearch for example)? I still have the 0.99.0 slave running on a test system with the faulty behaviour, and I'll try to keep it running today.

    Hope you can help, cheers, TW

  • RediSearch crash with BGSAVE

    RediSearch crash with BGSAVE

    RediSearch 1.0.5 , Redis 4.0.7 (also happens on RediSearch 1.0.4, Redis 4.0.6, but upgraded and repro'd to avoid any fuzz over a fixed issue)

    After issuing a BGSAVE:

    25396:C 24 Jan 14:43:34.387 # Redis 4.0.7 crashed by signal: 11
    25396:C 24 Jan 14:43:34.387 # Crashed running the instruction at: 0x7f3eea498085
    25396:C 24 Jan 14:43:34.387 # Accessing address: (nil)
    25396:C 24 Jan 14:43:34.387 # Failed assertion: <no assertion failed> (<no file>:0)
    
    ------ STACK TRACE ------
    EIP:
    /usr/local/bin/RediSearch/1.0.5/redisearch.so(SortingVector_RdbSave+0x15)[0x7f3eea498085]
    
    Backtrace:
    redis-rdb-bgsave *:14213(logStackTrace+0x3c)[0x46b81c]
    redis-rdb-bgsave *:14213(sigsegvHandler+0xa3)[0x46ca73]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f3eecd6f8d0]
    /usr/local/bin/RediSearch/1.0.5/redisearch.so(SortingVector_RdbSave+0x15)[0x7f3eea498085]
    /usr/local/bin/RediSearch/1.0.5/redisearch.so(DocTable_RdbSave+0x1ec)[0x7f3eea48d3ac]
    /usr/local/bin/RediSearch/1.0.5/redisearch.so(IndexSpec_RdbSave+0x9b)[0x7f3eea481e1b]
    redis-rdb-bgsave *:14213(rdbSaveObject+0x408)[0x44d0e8]
    redis-rdb-bgsave *:14213(rdbSaveKeyValuePair+0x9c)[0x44d2ec]
    redis-rdb-bgsave *:14213(rdbSaveRio+0x2b6)[0x44d5b6]
    redis-rdb-bgsave *:14213(rdbSave+0x95)[0x44d785]
    redis-rdb-bgsave *:14213(rdbSaveBackground+0xa0)[0x44da50]
    redis-rdb-bgsave *:14213(bgsaveCommand+0xdf)[0x44dc9f]
    redis-rdb-bgsave *:14213(call+0x87)[0x42dc97]
    redis-rdb-bgsave *:14213(processCommand+0x36d)[0x42e30d]
    redis-rdb-bgsave *:14213(processInputBuffer+0x10d)[0x43e5bd]
    redis-rdb-bgsave *:14213(aeProcessEvents+0x15d)[0x42945d]
    redis-rdb-bgsave *:14213(aeMain+0x2b)[0x42977b]
    redis-rdb-bgsave *:14213(main+0x512)[0x4323d2]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f3eec9d9b45]
    --snip--
    ------ CURRENT CLIENT INFO ------
    id=142179 addr=127.0.0.1:51334 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=bgsave
    argv[0]: 'bgsave'
    6428:M 24 Jan 14:43:34.553 # Background saving terminated by signal 11
    

    The BGSAVE failed, but the instance keeps running. I can do a BGREWRITEAOF, and use it to restore to another redis instance. A BGSAVE on that redis instance crashes as well.

    Sidenote: AOF is around 5GB and NDA.

  • Crash while inserting

    Crash while inserting

    While testing using node/redisearchclient by inserting wikipedia abstracts, redis server crashed.

    31654:C 03 Sep 2019 12:45:09.987 * RDB: 389 MB of memory used by copy-on-write
    27981:M 03 Sep 2019 12:45:10.092 * Background saving terminated with success
    warning: got a bad length when writing to pipe.
    27981:M 03 Sep 2019 12:45:35.019 # Out Of Memory allocating 13614287927004626944 bytes!
    
    
    === REDIS BUG REPORT START: Cut & paste starting from here ===
    27981:M 03 Sep 2019 12:45:35.019 # ------------------------------------------------
    27981:M 03 Sep 2019 12:45:35.019 # !!! Software Failure. Press left mouse button to continue
    27981:M 03 Sep 2019 12:45:35.019 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.c:3893
    27981:M 03 Sep 2019 12:45:35.019 # (forcing SIGSEGV in order to print the stack trace)
    27981:M 03 Sep 2019 12:45:35.019 # ------------------------------------------------
    27981:M 03 Sep 2019 12:45:35.019 # Redis 5.0.5 crashed by signal: 11
    27981:M 03 Sep 2019 12:45:35.019 # Crashed running the instruction at: 0x46f427
    27981:M 03 Sep 2019 12:45:35.019 # Accessing address: 0xffffffffffffffff
    27981:M 03 Sep 2019 12:45:35.019 # Failed assertion: <no assertion failed> (<no file>:0)
    
    ------ STACK TRACE ------
    EIP:
    redis-server 127.0.0.1:6379(_serverPanic+0x117)[0x46f427]
    
    Backtrace:
    redis-server 127.0.0.1:6379(logStackTrace+0x29)[0x4710f9]
    redis-server 127.0.0.1:6379(sigsegvHandler+0xac)[0x47179c]
    /lib64/libpthread.so.0(+0xf100)[0x7f103d140100]
    redis-server 127.0.0.1:6379(_serverPanic+0x117)[0x46f427]
    redis-server 127.0.0.1:6379(redisOutOfMemoryHandler+0x2e)[0x42e6ee]
    redis-server 127.0.0.1:6379(zmalloc+0x49)[0x437a29]
    ./redisearch.so(+0xbfbea)[0x7f1034247bea]
    ./redisearch.so(+0xc02de)[0x7f10342482de]
    ./redisearch.so(+0xc11d6)[0x7f10342491d6]
    ./redisearch.so(+0xc1307)[0x7f1034249307]
    ./redisearch.so(+0xc2260)[0x7f103424a260]
    ./redisearch.so(FGC_parentHandleFromChild+0x84)[0x7f103424a4f3]
    ./redisearch.so(+0xc2844)[0x7f103424a844]
    ./redisearch.so(+0xc86db)[0x7f10342506db]
    ./redisearch.so(+0x1314ef)[0x7f10342b94ef]
    /lib64/libpthread.so.0(+0x7dc5)[0x7f103d138dc5]
    /lib64/libc.so.6(clone+0x6d)[0x7f103ce6621d]
    
    ------ INFO OUTPUT ------
    # Server
    redis_version:5.0.5
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:24fc166a0b660279 
    redis_mode:standalone
    os:Linux 3.10.0-957.10.1.el7.x86_64 x86_64
    arch_bits:64
    multiplexing_api:epoll
    atomicvar_api:atomic-builtin
    gcc_version:4.8.5
    process_id:27981
    run_id:83d1972f93e1bcf5920d6202b80a018bfc270627
    tcp_port:6379
    uptime_in_seconds:1178
    uptime_in_days:0
    hz:10
    configured_hz:10
    lru_clock:7258846
    executable:/usr3/dbtest/redis/redis-server
    config_file:/usr3/dbtest/redis/./redis.conf
    
    # Clients
    connected_clients:1
    client_recent_max_input_buffer:374618
    client_recent_max_output_buffer:0
    blocked_clients:1
    
    # Memory
    used_memory:2061747840
    used_memory_human:1.92G
    used_memory_rss:1768161280
    used_memory_rss_human:1.65G
    used_memory_peak:2061747840
    used_memory_peak_human:1.92G
    used_memory_peak_perc:148.30%   
    used_memory_overhead:136151160  
    used_memory_startup:791456
    used_memory_dataset:1925596680  
    used_memory_dataset_perc:93.43% 
    allocator_allocated:1394146056  
    allocator_active:1397882880
    allocator_resident:1435181056   
    total_system_memory:50465402880 
    total_system_memory_human:47.00G
    used_memory_lua:37888
    used_memory_lua_human:37.00K
    used_memory_scripts:0
    used_memory_scripts_human:0B
    number_of_cached_scripts:0
    maxmemory:38654705664
    maxmemory_human:36.00G
    maxmemory_policy:noeviction
    allocator_frag_ratio:1.00
    allocator_frag_bytes:3736824
    allocator_rss_ratio:1.03
    allocator_rss_bytes:37298176
    rss_overhead_ratio:1.23
    rss_overhead_bytes:332980224
    mem_fragmentation_ratio:1.27
    mem_fragmentation_bytes:377867880
    mem_not_counted_for_evict:0
    mem_replication_backlog:0
    mem_clients_slaves:0
    mem_clients_normal:387280
    mem_aof_buffer:0
    mem_allocator:jemalloc-5.1.0
    active_defrag_running:0
    lazyfree_pending_objects:0
    
    # Persistence
    loading:0
    rdb_changes_since_last_save:149373
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1567539910   
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:22
    rdb_current_bgsave_time_sec:-1  
    rdb_last_cow_size:407920640
    aof_enabled:0
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:-1
    aof_current_rewrite_time_sec:-1 
    aof_last_bgrewrite_status:ok
    aof_last_write_status:ok
    aof_last_cow_size:0
    
    # Stats
    total_connections_received:9
    total_commands_processed:2377791
    instantaneous_ops_per_sec:3271  
    total_net_input_bytes:292353044 
    total_net_output_bytes:5135789  
    instantaneous_input_kbps:966.14 
    instantaneous_output_kbps:15.97 
    rejected_connections:0
    sync_full:0
    sync_partial_ok:0
    sync_partial_err:0
    expired_keys:0
    expired_stale_perc:0.00
    expired_time_cap_reached_count:0
    evicted_keys:0
    keyspace_hits:0
    keyspace_misses:0
    pubsub_channels:0
    pubsub_patterns:0
    latest_fork_usec:35924
    migrate_cached_sockets:0
    slave_expires_tracked_keys:0
    active_defrag_hits:0
    active_defrag_misses:0
    active_defrag_key_hits:0
    active_defrag_key_misses:0
    
    # Replication
    role:master
    connected_slaves:0
    master_replid:1920d458e45ff7b3fb9d574c8bdc38049f174e2b
    master_replid2:0000000000000000000000000000000000000000
    master_repl_offset:0
    second_repl_offset:-1
    repl_backlog_active:0
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:0
    repl_backlog_histlen:0
    
    # CPU
    used_cpu_sys:130.684637
    used_cpu_user:342.585239
    used_cpu_sys_children:66.336626 
    used_cpu_user_children:126.113013
    
    # Commandstats
    cmdstat_FT.CREATE:calls=9,usec=53995,usec_per_call=5999.44
    cmdstat_FT.ADD:calls=1007939,usec=110731446,usec_per_call=109.86
    cmdstat_FT.DROP:calls=9,usec=13324347,usec_per_call=1480483.00
    cmdstat_del:calls=1369806,usec=2325602,usec_per_call=1.70
    cmdstat_info:calls=28,usec=3026,usec_per_call=108.07
    
    # Cluster
    cluster_enabled:0
    
    # Cluster
    cluster_enabled:0
    
    # Keyspace
    db0:keys=2535451,expires=0,avg_ttl=0
    
    ------ CLIENT LIST OUTPUT ------
    id=1369867 addr=127.0.0.1:52188 fd=8 name= age=326 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=271435 qbuf-free=98915 obl=5 oll=0 omem=0 events=r cmd=FT.ADD
    
    ------ CURRENT CLIENT INFO ------
    id=1369867 addr=127.0.0.1:52188 fd=8 name= age=326 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=271435 qbuf-free=98915 obl=5 oll=0 omem=0 events=r cmd=FT.ADD
    argv[0]: 'ft.add'
    argv[1]: 'wp'
    argv[2]: '1007921'
    argv[3]: '1'
    argv[4]: 'FIELDS'
    argv[5]: 'url'
    argv[6]: 'https://en.wikipedia.org/wiki/Never_a_Dull_Moment_(1968_film)'
    argv[7]: 'title'
    argv[8]: 'Wikipedia: Never a Dull Moment (1968 film)'
    argv[9]: 'abstract'
    argv[10]: '| runtime        = 99 minutes'
    
    ------ REGISTERS ------
    27981:M 03 Sep 2019 12:45:35.021 #
    RAX:0000000000000000 RBX:000000000052d518
    RCX:00007f103ce57a4d RDX:0000000000000000
    RDI:0000000000000000 RSI:00007f103d12ca10
    RBP:0000000000000f35 RSP:00007f0fc37f41d0
    R8 :00007f0fc37f6700 R9 :0000000000000053
    R10:0000000000000030 R11:0000000000000000
    R12:0000000000000000 R13:00007f0fc37f69c0
    R14:00007f0fc37f6700 R15:000000000000000f
    RIP:000000000046f427 EFL:0000000000010206
    CSGSFS:0000000000000033
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41df) -> 00000000004fbfcd
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41de) -> 0000000000000000
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41dd) -> 00007f0fc0000900
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41dc) -> 00007f0fc37f4340
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41db) -> 00007f0fc0004840
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41da) -> 0000000000000000
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d9) -> 00007f0fc37f4eb0
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d8) -> 00007f0fc37f4300
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d7) -> 59524f4d454d2046
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d6) -> 4f2054554f20726f
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d5) -> 6620676e6974726f
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d4) -> 6261207369646552
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d3) -> 00007f0fc37f42f0
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d2) -> 00007f0fc37f43c0
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d1) -> 0000003000000018
    27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d0) -> 00007f0fc37f4330
    
  • Support for Chinese tokenization

    Support for Chinese tokenization

    Following the recent refactor of the tokenizer API, it's now much easier to add additional tokenizers.

    Before we'll add it to the extnesion API, let's first add a Chinese one.

    A few notes on that:

    1. The candidates I'm seeing as valid options are:

      a. https://github.com/yanyiwu/cjieba
      b. https://github.com/lionsoul2014/friso

    Another option would be to take their dictionaries, and implement something based on our own triemap, with the advantage of scanning the dictionary letter by letter and not doing a full lookup for each term. That can be left to a later stage or if the libraries are not fast enough.

    1. Currently we do not support external data files, and all the algorithms rely on a very big dictionary (a few megabytes). There are two options here: include it as static data in the module and make it very big, or rely on an external file that will need to be distributed alongside redisearch (not a problem in the docker image, more so in Enterprise Cluster).

    If we're going with bundling it into the source, we might want to consider making the whole Chinese support a compile time option, as many people will not need it.

    Let's start by making this work, and then discuss the packaging options. Another would be a script that downloads the dictionary and saves it into Redis itself :) This might actually a good distribution option if we're going with a non bundled version.

    1. Please make sure that whatever we have works well with mixed Chinese / English documents.

    2. We will not do automatic language detection, and for now we will not add anything to the API. FT.ADD already supports the LANGUAGE modifier, we can use LANGUAGE chinese to trigger the special tokenizer. So when creating the add document context, based on the language you select the tokenizer.

    PS Make sure whatever we end up doing is thread safe as tokenization is done in the thread pool.

  • RediSearch - FT.SEARCH with Field Modifiers or INFIELDS option

    RediSearch - FT.SEARCH with Field Modifiers or INFIELDS option

    Trying to perform FT.SEARCH using Field Modifier and INFIELDS option but getting wrong results. Please see below. image I mentioned to search within Title but got result even if it contains in description. image Please suggest the right syntax if I am doing wrong. Thanks in advance.

  • Unused FT -> high CPU -> production issues, possibly GC related

    Unused FT -> high CPU -> production issues, possibly GC related

    Hi,

    We've fairly recently upscaled our usage of RediSearch, including some larger datasets than the last two years. This caused issues on unrelated Redis traffic. In my absence (holidays) our IT had already disabled populating AND using the larger FT index (this was possible, since the new functionality is still in beta), but it didn't solve the production issues. I just solved the issue by doing a simple FT.DROP . Here are some CPU graphs (larger FT dataset was introduced on April 1st 2019):

    image

    I'm showing the CPU usage of a Redis slave here (running on Azure, replicated from our LAN through a VPN tunnel). CPU usage doesn't seem like much, but one core is almost constantly at 85% CPU. It is directly related to the live issues, above a certain percentage. Live performance became unacceptable after/around April 23rd.

    Here's some stats, from just before I dropped the index (about 4M documents):

    index_name
    ~ftidx.lo.pklstat
    index_options
    
    fields
    cusno_public_shanos
    type
    TAG
    SEPARATOR
    ;
    SORTABLE
    pkkno
    type
    TAG
    SEPARATOR
    ;
    SORTABLE
    artno
    type
    TAG
    SEPARATOR
    ;
    SORTABLE
    unmcd
    type
    TAG
    SEPARATOR
    ;
    SORTABLE
    pklnobox
    type
    TAG
    SEPARATOR
    ;
    SORTABLE
    iShipped
    type
    NUMERIC
    SORTABLE
    iReadyToShip
    type
    NUMERIC
    SORTABLE
    iOnShelve
    type
    NUMERIC
    SORTABLE
    iIDLiPklStat
    type
    NUMERIC
    SORTABLE
    NOINDEX
    dtShipped
    type
    NUMERIC
    SORTABLE
    fMechanicalsVV
    type
    NUMERIC
    NOINDEX
    num_docs
    3949195
    max_doc_id
    4254756
    num_terms
    0
    num_records
    1.8446744073707577e+19
    inverted_sz_mb
    17592186044410.168
    offset_vectors_sz_mb
    0
    doc_table_size_mb
    511.26409149169922
    sortable_values_size_mb
    1079.3353042602539
    key_table_size_mb
    97.942524909973145
    records_per_doc_avg
    4671013731585.6904
    bytes_per_record_avg
    1
    offsets_per_term_avg
    0
    offset_bits_per_record_avg
    -nan
    gc_stats
    current_hz
    1
    bytes_collected
    6114654
    effectiv_cycles_rate
    0.018745357995123502
    cursor_stats
    global_idle
    0
    global_total
    0
    index_capacity
    128
    index_total
    0 
    

    My uneducated guess would be: The GC. We also noticed a very high number:

    num_records
    1.8446744073707577e+19
    

    But we don't know exactly what the num_records metric means - we only know that the index contains (almost) 4M rows, like shown in num_docs .

    What do you guys think? Can I tune this somehow?

    I'll be happy to give you an rdb, but it's a couple of G's. Just let me know.

  • Add info and info_len field for suggestion trie node

    Add info and info_len field for suggestion trie node

    The original trie node can only have 'str' field for matching. Some application needs more fields to save attached information for the terminal trie node. So we supply the info_len and dynamic info fields.

  • nil values returned for non-existent fields when sorting

    nil values returned for non-existent fields when sorting

    RS returns a nil value when I am sorting and the document doesn't have that field. I am investigating further to understand if that happens in all cases or there are special circumstances that lead to this.

    The document p:734802 does NOT have a field "price". Here I dump the entire document, add it to search, and then search for it with sorting.

    127.0.0.1:6379> HGETALL p:734802
     1) "imgurlsmall"
     2) "https://storage.googleapis.com/lwimg/mh/l/20190615/5bc8cded36741bf5e2bce5fad41fe6d4.png"
     3) "feed"
     4) "f:themotorcafenew"
     5) "_t"
     6) "1560589933"
     7) "title"
     8) "2019 MULTISTRADA 1260S TOURING - Ducati"
     9) "tags"
    10) "vm:77,w:couchride:t:europe,vm:146,w:couchride:t:adventure423,vm:162,vm:191,w:couchride:t:dealer,w:couchride:t:new,t:image,w:couchride:t:hasvin,w:couchride:t:touring,w:couchride:t:ducati,w:couchride:t:multistrada,w:couchride:t:multistrada1260,w:couchride:t:multistrada1260s,w:couchride,f:themotorcafenew,cf:d:729,cf:m:ari,CA"
    11) "location"
    12) "Sunnyvale, CA"
    13) "geo"
    14) "-122.0363496 37.36883"
    15) "created"
    16) "2019-06-15 09:12:13"
    17) "ctitle"
    18) "2019 MULTISTRADA 1260S TOURING Ducati"
    19) "mpid"
    20) "cf:m:ari"
    21) "ct"
    22) "l"
    23) "year"
    24) "2019"
    25) "imgurl"
    26) "https://cdnmedia.endeavorsuite.com/images/ThumbGenerator/Thumb.aspx?img=https%3a%2f%2fcdnmedia.endeavorsuite.com%2fimages%2fassets%2fimg%2fno-image-if.png&mw=350&mh=200&f=1&fz=1"
    27) "vin"
    28) "zdmaaekw1kb007485"
    29) "psd"
    30) "ari"
    31) "is"
    32) "\n\x0f\[email protected]\x93%\x12\x17[\xda#\xdf\x97F\a\x05"
    33) "mileage"
    34) "000003"
    35) "ssc"
    36) "2272"
    37) "st"
    38) "CA"
    39) "did"
    40) "cf:d:729"
    41) "ss"
    42) "88329"
    43) "url"
    44) "https://www.themotorcafe.com/inventory/2019-ducati-multistrada-1260s-touring-sunnyvale-ca-94087-2643036i"
    45) "imgurllarge"
    46) "https://storage.googleapis.com/lwimg/mh/l/20190615/105ec8b4ff94206ae43fd3fd9619aa8f.png"
    47) "ssf"
    48) "9399"
    
    
    127.0.0.1:6379> FT.ADDHASH il1 p:734802 1.0 REPLACE
    OK
    
    127.0.0.1:6379> "FT.SEARCH" "il1" "@ctitle:multistrada* 1260s* @tags:{w\\:couchride\\:t\\:adventure423} @year:[2019 inf] @geo:[-122.0363496 37.36883 10 mi]" "LIMIT" "0" "20" "SORTBY" "price" "DESC"
    1) (integer) 1
    2) "p:734802"
    3)  1) price                           \<----------- HERE
        2) (nil)                                \<----------- HERE
        3) imgurlsmall
        4) "https://storage.googleapis.com/lwimg/mh/l/20190615/5bc8cded36741bf5e2bce5fad41fe6d4.png"
        5) feed
        ... remaining fields .....
    

    Let's try to explicitly "delete" this field:

    127.0.0.1:6379> HDEL p:734802 price
    (integer) 0
    127.0.0.1:6379> FT.ADDHASH il1 p:734802 1.0 REPLACE
    OK
    127.0.0.1:6379> "FT.SEARCH" "il1" "@ctitle:multistrada* 1260s* @tags:{w\\:couchride\\:t\\:adventure423} @year:[2019 inf] @geo:[-122.0363496 37.36883 10 mi]" "LIMIT" "0" "20" "SORTBY" "price" "DESC"
    1) (integer) 1
    2) "p:734802"
    3)  1) price
        2) (nil)
        3) imgurlsmall
       ..... more .... 
    

    "price" is still there in the results. If I ask for specific fields with RETURN, then it's ok.

    This is causing the go library I use to parse the result to fail: https://github.com/gomodule/redigo/issues/437

  • Patches for OSX compilation warnings

    Patches for OSX compilation warnings

    Fix compilation warnings caught on OS X:

    • NormalizeFunc was defined twice: remove the duplicate and fix up #includes
    • Avoid unsigned integer overflow bug vector in trie_type.c: slen-len isn't safe when the lengths are unsigned.

    Also modified the main Makefile to use the -Werror flag in all environments, since OS X doesn't complain anymore.

  • Redis 4.0.11 crashed by signal: 11

    Redis 4.0.11 crashed by signal: 11

    === REDIS BUG REPORT START: Cut & paste starting from here === 6215:M 11 Oct 19:07:34.617 # Redis 4.0.11 crashed by signal: 11 6215:M 11 Oct 19:07:34.617 # Crashed running the instruction at: 0x555589cc5015 6215:M 11 Oct 19:07:34.617 # Accessing address: (nil) 6215:M 11 Oct 19:07:34.617 # Failed assertion: (:0)

    ------ STACK TRACE ------ EIP: /usr/bin/redis-server 0.0.0.0:6600(je_arena_tcache_fill_small+0x1e5)[0x555589cc5015]

    Backtrace: /usr/bin/redis-server 0.0.0.0:6600(logStackTrace+0x5a)[0x555589c5f65a] /usr/bin/redis-server 0.0.0.0:6600(sigsegvHandler+0xb1)[0x555589c5fe11] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fc393e47890] /usr/bin/redis-server 0.0.0.0:6600(je_arena_tcache_fill_small+0x1e5)[0x555589cc5015] /usr/bin/redis-server 0.0.0.0:6600(je_tcache_alloc_small_hard+0x14)[0x555589ce8544] /usr/bin/redis-server 0.0.0.0:6600(je_arena_ralloc+0x826)[0x555589cc7bf6] /usr/bin/redis-server 0.0.0.0:6600(je_realloc+0xf3)[0x555589cbaaf3] /usr/bin/redis-server 0.0.0.0:6600(zrealloc+0x22)[0x555589c24d02] /etc/redis/modules/redisearch-1.4.0.so(Buffer_Grow+0x54)[0x7fc391808eb4] /etc/redis/modules/redisearch-1.4.0.so(+0x5340b)[0x7fc39181c40b] /etc/redis/modules/redisearch-1.4.0.so(InvertedIndex_WriteEntryGeneric+0x7d)[0x7fc39181d1ed] /etc/redis/modules/redisearch-1.4.0.so(InvertedIndex_WriteForwardIndexEntry+0x7d)[0x7fc39181d2ed] /etc/redis/modules/redisearch-1.4.0.so(+0x5136a)[0x7fc39181a36a] /etc/redis/modules/redisearch-1.4.0.so(+0x519e6)[0x7fc39181a9e6] /etc/redis/modules/redisearch-1.4.0.so(+0x51ea8)[0x7fc39181aea8] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fc393e3c6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc393b6588f]

    ------ INFO OUTPUT ------

    Server

    redis_version:4.0.11 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:4bc3aa215ca9cf78 redis_mode:standalone os:Linux 4.15.0-36-generic x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:7.3.0 process_id:6215 run_id:23c9a2164d6237dd48dbf6aae6342f9f9df2dda0 tcp_port:6600 uptime_in_seconds:4405 uptime_in_days:0 hz:10 lru_clock:12558198 executable:/usr/bin/redis-server config_file:/etc/redis/redis-hi-instant-stag.zooster.nl.conf

    Clients

    connected_clients:4 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:1

    Memory

    used_memory:103099856 used_memory_human:98.32M used_memory_rss:219738112 used_memory_rss_human:209.56M used_memory_peak:198129824 used_memory_peak_human:188.95M used_memory_peak_perc:52.04% used_memory_overhead:3044344 used_memory_startup:786800 used_memory_dataset:100055512 used_memory_dataset_perc:97.79% total_system_memory:2090156032 total_system_memory_human:1.95G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:2.13 mem_allocator:jemalloc-4.0.3 active_defrag_running:0 lazyfree_pending_objects:0

    Persistence

    loading:0 rdb_changes_since_last_save:94615 rdb_bgsave_in_progress:1 rdb_last_save_time:1539284754 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:3 rdb_current_bgsave_time_sec:0 rdb_last_cow_size:17166336 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0

    Stats

    total_connections_received:34 total_commands_processed:521727 instantaneous_ops_per_sec:4988 total_net_input_bytes:22978214 total_net_output_bytes:11222261 instantaneous_input_kbps:4.76 instantaneous_output_kbps:0.18 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:633773 keyspace_misses:105 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:10180 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

    Replication

    role:master connected_slaves:0 master_replid:546a55113d9284281b4a829c37a10b4370345895 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

    CPU

    used_cpu_sys:28.42 used_cpu_user:203.27 used_cpu_sys_children:4.29 used_cpu_user_children:26.47

    Commandstats

    cmdstat_sadd:calls=4221,usec=25237,usec_per_call=5.98 cmdstat_del:calls=303450,usec=996872,usec_per_call=3.29 cmdstat_FT.SEARCH:calls=69,usec=4399,usec_per_call=63.75 cmdstat_hset:calls=4221,usec=201846,usec_per_call=47.82 cmdstat_FT.AGGREGATE:calls=232,usec=9559,usec_per_call=41.20 cmdstat_FT.ADDHASH:calls=66436,usec=26641655,usec_per_call=401.01 cmdstat_get:calls=112,usec=631,usec_per_call=5.63 cmdstat_mget:calls=2,usec=905,usec_per_call=452.50 cmdstat_exec:calls=67555,usec=411243,usec_per_call=6.09 cmdstat_keys:calls=5,usec=203513,usec_per_call=40702.60 cmdstat_scan:calls=5133,usec=702923,usec_per_call=136.94 cmdstat_FT.CREATE:calls=4,usec=20690,usec_per_call=5172.50 cmdstat_smembers:calls=6,usec=79187,usec_per_call=13197.83 cmdstat_hgetall:calls=68852,usec=2106792,usec_per_call=30.60 cmdstat_multi:calls=1322,usec=962,usec_per_call=0.73 cmdstat_info:calls=36,usec=3750,usec_per_call=104.17 cmdstat_FT.DROP:calls=4,usec=4481329,usec_per_call=1120332.25 cmdstat_ping:calls=63,usec=100,usec_per_call=1.59 cmdstat_FT.INFO:calls=4,usec=8629,usec_per_call=2157.25

    Cluster

    cluster_enabled:0

    Keyspace

    db0:keys=26899,expires=0,avg_ttl=0

  • ReplyError: Could not load document

    ReplyError: Could not load document

    Works perfectly with latest dockerimage but fails with RedisLabs ePack.

    I am using FT.ADDHASH after a hmset.

    Any hint on how to get a more descriptive error?

  • ft.search returns duplicated result & ft.aggregate returns empty array

    ft.search returns duplicated result & ft.aggregate returns empty array

    (1) ft.search returns duplicate result ft.search idx "" limit 0 10 ft.search idx "" limit 10 10 the above searchs return duplicate result

    (2) ft.aggregate returns empty array ft.aggregate idx "*" limit 0 10 integer 1 (empty array) (empty array) ...

  • adding enterprise memory limit to vecsim index build

    adding enterprise memory limit to vecsim index build

    This PR fixes an issue where on enterprise cluster, another memory limit is introduced and we ignored it.

    there are no tests for this flag here as we don't test enterprise cluster in redisearch, but all the existing tests are passing. @TomerHekmati please test this

  • [2.4] Set timer for temporary index on master only (#2825)

    [2.4] Set timer for temporary index on master only (#2825)

    • temporary index has timer only on master with FT.DROPINDEX

    • per review

    • clean

    • remove Index_Temporary flag on _FT._DROPINDEXIFX

    • add _FORCEKEEPDOCS to prevent slave from calling DEL

    • Add testDropWith__FORCEKEEPDOCS

    (cherry picked from commit 790eb3978bc58aef7402603ec9bd7867503b0b6b)

  • [2.2] Set timer for temporary index on master only (#2825)

    [2.2] Set timer for temporary index on master only (#2825)

    • temporary index has timer only on master with FT.DROPINDEX

    • per review

    • clean

    • remove Index_Temporary flag on _FT._DROPINDEXIFX

    • add _FORCEKEEPDOCS to prevent slave from calling DEL

    • Add testDropWith__FORCEKEEPDOCS

    (cherry picked from commit 790eb3978bc58aef7402603ec9bd7867503b0b6b)

  • Is there a limit of the max number of schemas that can be created using

    Is there a limit of the max number of schemas that can be created using "FT.create index"?

    For one of the use cases, we are considering if we can create a schema per user id, e.g: "FT.create <user_id>". We have millions of users and we want to keep items applicable per user in the schema for faster personalized lookups with some flavor of search.

Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.
Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Jun 24, 2022
Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB. Scylla embraces a shared-nothing approach that increases throughput and storage capacity to realize order-of-magnitude performance improvements and reduce hardware costs.

Jun 24, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large-scale graphs with billions of vertices (nodes) and trillions of edges, with milliseconds of latency. It delivers enterprise-grade high performance to simplify the most complex data sets imaginable into meaningful and useful information.

Jun 24, 2022
🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.
🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

?? ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

Jun 24, 2022
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB

Jun 20, 2022
FEDB is a NewSQL database optimised for realtime inference and decisioning application
FEDB is a NewSQL database optimised for realtime inference and decisioning application

FEDB is a NewSQL database optimised for realtime inference and decisioning applications. These applications put real-time features extracted from multiple time windows through a pre-trained model to evaluate new data to support decision making. Existing in-memory databases cost hundreds or even thousands of milliseconds so they cannot meet the requirements of inference and decisioning applications.

Jun 16, 2022
network packet indexing and querying
network packet indexing and querying

_______ _____.___. ________ _____ _____ \ \\__ | |/ _____/ / \ / _ \ / | \/ | / \ ___ / \ / \ / /_\

Dec 15, 2021
A redis module, similar to redis zset, but you can set multiple scores for each member to support multi-dimensional sorting

TairZset: Support multi-score sorting zset Introduction Chinese TairZset is a data structure developed based on the redis module. Compared with the na

Jun 15, 2022
An OLED gauge for the Speeduino ECU. Uses UART (secondary serial) for communication.
An OLED gauge for the Speeduino ECU. Uses UART (secondary serial) for communication.

speeduino-ardugauge An OLED gauge for the Speeduino ECU. Uses UART (secondary serial) for communication. See demo video. See screenshots. NOTE: The ga

Jun 17, 2022
redis-cpp is a header-only library in C++17 for Redis (and C++11 backport)

redis-cpp - lightweight C++ client library for Redis redis-cpp is a C++17 library for executing Redis commands with support for pipelines and the publ

Jun 7, 2022
Typesense is a fast, typo-tolerant search engine for building delightful search experiences.
 Typesense is a fast, typo-tolerant search engine for building delightful search experiences.

Fast, typo tolerant, fuzzy search engine for building delightful search experiences ⚡ ??

Jun 17, 2022
A lightweight header-only C++11 library for quick and easy SQL querying with QtSql classes.

EasyQtSql EasyQtSql is a lightweight header-only C++11 library for quick and easy SQL querying with QtSql classes. Features: Header only C++11 library

Jun 1, 2022
TrailDB is an efficient tool for storing and querying series of events
TrailDB is an efficient tool for storing and querying series of events

TrailDB is an efficient tool for storing and querying series of events. This repository contains the core C library and the tdb command line tool.

Jun 15, 2022
Computational geometry and spatial indexing on the sphere

S2 Geometry Library Overview This is a package for manipulating geometric shapes. Unlike many geometry libraries, S2 is primarily designed to work wit

Jun 16, 2022
Slow5tools is a toolkit for converting (FAST5 <-> SLOW5), compressing, viewing, indexing and manipulating data in SLOW5 format.

slow5tools Slow5tools is a simple toolkit for converting (FAST5 <-> SLOW5), compressing, viewing, indexing and manipulating data in SLOW5 format. Abou

Jun 15, 2022
Text utilities, including beam search decoding, tokenizing, and more, built for use in Flashlight.

Flashlight Text: Fast, Lightweight Utilities for Text Quickstart | Installation | Python Documentation | Citing Flashlight Text is a fast, minimal lib

Jun 16, 2022
Hexagonal hierarchical geospatial indexing system

H3: A Hexagonal Hierarchical Geospatial Indexing System H3 is a geospatial indexing system using a hexagonal grid that can be (approximately) subdivid

Jun 17, 2022
Experimental 'RTree' Spatial Indexing

rtree The goal of rtree is to experiment with a few different types of spatial indexes at a pretty low level. Installation You can install the develop

May 20, 2022
This project Orchid-Fst implements a fast text string dictionary search data structure: Finite state transducer (short for FST) in c++ language.This FST C++ open source project has much significant advantages.
This project Orchid-Fst implements a fast text string dictionary search data structure: Finite state transducer (short for FST) in c++ language.This FST C++ open source project has much significant advantages.

Orchid-Fst 1. Project Overview This project Orchid-Fst implements a fast text string dictionary search data structure: Finite state transducer , which

Jan 5, 2022
Read file to console, automatically recognize file encoding, include ansi, utf16le, utf16be, utf8. Currently output ansi as gbk for chinese text search.

rgpre A tool for rg --pre. Read file to console, automatically recognize file encoding, include ansi, utf16le, utf16be, utf8. Currently output ansi as

Mar 18, 2022