IO Error - Error reading the segment infos

[dennis@…]

I was doing a bunch of querries after another and after a while it just failed with a very basic call:

clazz.find_by_contents('"' + [terms[index], terms[index+2]].join(" ") + '"~4', find_options).total_hits

The Trace:

..../usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:710:in `latest?': IO Error occured at <except.c>:93 in xraise (IOError) Error occured in index.c:901 - sis_find_segments_file

Error reading the segment infos. Store listing was

from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:710:in `ensure_reader_open' from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:733:in `ensure_searcher_open' from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:675:in `process_query' from /usr/lib/ruby/1.8/monitor.rb:229:in `synchronize' from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:674:in `process_query' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/local_index.rb:87:in `find_id_by_contents' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:82:in `find_id_by_contents' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:134:in `ar_find_by_contents' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:128:in `find_records_lazy_or_not' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:54:in `find_by_contents' from ./script/indexing/../../config/../lib/ferret_linker.rb:63:in `get_terms' from ./script/indexing/../../config/../lib/ferret_linker.rb:58:in `get_terms' from ./script/indexing/../../config/../lib/ferret_linker.rb:12:in `get_overlaps_to' from script/indexing/ferret_based_linker.rb:26 from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/acts_as_ferret.rb:87:in `method_missing' from script/indexing/ferret_based_linker.rb:22 from script/indexing/ferret_based_linker.rb:9

[dennis@…]

could this be caused by a search table where the initial index did not happen properly? I just deleted the index directory and am ensuring the initial index happens properly by going in the console and doing a full_text_search (part of acts_as_ferret's magic).

dennis

[dennis@…]

that seems to have caused the problem, deleting the index and regenerating did the trick

[dennis@…]

Apparently this error just occurs after some time. I ran into it again

../usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:710:in `latest?': IO Error occured at <except.c>:93 in xraise (IOError) Error occured in index.c:901 - sis_find_segments_file

Error reading the segment infos. Store listing was

from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:710:in `ensure_reader_open' from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:733:in `ensure_searcher_open' from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:675:in `process_query' from /usr/lib/ruby/1.8/monitor.rb:229:in `synchronize' from /usr/lib/ruby/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:674:in `process_query' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/local_index.rb:87:in `find_id_by_contents' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:82:in `find_id_by_contents' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:134:in `ar_find_by_contents' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:128:in `find_records_lazy_or_not' from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/class_methods.rb:54:in `find_by_contents' from ./script/indexing/../../config/../lib/ferret_linker.rb:66:in `get_terms' from ./script/indexing/../../config/../lib/ferret_linker.rb:64:in `get_terms' from ./script/indexing/../../config/../lib/ferret_linker.rb:12:in `get_overlaps_to' from script/indexing/ferret_based_linker.rb:26 from /usr/lib/ruby/gems/1.8/gems/acts_as_ferret-0.4.0/lib/acts_as_ferret.rb:87:in `method_missing' from script/indexing/ferret_based_linker.rb:22 from script/indexing/ferret_based_linker.rb:9

[wmorgan-ferret@…]

I have some Sup users who also see this on a regular basis. Smallish index, lots of read and write. Haven't been able to track down the exact sequence of operations that triggers it.

Here's the top part of the backtrace, which is slightly different from the two above, though I don't know that it contains any more information:

/var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:726:in `initialize': IO Error occured at <except.c>:93 in xraise (IOError) Error occured in index.c:901 - sis_find_segments_file

Error reading the segment infos. Store listing was

from /var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:726:in

`new'

from /var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:726:in

`ensure_reader_open'

from /var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:733:in

`ensure_searcher_open'

from /var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:755:in

`do_search'

from /var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:334:in

`search'

from /usr/lib/ruby/1.8/monitor.rb:238:in `synchronize' from /var/lib/gems/1.8/gems/ferret-0.11.4/lib/ferret/index.rb:333:in

`search'

[dennis@…]

This guy mentions that there seems to be a limit with too many open files. Could it be that ferret is not closing files properly? I don't know where in the source i'd look but i'll try.

 http://www.ruby-forum.com/topic/120609

[anonymous]

test

[dbalmain]

Replying to dennis@dennisschaaf.com:

This guy mentions that there seems to be a limit with too many open files. Could it be that ferret is not closing files properly? I don't know where in the source i'd look but i'll try.  http://www.ruby-forum.com/topic/120609

It may have something to do with this although I can't reproduce the problem here so I'm not sure. Even if this is the cause of the error, it isn't necessarily a bug in Ferret. Because of the way Ferret's index is designed, it does tend to use a lot of file handles, especially if you set :use_compound_file => false. However, Eric Morton's post shows that he is using compound files so it does seem like there might be a bug somewhere.

[anonymous]

I got around this issue by calling flush() on the index after deleting entries.