Option for fuzzy search by default?

[marlun]

I'm guessing you've chosen to do this for performance reason but I'm going to bring it up anyway 😄 What do you think about an option to enable this by default without having to add a * to the search? If possible without a noticeable performance hit I would really prefer this in my setup.

[pgdouyon]

Sorry for the delay in responding to this, it's been a crazy couple of weeks and I wanted to make sure I thought through this proposal thoroughly.

Adding fuzzy finding would actually be pretty straightforward, once you have an algorithm, the sorting function just replaces the current sort_by_mru() function. However, I think I don't want Niffler to provide true fuzzy finding for the following reasons (I was actually surprised at how low on the list performance was when I came up with this list):

If Niffler has a fuzzy matcher, then what differentiates it from CtrlP?

• CtrlP with Ag or Pt as the user command is about as fast as Niffler already
• Niffler might have a nicer codebase, but CtrlP has a massive and thriving ecosystem of extensions available to it.
• CtrlP is battled tested on a variety of different Vim setups and OSes, plus the vast majority of the Vim community is familiar with CtrlP which is incredibly advantageous when pairing or seeking help
• My point is that CtrlP has numerous advantages over Niffler, and I think Niffler's only advantages over CtrlP is its simplicity and its faux-fuzzy method of filtering files.
  • Niffler's cleaner codebase is also somewhat of an advantage over CtrlP but is pointless if all we do is rebuild CtrlP on top of it.
  • The reason I think the faux-fuzzy matching is an advantage is because I've found it's generally faster than fuzzy matching and has much less cognitive overhead.

Niffler's simplicity is one of its greatest strengths and I think the faux-fuzzy matching is part of the simple philosophy

• Fuzzy matching wouldn't be difficult to implement or compromise too much in the way of simplicity, but it is another option to add and more cognitive load for the user
• Personally, I tend to prefer opinionated plugins and I think the simplicity that they afford is part of the reason for that preference.

I may be the only person in the world who hates fuzzy matching.

• The main reason is that it's too unpredictable.
  • I find it difficult to consistently come up with a query with the right blend of specificity and fuzziness such that you don't need to type another 10-15 characters just to locate your file.
  • It's such an unpleasant experience when typing "foobar" will get me the file I want, but God forbid I leave out the second "o" one time or my file system changes and 20 other files now also match "fobar" or "foobar"
  • I've seen too many cases where "foobar" fuzzy matches the file you want, but if you went too fast and typed another fuzzy character and the query becomes "foobarx" and all of a sudden 5 other files are better matches. This is madness to me.
• Personally, I'm more likely to know the name of the file I'm looking for, than any part of the full path to it. I'm terrible with directories.
  • I've also found that I can generally type out the full name of a file faster than several disjointed components of the path. Simply because there's less overhead, i.e. I only have to hold one name in my head instead of a path.

Performance, however, is definitely still a drawback.

• Because the fuzzy sorter (not matcher) would need to be implemented in VimL, which is slow, I would have to limit the number of files passed to it. Most likely limited to the number that can fit on screen.
  • This is a pretty poor fit for fuzzy matching because out of thousands of candidate files 30 or so will be arbitrarily selected and sorted and displayed on screen.
  • There can be no regard to how well these 30 files match the fuzzy pattern because they're arbitrarily selected -- i.e. the length of the fuzzy match could be the length of the whole file path
• This is actually how CtrlP works and I tried it out on a fairly large project (~6000 files) and found its performance difficult to predict.
  • Sometimes it would find a file as fast as Niffler, other times I would have 15 character queries and still not find the file I was looking for.

Those at least are my arguments against adding a fuzzy matcher. To be fair, I can at least see one strong argument in favor which is that since adding fuzzy matching to Niffler wouldn't compromise too much on simplicity Niffler could still function as a simpler CtrlP. I can definitely see the value in this proposition.

The other argument I can see in favor of fuzzy matching is if you have a project structure with several identically named files in different folders (in which case you need other components of the pathname to identify them). Although in that case I think something like Projectionist is probably a better fit than fuzzy matching.

Also if it helps, I actually rarely use the * with Niffler, it's more of a last resort for me. Usually I just type the filename itself and if that didn't narrow it down enough for some strange reason, I just type a space and then some other component of the path. I find the space somewhat less intrusive than typing a * character.

• Just realized that this isn't in the README, Niffler supports conjunctive searching where you separate search terms with a space. So Niffler will match any files that contain all the space-separated terms in any order.

[marlun]

No worries, better to leave a great answer than a hasted one 😄

For me the UI of Niffler is alot nicer than CtrlP and a big reason of why I like it so much. I agree on many points and I didn't know you could use separate search terms. I'm going to try and use that since most of the times when I want fuzzy search is when I've got a long filename like longFailenameAboutSomethingGood.js. I usually write something like "longgood" but "long good" should work just as fine.

That got me thinking about another similar feature that could be intresting to bring up now too: camcelCase matching. For example if I do :NifflerTags % and I've got a function named doSometingUseful I could just write "dSU" and it would select it. It's not something I actually think is needed but it deserves to be brought up 😄

[marlun]

Thought about this again and your comment

Personally, I'm more likely to know the name of the file I'm looking for, than any part of the full path to it. I'm terrible with directories.

I've got a folder which looks like x/x/x/foo-bar/foo-bar-service.js and if I run Niffler and write foo-bar it matches the directory foo-bar and has a lot of other files above my foo-bar-service.js.

It would've been great if matches on filenames were at the top, not sure if it's simple enough to implement? Maybe just searching the whole path when there's a / in the filter?

[pgdouyon]

Yea this has been bothering me as well and I've thought about doing some kind of filename sorting. The only reason I've held off on it is because I wasn't sure if I could find a performant way to implement it.

I'll try to play around with it this weekend and see what I can get.

[marlun]

I'm guessing you haven't had time to look into this but do you have any thoughts about it that could help if I decide to look into it myself? Hopefully got some free time over easter and thought I could do something useful 😄

[pgdouyon]

Yea still have no idea how I want to approach it. There are a couple approaches that I can think of but all have their drawbacks:

• You can add a second grep command in filter_candidate_list to first only look for files matching the basename, and then grep a second time for everything else and combine the two lists removing duplicates. This one is really tricky though because it has to only be done for file searches (as opposed to tags or custom ones) so there either needs to be a flag or a separate function stored in the b:niffler dict that is called if present.
• You could somehow pass or give access to the current query string to sort_by_mru so that it can sort off the basename of the files in addition to by mru status. would probably need to rename the method at that point and break it out into two separate sorting functions.

The first approach has the advantage of providing better filtering/matching. If you filter after the grep command has run, there's a good chance that the grep command returned a bunch of files whose basename doesn't match the pattern. So while it's more complicated, you'll get less noise and better results.

The second approach is incredibly simple/straightforward, but will probably be less performant since I'm guessing a grep filter is usually going to be faster than a sort in VimL.

[marlun]

Sorry for not getting back to you on this one but as usual the free time I had went into other things than I'd planned.

If I understand you correctly it's the first approach that we should try to look into first since if that works out it would probably be the best one. A question about the first approach: You write "to first only look for files matching the basename, and then grep a second time for everything else" what is everything else? 😄

Will make another try this weekend.

[pgdouyon]

To match the rest of the path, in case for some reason someone wanted to filter by directory names or by parts of the path. In a Vim project for instance, the folder structure usually has multiple files with the same basename, but in different directories. So it would actually be easier to search for a file by typing "autoload" or "plugin" or "doc" or something like that.

The everything else is basically the grep filter that the code currently executes, and the filter for basename would be added just before that.

[marlun]

What about adding the possibility to add a "pre_processor" to niffler_options and then if it exists we run it and unless it returns false (or some other value, empty list?) we use that candidate_list if it's empty we keep running the normal function? Then the first thing we could do in the basename filter function is check for a slash and if there is one we just return false (or empty) and the normal function gets used but if not we just filter by basename? Then you would have to plug / to filter by path though... 😄 but it would be possible to use it for simlilar features in other "modes" too.