Slow performance when opening files or invoking functions

[nadavspi]

Love the idea of this package!

I have about 5000 files and I'm seeing a delay of a few seconds whenever I open a file in my Obsidian path, or callobsidian-jump, obsidian-backlink-jump, etc. I'm using a pretty much out of box Doom config. Files within my obsidian directory open quickly when obsidian.el isn't configured.

Is this expected? Anything I can do to help debug?

[licht1stein]

We need to add caching, it would fix the lag. I was expecting somebody with a large vault to comment.

…

On 5 Dec 2022 at 18:10 +0000, Nadav Spiegelman ***@***.***>, wrote: Love the idea of this package! I have about 5000 files and I'm seeing a delay of a few seconds whenever I open a file in my Obsidian path, or callobsidian-jump, obsidian-backlink-jump, etc. I'm using a pretty much out of box Doom config. Files within my obsidian directory open quickly when obsidian.el isn't configured. Is this expected? Anything I can do to help debug? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nadavspi]

I figured as much after looking through the code 😀

[licht1stein]

I would be very grateful for a PR :)

…

On 5 Dec 2022 at 18:52 +0000, Nadav Spiegelman ***@***.***>, wrote: I figured as much after looking through the code 😀 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[nadavspi]

I'm not sure my lisp chops will allow for that, but I may try.

[licht1stein]

I'll be glad to help. Don't have time to do it quickly myself, but I'll be glad to comment on your efforts.

…

On 5 Dec 2022 at 19:20 +0000, Nadav Spiegelman ***@***.***>, wrote: I'm not sure my lisp chops will allow for that, but I may try. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[DrBanana419]

Did this issue get fixed for anyone? Jumping to notes is still slow for me.

[licht1stein]

Did this issue get fixed for anyone? Jumping to notes is still slow for me.

yes

[jayemar]

Did this issue get fixed for anyone? Jumping to notes is still slow for me.

In my own branch I've commented out the (obsidian-update) call inside the obsidian-jump function. This greatly increases how quickly I can jump between notes, but obviously sacrifices some of the guarantees of updated tags and file cache.

[smallstepman]

im also experiencing slowness

In my own branch I've commented out the (obsidian-update) call inside the obsidian-jump function. This greatly increases how quickly I can jump between notes, but obviously sacrifices some of the guarantees of updated tags and file cache.

@licht1stein would be nice to have an option to disable this in the config

[licht1stein]

Sure, I will look into it when I have time, if somebody else doesn't implement it

[fstamour]

I did a bunch of tests today. For my use case, I managed to make things 1000 times faster.

I've made a very very low quality commit of the things I've tried here.

Here's a summary:

• avoid building big intermediary lists (e.g. pass the predicates directly to directory-files-recursively)
• try to make obsidian-file-p as fast as possible
  • run faster, simpler predicates earlier and the heaviest later
  • check if it ends in .md early
  • check if path is inside .git or node_modules early
  • avoid obsidian-descendant-of-p
• listing files using projectile-dir-files-alien instead of directory-files-recursively (This wouldn't be generally applicable, as it assumes that projectile is installed and that we're currently in a project.)
• reduce the number of with-temp-buffer, this one I discovered using the profiler

Here's some utilities that I used over and over to help be optimize stuff:

(defmacro fsta/comment (&rest body)
  "Comment BODY."
  nil)

(defmacro fsta/measure-time (&rest body)
  "Measure the time it takes to evaluate BODY."
  `(let ((time (current-time)))  ;; TODO Time should be gensym'd
     (prog1
         (progn ,@body)
       (message "%.06fs" (float-time (time-since time))))))

(defun fsta/profile (fn)
  (unwind-protect
      (progn
        (profiler-start 'cpu+mem)
        (funcall fn)
        (profiler-report))
    (profiler-stop)))

Here are the functions that takes the most time:

• obsidian-file-p: check if a path is a markdown file, it's not slow per-se, but it is called repeatedly
• obsidian-reset-cache: it lists all the files)
• obsidian-update-tags-list: it reads all files and looks for tags
• obsidian--update-all-from-front-matter: it reads all the files and parse their "front matter"

---

Here are some other ideas that I didn't have time to test:

• use a hash-table instead of a list for caching the list of files
• use one timestamp per file (easy to do once you use a hash-table)
  • if the entry is expired, check the file's modification time to decide whether to re-read it or not
• use emacs' file watching facilities to update the cache as soon as a file is changed
• use hooks (like after-save-hook) to update a file's entry in the cache as soon as it's saved
• in obsidian-file-p: if a file is already in the cache, then it means it already passed the predicate, we don't need to check every conditions again
• extracting tags and extracting the front matter should be done at the same time, currently (AFAIR) every files are read twice when calling (obsidian-update)

---

(I might make a PR, if I have time, no promise 😅)

[licht1stein]

Wow, this is great work! I would be very grateful for the PR, with exception of the projectile-related part. I also think that hash-tables would be a better way to handle things.

[fstamour]

Notes for myself:

• there's 2 ci jobs: one for testing and one for publishing
• the test jobs calls make test
• make test calls eldev -C --unstable -T test
  • -T is short for --time
  • -C is short for --color
  • --unstable tells eldev to use melpa-unstable
• https://github.com/emacs-eldev/eldev
• eldev is configured with the Eldev file
• the tests use the (BDD) test framework buttercup
• there's already a dummy vault in tests/test-vault/

[licht1stein]

Notes for myself:

• there's 2 ci jobs: one for testing and one for publishing

• the test jobs calls make test

• make test calls eldev -C --unstable -T test

  • -T is short for --time
  • -C is short for --color
  • --unstable tells eldev to use melpa-unstable

• https://github.com/emacs-eldev/eldev

• eldev is configured with the Eldev file

• the tests use the (BDD) test framework buttercup

• there's already a dummy vault in tests/test-vault/

That would actually be a great addition to readme

[fstamour]

It would have been funny if it was already there, because I didn't even look at the readme 😆

[Dima-369]

I find the lag on (obsidian-jump) particularly annoying.

Here is my approach (which org-roam also uses) to soften this a bit, so at least jumping to existing notes is quick.

(defvar dima-obsidian-vault-path "~/Documents/Obsidian Vault/"
  "The directory to the Obsidian Vault.")

(defun dima-obsidian-update-after-save-hook ()
  (when (s-starts-with-p default-directory (f-expand dima-obsidian-vault-path))
    (message "Updating Obsidian Vault...")
    (obsidian-update)))

(add-hook 'after-save-hook #'dima-obsidian-update-after-save-hook)

Then create a custom (obsidian-jump) function like this.

Mine also invokes (obsidian-capture) when no match is found, which also mimics how org-roam does its find note.

(defun dima-obsidian-jump ()
    "Jump to Obsidian note.

Patches:
- remove slow call to `obsidian-update'
- when no target is there, create a new note"
    (interactive)
    (let ((gc-cons-threshold most-positive-fixnum))
      (let* ((files (obsidian-list-all-files))
             (dict (make-hash-table :test 'equal))
             (_ (-map (lambda (f) (puthash (file-relative-name f obsidian-directory) f dict)) files))
             (choices (-sort #'string< (-distinct (-concat (obsidian--all-aliases) (hash-table-keys dict)))))
             (choice (completing-read "Jump to: " choices))
             (target (obsidian--get-alias choice (gethash choice dict))))
        (if target
            (find-file target)
          ;; inline `obsidian-capture' without title variable
          (let* ((filename
                  (s-concat obsidian-directory "/" obsidian-inbox-directory "/" choice ".md"))
                 (clean-filename (s-replace "//" "/" filename)))
            (find-file (expand-file-name clean-filename) t)
            (save-buffer)
            (add-to-list 'obsidian-files-cache clean-filename))))))

Still, (obsidian-update) seems annoyingly slow to me with just 1500 Markdown files, so I will check the notes of @fstamour.