Segfault on Linux

Help & Support
1

have an annoying issue occurring on Linux but working fine on MacOS
spider-gazelle/tasker@f7e462c

Invalid memory access (signal 11) at address 0x0
[0x561a7c4a6b36] *Exception::CallStack::print_backtrace:(Int32 | Nil) +118
[0x561a7c49397c] __crystal_sigfault_handler +316
[0x7efcceea8980] ???
[0x0] ???

That’s using the latest ubuntu docker image
I’ve isolated the issue to this spec:
tasker/tasker_spec.cr

2

I would check if setting CRYSTAL_LOAD_DWARF=1 or CRYSTAL_LOAD_DWARF=0 leads to a different output. That would narrow/discard if the problem itself is on the dwarf loading which is different on linux and darwin.

1 Like
3

@bcardiff the dwarf loading made no difference

I used GDB and tracked the segfault to this line:

image
image1220×566 49.9 KB

Stepping through on GDB I see:

image
image1488×392 24.1 KB

1 Like
4

Grabbed this just before the seg fault

  1. break point set for the break on line 149
  2. called step and it jumped into the else statement - but it should have exited the loop

image
image1484×1392 184 KB

A screen shot of state just after the second step statement

image
image1490×580 26.5 KB

5

Although when I overloaded the reschedule function with an alternative implementation I’m still seeing the crash.

What’s also weird is that the crash doesn’t consistently happen either… Crashes 2 out of 3 runs - just to make life difficult

7

I solved it by refactoring away from using Fiber scheduling to using channels
as I’d built the library before select timeout was available

2 Likes
8

Good! You mean this change right? Getting rid of those sleep is :rocket:

Maybe the segfault was coming from rescheduling a fiber in an invalid state, so it’s definitely better to use Channel for synchronization.

On that note, maybe you want the cancel channel to have a capacity of 1 (instead of 0) so the #cancel method will never block. I think right now it could in some race condition. The cancel.send will block until a receiver gets the message, but that is not what you want here.

3 Likes
9

Is this multi-threaded? Any simplified code to repro it?

10

not multi-threaded and no simplified code…
it was pretty hacky fiber manipulation - waking fibers from sleep early, which might have left some dangling pointers somewhere?

This was the fiber timer

Effectively:

  • sleep for the timer period and then perform an action if not cancelled
  • cancelling the timer involved flagging that it was cancelled and then waking it up early to clean up memory
11

I’m not sure and I could be totally off, but would a mutex lock around @cancelled help?

12

Shouldn’t be required as running on the one thread, so most likely unrelated.
The issue only occurred on Linux and not every run, so probably going to be a tough one to track down

13

Hard to diagnose without a simplified example. Maybe valgrind help? Good luck!

14

All good, I refactored around it.