reentrant sprintf causes segmentation fault

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

reentrant sprintf causes segmentation fault

David Lawrence
Dear newlib maintainers and friends,

I am having some trouble with newlib on a bare-metal embedded system
that makes reentrant calls to sprintf. The reentrant call reliably
experiences a segmentation fault. I have tested with both newlib 2.4.0
and newlib 3.0.0, with both newlib-nano and regular newlib, to no
effect.

Here is some code that reproduces the issue. This code runs on an NXP
LPC1549 microcontroller -- I have not yet attempted to reproduce the
issue in my desktop Linux environment. The binary is statically
linked.

#include <stdio.h>
#include "chip.h"
__asm__(".global _printf_float");
__asm__(".global _scanf_float");
float f_main = 1.111111;
float f_irq = 5.555555;
char main_buf[200];
char irq_buf[200];
void main(void) {
  // Set up a repetitive interrupt every 100000 clock cycles.
  Chip_MRT_Init();
  NVIC_EnableIRQ(MRT_IRQn);
  Chip_MRT_SetMode(LPC_MRT_CH1, MRT_MODE_REPEAT);
  Chip_MRT_SetInterval(LPC_MRT_CH1, 100000 | MRT_INTVAL_LOAD);
  Chip_MRT_SetEnabled(LPC_MRT_CH1);
  for (;;) {
    sprintf(main_buf, "%f %f %f\r\n", f_main, f_main, f_main);
  }
}
void MRT_IRQHandler() {  // called asynchronously and preempts main()
  Chip_MRT_IntClear(LPC_MRT_CH1);
  sprintf(irq_buf, "%f %f %f\r\n", f_irq, f_irq, f_irq);
}

Below is the backtrace from a segmentation fault within the interrupt:

Program received signal SIGSEGV, Segmentation fault.
_Balloc (ptr=ptr@entry=0x2000018 <impure_data>, k=0)
    at ../../../../../../../newlib/libc/stdlib/mprec.c:117
(gdb) bt
#0  _Balloc (ptr=ptr@entry=0x2000018 <impure_data>, k=0)
    at ../../../../../../../newlib/libc/stdlib/mprec.c:117
#1  0x00002fb0 in _dtoa_r (ptr=ptr@entry=0x2000018 <impure_data>,
    _d=<optimized out>, mode=<optimized out>, mode@entry=3,
ndigits=ndigits@entry=6,
    decpt=decpt@entry=0x2007c1c, sign=sign@entry=0x2007c20,
rve=rve@entry=0x2007c2c)
    at ../../../../../../../newlib/libc/stdlib/dtoa.c:426
#2  0x0000298c in cvt (buf=0x6ab6 " %f %f\r\n", length=<synthetic pointer>,
    ch=<optimized out>, decpt=0x2007c1c, sign=<synthetic pointer>, flags=256,
    ndigits=6, value=5.5555548667907715, data=<optimized out>)
    at ../../../../../../../newlib/libc/stdio/vfprintf.c:1877
#3  _svfprintf_r (data=<optimized out>, fp=fp@entry=0x2007cc0,
    fmt0=fmt0@entry=0x7fffffff <error: Cannot access memory at address
0x7fffffff>,
    ap=..., ap@entry=...) at
../../../../../../../newlib/libc/stdio/vfprintf.c:1314
#4  0x000016e8 in sprintf (str=<optimized out>, fmt=0x6ab4 "%f %f %f\r\n")
    at ../../../../../../../newlib/libc/stdio/sprintf.c:618
#5  0x00001426 in MRT_IRQHandler () at src/main.c:22
#6  <signal handler called>
#7  _dtoa_r (ptr=ptr@entry=0x2000018 <impure_data>, _d=<optimized out>,
    mode=<optimized out>, mode@entry=3, ndigits=ndigits@entry=6,
    decpt=decpt@entry=0x2007ea4, sign=sign@entry=0x2007ea8,
rve=rve@entry=0x2007eb4)
    at ../../../../../../../newlib/libc/stdlib/dtoa.c:429
#8  0x0000298c in cvt (buf=0x6ab6 " %f %f\r\n", length=<synthetic pointer>,
    ch=<optimized out>, decpt=0x2007ea4, sign=<synthetic pointer>, flags=256,
    ndigits=6, value=1.111111044883728, data=<optimized out>)
    at ../../../../../../../newlib/libc/stdio/vfprintf.c:1877
#9  _svfprintf_r (data=<optimized out>, fp=fp@entry=0x2007f48,
    fmt0=fmt0@entry=0xfffffff9 <error: Cannot access memory at address
0xfffffff9>,
    ap=..., ap@entry=...) at
../../../../../../../newlib/libc/stdio/vfprintf.c:1314
#10 0x000016e8 in sprintf (str=<optimized out>, fmt=0x6ab4 "%f %f %f\r\n")
    at ../../../../../../../newlib/libc/stdio/sprintf.c:618
#11 0x00001214 in main () at src/main.c:17

The backtrace after a segfault can vary slightly (based on the
specific timing and whether it's the main loop's call to sprintf or
the interrupt's call to sprintf that segfaults first), but it is
always in a call to either _Bfree() or _Balloc() in mprec.c.

I implemented __malloc_lock() and __malloc_unlock() as below, but
unfortunately, this had no impact on the problem and I still observed
segmentation faults.

volatile int32_t lock_count = 0;
void __malloc_lock(void *ptr) {
  __disable_irq();
  ++lock_count;
}
void __malloc_unlock(void *ptr) {
  if (--lock_count == 0) {
    __enable_irq();
  }
}

I then used the linker to wrap calls to _dtoa_r(), effectively making
these calls atomic. This eliminated segmentation faults for this
particular example.

// Additional linker flags: --wrap=_dtoa_r
extern char * __real__dtoa_r(void *ptr, double _d, int mode, int ndigits,
                             int *decpt, int *sign, char **rve);
__attribute__ ((used)) char * __wrap__dtoa_r(
    void *ptr, double _d, int mode, int ndigits,
    int *decpt, int *sign, char **rve) {
  __malloc_lock(ptr);
  return __real__dtoa_r(ptr, _d, mode, ndigits, decpt, sign, rve);
  __malloc_unlock(ptr);
}

Although making _dtoa_r() atomic appeared to fix my problem, upon
further testing, it is not a robust solution. I next added the single
line 'printf("a");' to my main loop (just below the first call to
sprintf) and immediately obtained a different segmentation fault:

Program received signal SIGSEGV, Segmentation fault.
__swsetup_r (ptr=ptr@entry=0x2000014 <impure_data>, fp=0x620a, fp@entry=0x1)
    at ../../../../../../../newlib/libc/stdio/wsetup.c:93
(gdb) bt
#0  __swsetup_r (ptr=ptr@entry=0x2000014 <impure_data>, fp=0x620a, fp@entry=0x1)
    at ../../../../../../../newlib/libc/stdio/wsetup.c:93
#1  0x00004276 in _vfprintf_r (data=data@entry=0x2000014 <impure_data>, fp=0x1,
    fmt0=0x620a "a", fmt0@entry=0x0, ap=..., ap@entry=...)
    at ../../../../../../../newlib/libc/stdio/nano-vfprintf.c:491
#2  0x00003388 in printf (fmt=0x620a "a")
    at ../../../../../../../newlib/libc/stdio/printf.c:56
#3  0x00001310 in main () at src/main.c:18

I'm compiling and linking my program on Ubuntu 18.04 using
arm-none-eabi-gcc version 6.3.1. I use the following compiler flags:

-mthumb -mcpu=cortex-m3 -std=c99 -Os -Wall
--fno-common -ffreestanding -static -nostartfiles
--specs=nano.specs --specs=nosys.specs

I have repeated all tests with the --specs=nano.spacs flag omitted,
with no change in results.

Do I have a fundamental misunderstanding of newlib's capability to
support reentrant calls? Or does this seem like a legitimate bug? I
would appreciate any suggestions for workarounds, and I would be happy
to provide additional information or perform any further tests to
assist the maintainers with this possible issue.

Best regards,

David Lawrence

Electrical engineer at Markforged Inc.
https://markforged.com
Reply | Threaded
Open this post in threaded view
|

Re: reentrant sprintf causes segmentation fault

Craig Howland
On 5/3/19 12:43 PM, David Lawrence wrote:

> Dear newlib maintainers and friends,
>
> I am having some trouble with newlib on a bare-metal embedded system
> that makes reentrant calls to sprintf. The reentrant call reliably
> experiences a segmentation fault. I have tested with both newlib 2.4.0
> and newlib 3.0.0, with both newlib-nano and regular newlib, to no
> effect.
>
> ...
>
> Do I have a fundamental misunderstanding of newlib's capability to
> support reentrant calls? Or does this seem like a legitimate bug? I
> would appreciate any suggestions for workarounds, and I would be happy
> to provide additional information or perform any further tests to
> assist the maintainers with this possible issue.
>
> Best regards,
>
> David Lawrence
David:
      Reentrancy requires setup on your part, of which I see none in your
supplied fragments.  Generate the manual and look at the reentrancy section. 
(You can also look at libc/reent/reent.tex, avoiding the need to make the manual
but then being in source with formatting commands.)  It describes the 2 methods
for handling it. In brief, you either need to directly call the reentrant forms
of the functions with a per-thread structure, or you need to "context switch" by
setting the global reentrancy pointer for each thread. (No, they do not need to
be proper threads, but it gets the point across easily to use that term as an
analogy.)  The second method is easier from the perspective of hiding reentrency
from the main body of application code.  Both cases require you to define at
least one of your own reentrancy structures in addition to the single default one.
      A simple technique I have used in an application quite like yours is to
have the interrupt handler save _impure_ptr, overwrite it with the ISR's own,
print, then restore _impure_ptr.
                 Craig
Reply | Threaded
Open this post in threaded view
|

Re: reentrant sprintf causes segmentation fault

David Lawrence
Dear Craig,

Thank you very much for the explanation and pointer to the manual.
Indeed, the manual is quite clear on this topic and I'm embarrassed
that I missed it on my first reading. Sorry to waste your time with
this question :-)

Best regards,
David

On Fri, May 3, 2019 at 1:18 PM Craig Howland <[hidden email]> wrote:

>
> On 5/3/19 12:43 PM, David Lawrence wrote:
> > Dear newlib maintainers and friends,
> >
> > I am having some trouble with newlib on a bare-metal embedded system
> > that makes reentrant calls to sprintf. The reentrant call reliably
> > experiences a segmentation fault. I have tested with both newlib 2.4.0
> > and newlib 3.0.0, with both newlib-nano and regular newlib, to no
> > effect.
> >
> > ...
> >
> > Do I have a fundamental misunderstanding of newlib's capability to
> > support reentrant calls? Or does this seem like a legitimate bug? I
> > would appreciate any suggestions for workarounds, and I would be happy
> > to provide additional information or perform any further tests to
> > assist the maintainers with this possible issue.
> >
> > Best regards,
> >
> > David Lawrence
> David:
>       Reentrancy requires setup on your part, of which I see none in your
> supplied fragments.  Generate the manual and look at the reentrancy section.
> (You can also look at libc/reent/reent.tex, avoiding the need to make the manual
> but then being in source with formatting commands.)  It describes the 2 methods
> for handling it. In brief, you either need to directly call the reentrant forms
> of the functions with a per-thread structure, or you need to "context switch" by
> setting the global reentrancy pointer for each thread. (No, they do not need to
> be proper threads, but it gets the point across easily to use that term as an
> analogy.)  The second method is easier from the perspective of hiding reentrency
> from the main body of application code.  Both cases require you to define at
> least one of your own reentrancy structures in addition to the single default one.
>       A simple technique I have used in an application quite like yours is to
> have the interrupt handler save _impure_ptr, overwrite it with the ISR's own,
> print, then restore _impure_ptr.
>                  Craig
Reply | Threaded
Open this post in threaded view
|

Re: reentrant sprintf causes segmentation fault

Dave Nadler-2
David - If you're going to use FreeRTOS or similar, this might prove
helpful:
http://www.nadler.com/embedded/newlibAndFreeRTOS.html

On 5/3/2019 1:25 PM, David Lawrence wrote:

> Dear Craig,
>
> Thank you very much for the explanation and pointer to the manual.
> Indeed, the manual is quite clear on this topic and I'm embarrassed
> that I missed it on my first reading. Sorry to waste your time with
> this question :-)
>
> Best regards,
> David
>
> On Fri, May 3, 2019 at 1:18 PM Craig Howland <[hidden email]> wrote:
>> On 5/3/19 12:43 PM, David Lawrence wrote:
>>> Dear newlib maintainers and friends,
>>>
>>> I am having some trouble with newlib on a bare-metal embedded system
>>> that makes reentrant calls to sprintf. The reentrant call reliably
>>> experiences a segmentation fault. I have tested with both newlib 2.4.0
>>> and newlib 3.0.0, with both newlib-nano and regular newlib, to no
>>> effect.
>>>
>>> ...
>>>
>>> Do I have a fundamental misunderstanding of newlib's capability to
>>> support reentrant calls? Or does this seem like a legitimate bug? I
>>> would appreciate any suggestions for workarounds, and I would be happy
>>> to provide additional information or perform any further tests to
>>> assist the maintainers with this possible issue.
>>>
>>> Best regards,
>>>
>>> David Lawrence
>> David:
>>        Reentrancy requires setup on your part, of which I see none in your
>> supplied fragments.  Generate the manual and look at the reentrancy section.
>> (You can also look at libc/reent/reent.tex, avoiding the need to make the manual
>> but then being in source with formatting commands.)  It describes the 2 methods
>> for handling it. In brief, you either need to directly call the reentrant forms
>> of the functions with a per-thread structure, or you need to "context switch" by
>> setting the global reentrancy pointer for each thread. (No, they do not need to
>> be proper threads, but it gets the point across easily to use that term as an
>> analogy.)  The second method is easier from the perspective of hiding reentrency
>> from the main body of application code.  Both cases require you to define at
>> least one of your own reentrancy structures in addition to the single default one.
>>        A simple technique I have used in an application quite like yours is to
>> have the interrupt handler save _impure_ptr, overwrite it with the ISR's own,
>> print, then restore _impure_ptr.
>>                   Craig

--
Dave Nadler, USA East Coast voice (978) 263-0097, [hidden email], Skype
  Dave.Nadler1

Reply | Threaded
Open this post in threaded view
|

Re: reentrant sprintf causes segmentation fault

Joel Sherrill
Since FreeRTOS was mention, I will throw out that for RTEMS, the
reentrancy structure is completely managed for you and there are
no oddities. Works fine even in SMP configurations. This is all integrated
into the standard source.

--joel

On Fri, May 3, 2019 at 12:43 PM Dave Nadler <[hidden email]> wrote:

> David - If you're going to use FreeRTOS or similar, this might prove
> helpful:
> http://www.nadler.com/embedded/newlibAndFreeRTOS.html
>
> On 5/3/2019 1:25 PM, David Lawrence wrote:
> > Dear Craig,
> >
> > Thank you very much for the explanation and pointer to the manual.
> > Indeed, the manual is quite clear on this topic and I'm embarrassed
> > that I missed it on my first reading. Sorry to waste your time with
> > this question :-)
> >
> > Best regards,
> > David
> >
> > On Fri, May 3, 2019 at 1:18 PM Craig Howland <[hidden email]>
> wrote:
> >> On 5/3/19 12:43 PM, David Lawrence wrote:
> >>> Dear newlib maintainers and friends,
> >>>
> >>> I am having some trouble with newlib on a bare-metal embedded system
> >>> that makes reentrant calls to sprintf. The reentrant call reliably
> >>> experiences a segmentation fault. I have tested with both newlib 2.4.0
> >>> and newlib 3.0.0, with both newlib-nano and regular newlib, to no
> >>> effect.
> >>>
> >>> ...
> >>>
> >>> Do I have a fundamental misunderstanding of newlib's capability to
> >>> support reentrant calls? Or does this seem like a legitimate bug? I
> >>> would appreciate any suggestions for workarounds, and I would be happy
> >>> to provide additional information or perform any further tests to
> >>> assist the maintainers with this possible issue.
> >>>
> >>> Best regards,
> >>>
> >>> David Lawrence
> >> David:
> >>        Reentrancy requires setup on your part, of which I see none in
> your
> >> supplied fragments.  Generate the manual and look at the reentrancy
> section.
> >> (You can also look at libc/reent/reent.tex, avoiding the need to make
> the manual
> >> but then being in source with formatting commands.)  It describes the 2
> methods
> >> for handling it. In brief, you either need to directly call the
> reentrant forms
> >> of the functions with a per-thread structure, or you need to "context
> switch" by
> >> setting the global reentrancy pointer for each thread. (No, they do not
> need to
> >> be proper threads, but it gets the point across easily to use that term
> as an
> >> analogy.)  The second method is easier from the perspective of hiding
> reentrency
> >> from the main body of application code.  Both cases require you to
> define at
> >> least one of your own reentrancy structures in addition to the single
> default one.
> >>        A simple technique I have used in an application quite like
> yours is to
> >> have the interrupt handler save _impure_ptr, overwrite it with the
> ISR's own,
> >> print, then restore _impure_ptr.
> >>                   Craig
>
> --
> Dave Nadler, USA East Coast voice (978) 263-0097, [hidden email], Skype
>   Dave.Nadler1
>
>