Utf8 issues with alpine linux

Help & Support
1

Hi,

My application runs just fine from my laptop (OSX). I’m trying to build a docker image to run my app based on alpine (edge) linux. The application (a http server) builds fine and runs but crashes when trying to process a http response:

Unhandled exception in spawn(name: d7eb4036): Invalid encoding: utf-8 (ArgumentError)
  from /usr/lib/crystal/core/iconv.cr:29:35 in 'initialize'
  from /usr/lib/crystal/core/iconv.cr:9:3 in 'new'
  from /usr/lib/crystal/core/io/encoding.cr:61:16 in 'initialize'
  from /usr/lib/crystal/core/io/encoding.cr:60:5 in 'new'
  from /usr/lib/crystal/core/io.cr:496:7 in 'decoder'
  from /usr/lib/crystal/core/io.cr:545:10 in 'gets_to_end'
  from /usr/lib/crystal/core/http/client/response.cr:81:15 in 'consume_body_io'
  from /usr/lib/crystal/core/http/client/response.cr:105:9 in 'from_io?'
  from /usr/lib/crystal/core/http/client.cr:562:5 in 'exec_internal_single'
  from /usr/lib/crystal/core/http/client.cr:548:5 in 'exec_internal'
  from /usr/lib/crystal/core/http/client.cr:544:5 in 'exec'
  from /usr/lib/crystal/core/http/client.cr:666:5 in 'exec'
  from /usr/lib/crystal/core/http/client.cr:370:3 in 'post'
:

I’m guessing I’m missing some libraries. Here’s the apk command from my Dockerfile:

RUN apk add --update --no-cache crystal shards musl-dev libressl-dev zlib-dev libpq sqlite-dev

I’m using alpine edge because I want the current version of crystal.

As an extra data point, if I build my docker image from crystallang/crystal rather than alpine:edge it runs without crashing.

Any suggestions?

Thanks,

Steve

2

Another data point. I’ve built against alpine:latest (3.10) which now contains crystal 0.29 and see exactly the same issue.

3

I’ve managed to reproduce the issue with a simple test case.

The git repo is here.

Changing the first line of the Dockerfile to

FROM alpine:3.9

causes the error to go away. This builds against crystal 0.27.0.

Cheers,

Steve

4

Something changed in musl-libc or alpine 3.10 related to iconv encodings. Maybe it needs an additional apk?

5

Same error here, trying to build a static executable from the latest alpine docker image, but http/client throws an exception about utf-8 encoding. Any idea about which apk would be missing ? Been trying to investigate the changes in musl-libc, but I found nothing.

6

Write a small program like this:

from = "utf-8"
to = "utf-8"
Iconv.new(from, to)

if that fails, try with “UTF-8” and see if that works. If that works it means that iconv changed what they accept for valid encodings, maybe they dropped support for “utf-8” and they only accept “UTF-8”, I don’t know.

Could someone try this on their alpine docker image?

The fix might be to always send encodings uppercased.

8

So, no error with the exact code you wrote.

Crystal 0.29.0
musl 1.1.22-r2

9

I guess you’ll need to debug the application to find out what’s getting to Iconv.new and why it’s failing.

This is easy: you can modify iconv.cr directly in your Crystal distribution, or reopen it, copy the initialize method in your program and add debug info there (puts). Same goes with whatever you get from the http client.

It would be helpful to know which HTTP endpoint you are hitting too.

But all of this can be solved with debugging the code.

10

Good, will do that when I have time.

I’m hitting the bitbucket api (api.bitbucket.org) and using basic auth.

Thanks for the info

EDIT:

@asterite @steve-baldwin

While compiling the :musl flag isn’t on even if it’s compiling with it.

if you add -Dmusl when building it will set it manually. I don’t know why it’s not on or if it should be.

Hope this help you!

12

I’ve been debugging this with

  def initialize(from : String, to : String, invalid : Symbol? = nil)
    original_from, original_to = from, to
    puts from
    puts to
    puts invalid

in /usr/lib/crystal/core/iconv.cr

This is what it’s showing:

utf-8
UTF-8
skip

Crash is reproducible with

from = "utf-8"
to = "UTF-8"
invalid = :skip
Iconv.new(from, to, invalid)
13

Setting “@skip_invalid = false” in /usr/lib/crystal/core/iconv.cr “fixes” the issue.

Why call iconv at all when trying to convert utf-8 to UTF-8? Only call it if lowercase(from) != lowercase(to).

Edit: Building with -Dmusl (crystal build -Dmusl ...) fixes the issue for me. See https://github.com/crystal-lang/crystal/issues/7946

14

I can reproduce it in Alpine 3.10, but not in Alpine Edge.
We can consider it being fixed in the future stable release, 3.11.